OpenPX

OpenPX is an open-source take on the parallel ETL engines I worked with early in my career: one Rust engine with a visual designer, composable operators, distributed execution and a portable dataset format. You design a pipeline on a laptop and scale it to an on-prem cluster or cloud object storage with the same engine and compiler, and no rewrite.

Architecture

• Arrow-native core. A pull-based, partition-parallel execution engine that keeps data in Apache Arrow in memory and Parquet on disk, with a deterministic type system across 18 composable operators.
• Inspectable compiler. YAML jobs and Designer graphs lower to a logical plan, then a physical plan, with schema inference, cycle detection and per-node diagnostics. You can inspect a job before a single row moves.
• Explicit partitioning. HASH, RANGE, ROUND_ROBIN, ENTIRE and SAME partitioners. The compiler inserts shuffles only where operator contracts require them.
• Pluggable backends. A native compute backend plus an optional Polars lazy backend. Consecutive stages on the same backend are fused into one plan.
• Distributed execution. A conductor drives gRPC workers across hosts, and output is verified byte-identical from 1 to N workers.
• Portable .opx datasets. Parquet parts plus a metadata descriptor with relative paths, so datasets move between environments unchanged.
• Pluggable storage. One storage interface dispatches by URI scheme to the local filesystem, S3, MinIO, Azure Blob or GCS.

Editions

• Local (community). Self-managed and disk-only, for developers, consultants and air-gapped or on-prem use. Runs single-node with Docker Compose, or multi-node with a conductor and gRPC workers.
• Cloud. A strict superset of Local that adds object storage, hosted worker pools and Terraform-based deployment on AWS ECS Fargate.

Datasets written by one edition open in the other with no conversion.

UI suite

Four React apps talk to either edition over the same REST API:

• Designer: visual job builder (compile and run from the canvas)
• Admin: platform administration
• Director: runtime monitoring
• Quality: data governance and column-level lineage

Pipelines can also be driven from the HTTP API or the openpx CLI.