Rust AI at Adyen: Building the Next-Gen Data Engine

The Night Rust Met Spark at Adyen

On the evening of 9 September 2026, Adyen’s Amsterdam office hosted the Rust AI Europe meetup, the evening before PyData Amsterdam. The theme was that Rust is becoming a foundation for data and AI infrastructure, and the evening had three talks: Zemin Piao on evolving Spark at Adyen, the LakeSail team on how they rebuilt Spark in Rust, and Robin Everaars (HoistIT) on data sovereignty and Rust.

The Venue: Adyen’s Amsterdam Event Space

The talks ran in a large open-plan event space with a stage, a wide LED screen and rows of white chairs. Before the start, the screen showed Adyen’s own house rules for its rooms: “Making the most of our meeting spaces”.

Adyen’s Amsterdam event space filling up before the first talk.

Zemin Piao: Evolving Spark at Adyen

Zemin Piao (Big Data Platform at Adyen, as his slides put it) opened with “Evolving Spark at Adyen”, with the subtitle “Scheduling, Kubernetes, Observability, AI, Rust”.

Zemin Piao’s opening slide, “Evolving Spark at Adyen”, seen from the audience.

His introduction slide was titled “My life, in one number”: “Everything about me comes in sixes.” Six years at Adyen, six years of Apache Spark, and a cat who is six years old.

“Everything about me comes in sixes”: six years at Adyen and six years of Apache Spark.

Data Has Consequences: Why Compute Matters

The “Why compute matters” slide set the scene for one financial technology platform, where the data platform feeds payment decisions, finance and reconciliation, pricing and billing, merchant insights, and compliance. It summed up the requirements in three words: correct, on time, and at PB scale.

The three requirements for production data: correct, on time, at PB scale.

From there the talk walked through how Spark at Adyen has changed:

• Scheduling: low-criticality work is useful until demand collides, so YARN application priority was used to “use spare capacity, protect critical work”.
• Kubernetes: compute and storage used to compete on the same YARN + HDFS worker hosts. The move was “Move compute. Keep shared data”: Spark workloads go through a shared platform launcher to either YARN or Kubernetes, with resource boundaries between critical, standard and backfill workloads.
• Remote shuffle: local shuffle constrains elastic compute, so a Celeborn remote-shuffle experiment (marked work in progress) targets decoupling compute from shuffle.
• Observability and optimisation: “Manual tuning does not scale”. The approach was deterministic analysis first, then a Spark History MCP to “ask the event log”, producing “suggestions, not automatic tuning”, with a human who approves or rejects each one.

Before Kubernetes: Spark executors and HDFS DataNodes shared the same hosts, so compute load turned into storage latency.

The Celeborn remote-shuffle experiment, marked “work in progress” on the slide.

A Sail Proof of Concept at Adyen

The last part of Zemin’s talk was the Rust experiment. Spark Connect was “the seam that made the experiment possible”: a PySpark client talks a protocol to a server, and that server can be Apache Spark on the JVM or Sail, the Rust-native server. The slide title said it plainly: “Keep PySpark. Change the server.”

The test used a production-shaped ingestion path: “Same DAG. Different lifecycle.” An Airflow task with a LakeSail operator started a single Kubernetes container holding the PySpark client and the Sail server, reading from Postgres over JDBC and writing to Delta Lake on HDFS, while keeping the orchestration and platform contracts as they were.

The result slide was careful: “Correct—and directionally faster.” Row counts and values matched, and task wall-clock time went down. The slide also called it a bounded signal: a bounded sample, with deployment and engine effects not isolated. It was “enough evidence to continue exploring”, not a production migration. The closing slide listed three parts of the journey: share capacity intentionally, ground suggestions in evidence, and test Sail without rewriting PySpark.

The Rust Stack: Why Language Matters at Scale

LakeSail’s talk explained why you’d build the whole Spark-compatible stack in Rust. The architecture, as the slides described it:

1. Rust runtime replaces the JVM. Sail is “100% Rust, JVM-free”.
2. Spark Connect protocol, so existing PySpark code keeps working: “switch your endpoint, keep your code”.
3. Apache Arrow and Apache DataFusion end to end, columnar throughout, with no row-to-columnar conversion tax.
4. PyO3 for UDFs: Python UDFs run in-process and share Arrow memory with the engine.

Inside Sail: How LakeSail Rebuilt Spark in Rust

The LakeSail team (two presenters) gave the second talk, “Inside Sail: Rebuilding Spark in Rust”, subtitled “A technical tour of Spark compatibility, Arrow-native execution, query planning, distributed runtime, Python UDFs, and Rust-native lakehouse formats.” The deck was 48 pages long. These are the parts I photographed.

The opening slide of LakeSail’s “Inside Sail” talk.

Accelerate vs. replace

The first slides explained why Sail rebuilds the engine instead of bolting native operators onto Spark. Accelerators such as Photon, Comet, Gluten/Velox and Auron intercept the physical plan and offload operators to native code, but stay inside the JVM boundary. Sail replaces the JVM with a Rust-native engine built on Arrow and DataFusion. The slide’s summary: “Spark compatibility is the contract and the execution runtime is the opportunity.”

A drop-in Spark Connect server

“What Is Sail” described a drop-in Spark Connect server in Rust. You point SparkSession.builder.remote() at a Sail endpoint (the slide used sc://sail-host:50051) and the SQL and DataFrame APIs stay the same. It targets Spark 3.5.x and Spark 4.x and has two deployment modes: a single node (simpler ops, smaller data, local dev and testing) and a distributed cluster on Kubernetes with stateless, elastic workers. The Sail README says the same: compatible with the Spark Connect protocol, with “no code rewrites required”, and designed for Spark 3.5.x, Spark 4.x and later.

Under the hood, the Spark Connect client is a thin gRPC stub that builds an unresolved logical plan. Sail decodes the protobuf plan and runs it in Rust on DataFusion and Arrow, behind the same wire format. SQL goes through Sail’s own parser, built with parser combinators into a type-safe, compile-time-checked AST.

“Switch your endpoint, keep your code”: Sail is Spark Connect compatible, on one node or on Kubernetes.

Python UDFs in-process

The slides called Python UDFs “the largest single source of speedup”. In Spark, Python sits outside the engine: every batch crosses a process boundary to a Python worker and pays a SerDe tax. In Sail, the engine and CPython share one address space through PyO3: the engine reaches the UDF node, invokes Python inline on the Arrow batch, and returns with zero copy and zero serialisation. Because Python is part of the execution plan, fusion and vectorisation still apply, across UDF, UDAF, UDWF and UDTF (Python, Pandas and Arrow). One slide also cited RedNote, where, the slide said, Spark and the accelerators they evaluated could not run a Python-heavy AI workload that now runs on Sail.

The actor model

The distributed runtime is built on actors: “Concurrency without locks: don’t communicate by sharing memory, share memory by communicating.” The driver and the workers are actors, and each one processes messages on a single thread, so there are no shared-memory races and no mutexes. The framework is purpose-built on Tokio, “with no actor hierarchy, no remote actors, and no JVM/Pekko weight”. Workers hold no durable state, so the driver can reschedule a failed region onto any available worker.

Sail’s actor model: one thread per actor, state changed by messages, and stateless workers.

Job graphs, task regions and pipelined shuffle

A job is a DAG of stages, broken into tasks and attempts. The driver manages the job graph, and workers run tasks through a task runner. The unit of scheduling and failure recovery is a task region: a set of tasks across stages.

With pipelined shuffle, partitions stream between stages in memory within one task region. Arrow batches travel over Arrow Flight between live tasks, and nothing is durably materialised. That’s the fast path, but it has a trade-off the slide named “coupled failure”: if one task fails, in-flight data is gone and the whole region retries. Tasks from different stages can share a task slot (“slot sharing”).

Pipelined shuffle: one task region where everything streams and runs together.

For blocking shuffle, a boundary separates task regions and data is written to object storage or Celeborn before downstream tasks read it. On object storage, Sail inserts a no-op reducer stage so that mapper chunks merge into one object per reduce partition, instead of every mapper writing one tiny file per partition. The slide compared this to the push-based shuffle merging in the Magnet paper. “Persistence is not overhead”, another slide said: it buys resiliency, elasticity and the ability to run big jobs on small clusters, because only one task region runs at a time.

Apache Celeborn, native to Sail

Apache Celeborn is “a second blocking shuffle backend, supported from Sail 0.7.1”. Sail’s driver actor registers with the Celeborn masters and gets slot allocation, and Sail’s worker actors push and fetch data to Celeborn workers, which store, merge and serve shuffle data. The Celeborn client in Sail is written in Rust and handles TCP, protobuf and binary framing, so no JVM runs inside Sail. It reuses Sail’s actor model and existing gRPC channel. The configuration on the slide was SAIL_CLUSTER__SHUFFLE_BACKEND__CELEBORN__MASTER_ENDPOINTS.

I checked this against the project. The Sail v0.7.1 release notes list “add Celeborn shuffle backend”, and the shuffle backends documentation describes three options: Arrow Flight (the default pipelined shuffle), a storage backend and Apache Celeborn, the last two being blocking shuffle.

Celeborn as a second blocking shuffle backend, built on Sail’s actors.

Lakehouse formats and MERGE on Delta Lake

The lakehouse part started with the difference between listing tables (a directory layout, where updates mean a full table rewrite) and lakehouse tables (atomic manifest writes, where updates are efficient). A support matrix showed Delta Lake, Iceberg and DuckLake. Delta Lake had ticks for read (partition pruning, time travel), write (schema evolution, column mapping), delete, merge and update. Iceberg had ticks for read, write and merge. DuckLake was marked as a pull request in progress. The note under the table: “Update is simpler than Merge, but we tackled Merge first to enable a more extensible DML architecture.”

The Delta Lake journey went from adding the delta-rs and delta-kernel-rs dependencies to removing them, because “lakehouse contract needs deep engine integration”. Metadata processing moved to the physical execution phase.

The MERGE case study was “the most complex DML operation, tackled first for informed architecture decisions”. At a high level, Sail removes all files touched by the ON and MATCHED AND conditions, writes new files for the UPDATE and INSERT results, and writes a manifest at the end for added and removed files. In the query planning pipeline, a SQL AST or Spark Connect plan becomes a MergeIntoNode, is expanded into a logical plan, and is turned into a physical plan by a custom planner extension. The simplified physical plan ends in a DeltaCommitExec.

MERGE on Delta Lake, in three steps.

The Delta Lake features page in the Sail docs now lists MERGE with matched, insert and WHEN NOT MATCHED BY SOURCE clauses, plus UPDATE and DELETE with copy-on-write and deletion vectors. The key takeaways slide said that lakehouse formats need deep engine integration, that “metadata is data”, and that composable plan nodes enable operations across formats. Future work was reusable plan nodes for Delta Lake, Iceberg and DuckLake, and testing and benchmarking distributed DML at scale.

Catalogs and the system catalog

Sail’s CatalogProvider is a single, flat, async trait for databases, tables and views. It covers catalog objects only (create, get, list and drop) and keeps query planning separate, and being async makes it easier to integrate external systems such as cloud catalogs. The comparison slide set this against DataFusion’s hierarchical CatalogProvider, SchemaProvider and TableProvider, which are not async (for performance, so there are no network requests in planning) and also carry query planning. The system catalog exposes internal execution state through SQL: “Observability Through SQL”. The slide showed SELECT * FROM system.session.sessions WHERE status = 'RUNNING' and listed system tables for sessions, jobs, stages, tasks and workers. It said this complements the OpenTelemetry integration, so metrics, traces and logs correlate with catalog queries.

Under the hood, a system table is a TableScan with a custom TableSource that supports filter pushdown. When execute() runs, it sends a message to the session manager actor and awaits the result on a Tokio channel, with no locks thanks to the single-threaded actor event loop. The system catalog documentation uses the same system.session.sessions example.

The system catalog: Sail’s internal state, queryable with SQL.

My take: being able to query sessions, jobs and workers with SQL is the kind of feature platform teams notice. It means the same dashboards and alerting that read your data can also read the state of the engine.

The Benchmarks, as LakeSail Presented Them

The last section of the deck was benchmarks. These are LakeSail’s own numbers from its slides, not independent measurements, and I’m reporting them exactly as shown.

TPC-H (derived), “Sail Speed-up Over Spark (2026 Derived TPC-H)”:

• 10x faster on average
• 98% total cost reduction
• 0 GB shuffle spill, versus more than 115 GB for Spark
• 26 GB peak memory, versus 72 GB for Spark
• Per-query speed-ups from 2,819% (q16) down to 176% (q19), where 100% means 2x faster

The benchmark results page in the Sail docs gives the setup: TPC-H-derived queries at scale factor 100 on an AWS EC2 r8g.4xlarge (16 vCPU, 128 GB RAM), with the 22 queries taking 52.81 seconds on Sail against 534.78 seconds on Spark.

LakeSail’s TPC-H-derived results: 10x faster on average and 98% total cost reduction, according to its slide.

ClickBench, “Sail vs. Spark & Accelerators (2026 ClickBench)”, as relative time (lower is better) on c8g.4xlarge:

ClickBench, “Sail vs. Databricks (2026 ClickBench)”, best hot run: Sail (Parquet) on c8g.4xlarge at x1.00 (16 vCPU, 32 GiB memory) against Databricks (32xXL) at x2.06 (256 vCPU, 1,952 GiB memory). The slide explained that it reported hot-run numbers because the cold run is dominated by data load time (Databricks 41 s against Sail 2 s, about a 20x gap), which reflects data loading and setup rather than query engine performance. ClickBench is ClickHouse’s open analytical benchmark.

LakeSail’s ClickBench slides against Spark, Comet and Databricks.

The final slide was two lines: “Stop tuning the JVM. Replace it.”

My take: vendor benchmarks are a reason to run your own test, not a replacement for one. What I liked is that the two talks fitted together. LakeSail showed big numbers on standard benchmarks, and Adyen showed a small, honest proof of concept on a production-shaped pipeline: correct, directionally faster, and with the caveats written on the slide.

Data Sovereignty and Rust

The third talk of the evening was Robin Everaars (HoistIT) on “Data Sovereignty and Rust”. It covered 27 Dutch mental-healthcare record systems, ownership as a Rust feature, and how to “destroy it and prove it”. I wrote it up separately: Data Sovereignty and Rust: Robin Everaars at Rust AI.

What This Means for Production AI

I spend a lot of time thinking about the gap between AI prototypes and production deployments. The most common failure mode I see isn’t in the model layer. It’s in the data layer. Teams get a model working, then hit a wall when they try to feed it production-scale data through a pipeline that was never designed for the throughput, the latency, or the reliability requirements.

Projects like LakeSail’s Sail, the DataFusion ecosystem, and the broader push to rebuild data infrastructure without the JVM aren’t only about faster benchmarks. They’re about a data foundation that can serve AI applications. When Python UDFs run in the same process as the engine, and the engine can be swapped behind a protocol your code already speaks, the cost of trying it drops a lot.

Key Takeaways

1. Spark Connect is the seam. Both talks relied on it: the protocol boundary lets you change the server while keeping PySpark code as it is.
2. Replace, don’t accelerate. LakeSail’s argument is that a Rust-native engine on Arrow and DataFusion removes costs that JVM accelerators can’t, especially for Python UDFs.
3. The runtime design matters as much as the language. Actors without locks, task regions, and a choice between pipelined shuffle, object storage and Celeborn are what make the distributed mode work.
4. Treat benchmarks as a starting point. LakeSail’s slides claim 10x faster and 98% lower cost on TPC-H-derived queries. Adyen’s proof of concept found “correct and directionally faster” on a bounded sample.
5. Python stays. The architecture is “Rust inside, Python outside”, with PyO3 bridging the two without serialisation overhead.

The PyData Preview

The timing was no accident. The next day at PyData Amsterdam, Shehab Amin (LakeSail co-founder and CEO) and Santosh Pingale (Adyen) were giving the joint talk “Modernizing Spark: Performance Boost without Rewrite”.

Resources

• LakeSail: Rust-native data and AI platform
• Sail on GitHub: drop-in Apache Spark replacement written in Rust
• Sail documentation: benchmark results
• Sail documentation: shuffle backends
• PyData Amsterdam 2026: “Modernizing Spark: Performance Boost without Rewrite” by Shehab Amin & Santosh Pingale
• rust.ai: Rust AI Europe meetup series

Recorded at the Rust AI Europe meetup at Adyen, Amsterdam, 9 September 2026, the evening before PyData Amsterdam.

Discussion questions:

• If your Spark workloads are CPU-bound, not I/O-bound, what’s the cost of keeping the JVM?
• Would you trust a vendor’s 10x claim, or run a bounded proof of concept like Adyen’s first?

Related

• Data Sovereignty and Rust: Robin Everaars at Rust AI
• Pixi: Building a Package Manager in Rust at RustNL Meetup
• Rust in 2026: Why Systems Programming Is Eating the World