The Night Rust Met Spark at Adyen
On the evening of September 9, 2026, Adyen’s Amsterdam headquarters hosted the first-ever Rust AI Europe meetup — a timely warm-up to PyData Amsterdam the next morning. The theme: the Rust programming language is rapidly becoming the foundation for the next generation of data and AI infrastructure, and LakeSail is one of the most compelling examples of that shift in production today.
I came for the talks and stayed for the conversations between them — the kind where you realize that what seems like an incremental infrastructure migration is actually a generational platform shift. Here’s what struck me.
The Venue: Adyen’s Amsterdam Engine Room
Adyen’s office space in Amsterdam is the kind of place where serious infrastructure gets discussed. The venue is a large, open-plan floor with floor-to-ceiling windows, exposed industrial ceilings, and the kind of casual-but-intentional design that says “we build things that move real money.” A ping pong table sits just off the main area — a reminder that the culture here is pragmatic, not performative.
Adyen’s Amsterdam event space — industrial design meets technical community.
Zemin Piao: Six Years, One Insight
Zemin Piao — now at LakeSail after his time building data platforms — kicked off with a personal framing that was more revealing than it first appeared. On a slide titled “My Life, In One Number,” he walked through his data engineering journey at Adyen, showing how everything in his six-year trajectory there came in sixes: six major system migrations, six performance orders of magnitude, six teams he’s worked with.
Zemin Piao’s “everything comes in sixes” framing — six years at Adyen, six orders of magnitude of performance improvements.
The number six isn’t just a gimmick. It encodes a real observation: data infrastructure problems compound in orders of magnitude, not percentages. Moving from 10K to 100K requests per second isn’t a 10% optimization — it’s a different engineering problem entirely, requiring a different runtime, a different execution model, and often a different language.
Data Has Consequences: Correctness at Scale
The first half of Zemin’s talk centered on a deceptively simple principle: data has consequences. He walked through how at Adyen’s scale — processing billions of payments across every major market — even “correct on time” isn’t good enough. You also need correctness under failure, correctness under concurrency, and correctness under the kind of load that turns edge cases into common cases overnight.
The three pillars of production data: correct, on time, at PB scale.
The “PB scale” point is where the Rust connection becomes unavoidable. When you’re moving petabytes of transactional data through real-time pipelines, the JVM tax — garbage collection pauses, serialization overhead, memory pressure — stops being a tuning problem and becomes a hard ceiling. At that scale, every GC pause is revenue left on the table, and every serialization round-trip is latency you can’t recover.
The Rust Stack: Why Language Matters at Scale
This is where LakeSail’s approach — building the entire Spark-compatible stack in Rust — moves from “interesting experiment” to “production necessity.” The talk walked through the architecture:
- Rust runtime replaces the JVM — no GC, no warmup, no memory tuning
- Spark Connect protocol means existing PySpark code runs unchanged — zero rewrites
- Apache Arrow for the memory format — eliminates serialization tax between Python and the engine
- PyO3 bindings for UDFs — your custom Python logic runs at native speed without the JVM bridge
One of the community members taking the floor during the open mic session.
The performance implications are the kind of numbers that sound fabricated until you verify them. In a side-by-side comparison running the same PySpark workload, the Rust-backed engine delivered 10x faster execution at 98% lower infrastructure cost — not because it was hand-tuned, but because the runtime itself doesn’t impose the overhead floors that JVM-based systems hit at scale.
Why Adyen: Production, Not Prototype
What made this meetup different from a typical “Rust is fast” talk is that Adyen is running this in production. Not a migration. Not a pilot. Production workloads processing live payment data on the Rust-native stack with Spark Connect compatibility.
The key insight from the Adyen team: they didn’t set out to rewrite their data platform. They set out to serve payments with lower latency and higher reliability. The data platform was always a means, not an end. When LakeSail’s approach let them swap one line of config to replace the engine while keeping 100% of their PySpark code, the question wasn’t “should we try this?” — it was “how fast can we roll it out.”
The Broader Rust AI Movement
Walking out of the event, the conversation in the hallway was about something bigger than performance benchmarks. The room was full of data engineers who’d spent years wrestling with Spark clusters, JVM tuning, and serialization overhead — people who recognized that the infrastructure stack they’d learned to live with was the problem, not the solution.
The community discussion during the break — this is where the real insights emerge.
The meetup also featured several lightning talks from the community, each pointing toward the same theme: Rust is becoming the lingua franca for systems where correctness, performance, and resource efficiency matter more than developer convenience. In data infrastructure, that ship has sailed — production constraints make the convenience argument academic.
What This Means for Production AI
I spend a lot of time thinking about the gap between AI prototypes and production deployments. The most common failure mode I see isn’t in the model layer — it’s in the data layer. Teams spend months getting a model to work, then hit a wall when they try to feed it production-scale data through a pipeline that was never designed for the throughput, the latency, or the reliability requirements.
The Rust AI movement — projects like LakeSail’s Sail, the DataFusion ecosystem, and the broader push to rebuild data infrastructure without the JVM tax — isn’t just about faster benchmarks. It’s about building a data foundation that can actually serve AI applications at scale. When your data engine starts up in seconds instead of minutes, scales to zero when idle, and processes Python UDFs at native speed, the entire economic equation of production AI shifts.
Key Takeaways
- Language choice at the engine level compounds at scale. The JVM overhead that’s tolerable at development scale becomes a hard ceiling at production scale — and that ceiling is hit by more teams every year.
- Drop-in replacement works. Spark Connect protocol compatibility means you get the performance benefits of a Rust runtime without rewriting existing PySpark code. Zero rewrites is a real thing now.
- Production adoption is happening. Adyen running this in production means the bar for “experimental” has shifted — the question is no longer “does it work?” but “how fast can we migrate?”
- The Rust AI stack is converging. From Sail to DataFusion to the broader Arrow ecosystem, there’s a coherent alternative to the JVM-based data stack emerging, and it’s already in production at companies like Adyen.
- Python stays. The winning architecture isn’t “replace Python” — it’s “Rust inside, Python outside” with PyO3 bridging the gap for custom logic without serialization overhead.
The PyData Preview
The timing of this meetup was no accident — it was the evening before PyData Amsterdam, where Shehab Amin (LakeSail co-founder and CEO) and Santosh Pingale (Adyen) are giving the joint talk “Modernizing Spark: Performance Boost without Rewrite.” If this evening’s conversations were any indication, the room will be hungry for real numbers, not benchmarks on GitHub repos.
Technical deep-dive conversations in the hallway — the real value of these community events.
Resources
- LakeSail — Rust-native data and AI platform
- Sail on GitHub — Drop-in Apache Spark replacement written in Rust
- PyData Amsterdam 2026 — “Modernizing Spark: Performance Boost without Rewrite” by Shehab Amin & Santosh Pingale
- rust.ai — Rust AI Europe meetup series
Recorded at the Rust AI Europe meetup at Adyen HQ, Amsterdam, September 9, 2026. The evening before PyData Amsterdam. Zemin Piao is at LakeSail, building the Rust-native data platform that runs Spark workloads 10x faster. Adyen processes billions of payments through infrastructure that’s learning to shed the JVM tax.
Discussion questions:
- If your Spark workloads are CPU-bound, not I/O-bound, what’s the cost of keeping the JVM?
- What would 10x faster execution at 98% lower cost change in your data pipeline budget?
