On Wednesday 23 April 2025 I went to an Amsterdam.dev meetup in a bright, plant-filled room at Vandebron in Amsterdam. The evening had two sessions that I photographed: a comparison of Airflow vs Dagster, and a talk from Overstory on Kubernetes and AI for wildfire prevention. The Airflow vs Dagster speaker’s name was not on any slide I photographed, so I describe that talk by its slides only.
Where the orchestrator story started
The talk opened with a “History” slide titled “Orchestrator”. It said the team used to have Jenkins, used as a simple scheduler, and that they ran complicated workflows fully in Python. Then came the line that explains the rest of the talk: Airflow did not solve the problems they had with Jenkins, so they kept looking for a better tool.
The history slide: Jenkins as a simple scheduler, then a search for a better orchestrator.
Airflow vs Dagster on one slide
The comparison slide was a two-column pros and cons list with the Airflow logo on the left and the Dagster logo on the right. As I read it:
- Airflow: a green tick for “Community Standard”; red crosses for “No data-flows within Airflow”, “Dependency hell (single env)”, “Traditional workflow-based architecture” and “Limited integrations”.
- Dagster: a red cross for “Dagster community still small”; green ticks for “Data and metadata can flow between jobs”, “Projects are independent”, “Software-Defined Assets are revolutionary” and “Integrations (k8s, dbt, more …)”.
The Airflow vs Dagster slide. These are the speaker’s pros and cons, not a neutral benchmark.
To check the vocabulary against the docs: Apache Airflow describes itself as an open-source platform for developing, scheduling and monitoring workflows, where a DAG holds the tasks, dependencies and schedule and the workflow is defined in Python. Dagster’s concepts documentation defines an asset as a logical unit of data such as a table, dataset or machine learning model. Assets can depend on other assets, which gives lineage. That is the difference the slide is pointing at: Airflow organises work around tasks in a DAG, while Dagster organises it around the data those tasks produce.
One Dagster, many projects
“Projects are independent” was explained with a second slide about the deployment layout. It showed a Dagster Daemon reading a workspace.yaml that lists two gRPC servers, usercode_deployment_one and usercode_deployment_two, on separate ports, each drawn as a container and labelled “projects”.
The workspace.yaml slide: the daemon loads two separate user code deployments.
In Dagster’s terms, each of those is a code location: a collection of definitions deployed in its own environment, with its own Python dependencies. That is the answer to the “Dependency hell (single env)” cross on the Airflow side of the earlier slide, because two teams can ship different library versions without fighting over one environment.
How the first speaker described the setup
The recordings I made of this talk add detail that the slides only hint at, so this section paraphrases what was said.
- Server side and user side are split. The Dagster server side is the daemon, the web server that provides the UI, and a Postgres database holding run history and metadata. Once deployed it mostly runs by itself, apart from upgrades. As a developer you only touch the user code side, which the speaker called really powerful.
- A config map operator registers new projects. In
workspace.yamlyou have to register every new project. On Kubernetes the file can be a ConfigMap, so the team built an operator that scans the cluster for new Dagster deployments and registers their user code with the daemon automatically, which makes registration invisible to developers. - One job per permutation. For one job they keep a long YAML file of settings and want a run for every combination (the slide’s example multiplied several options of two values each). With a maximum concurrency setting of 15, Dagster creates a couple of hundred runs but executes at most 15 at a time. The code is a generate-permutations step, a dynamic map over them, and a final step that collects the results.
- Software-defined assets in practice. The speaker defined an asset as anything that can be updated, usually a table or a model, shown as small green blocks in the UI that you can materialise individually or in combination. He showed a model fed by two sources, a manually filled Google Sheet that needs validation and an API ingestion, and argued that with assets you say “update when upstream changes” instead of writing one workflow per stream and worrying about concurrent updates to shared downstream models.
- dbt as assets. With a few lines of code a dbt project is imported as assets, so the command-line model becomes something analysts or managers can inspect in a UI, failing assets turn red, and dbt metadata becomes visible. Lineage then links an ingestion step with the dbt model, and because they are separate code locations they can even run different Python versions, for example 3.11 and 3.12, while the data still flows between them.
Overstory: Kubernetes and AI to protect our forests
The second session was from Overstory. The title slide read “Kubernetes and AI To Protect Our Forests: Cloud Native Infrastructure for Wildfire Prevention”, presented by Andrea Giardini, listed on the slide as SRE at Overstory. The slide also carried the “KubeCon + CloudNativeCon EU 2025” footer, and the official KubeCon EU 2025 schedule lists the same title for 4 April 2025 in London, so this was a local run of that talk. According to that listing, the talk is about the infrastructure behind AI-driven wildfire prevention: data pipelines for satellite imagery and environmental data, GPU acceleration, and storage for large datasets.
The Overstory talk. A small contact line on the slide is blurred in this photo.
The recording of the Overstory talk fills in the engineering story. The speaker said it was adapted from the KubeCon talk but focused on how Overstory uses Dagster. The details below are from the audio, paraphrased:
- Why wildfire work needs this infrastructure. The company combines satellite and aerial imagery, and sometimes drones, to find where vegetation has grown close to power lines, combines that with utility data about poles and line heights, and produces a risk profile so the utility can trim vegetation before a fire. Imagery can be as fine as 15 centimetres per pixel, so data volumes are large, and workloads vary between roughly one CPU with four GB of memory and 75 CPUs with terabytes.
- From notebooks to Dagster. They started with JupyterHub on Kubernetes. Repeatability and tracking became painful as clients grew (“which pandas version did we use a year ago?”), and delivery was manual. The goals were to get notebooks out of the pipelines and to automate. They evaluated Airflow, Prefect and, briefly, Argo Workflows, and chose Dagster for being open source with a good community, quick to scaffold, easy to test locally as Python modules, and designed with the cloud in mind.
- Code locations per team and I/O managers. Each team owns one or more code locations deployed independently with their own environment and schedule. I/O managers abstract storage, so the same pipeline reads and writes local files on a laptop and Google Cloud Storage or S3 on Kubernetes without if-statements. He called this hard to do in Airflow.
- An internal wrapper library. The team built a library on top of Dagster ops and assets that adds Kubernetes resource requests to a step, such as a node with a single GPU and at least 32 CPUs and about 390 GB of memory, plus ephemeral volumes of 150 GB as scratch space that disappear with the pod. They plan to hide the Kubernetes specifics further.
- Two execution modes. Either a single pod for a short pipeline with fixed resources, or multiple pods where each step gets its own resources, so a 12-hour pipeline needs a GPU for only one hour.
- A platform on top. The Dagster UI becomes overwhelming with many pipelines, code locations and assets, and per-team statistics are hard to get. So Dagster remains the engine, and they added Cloud Run services (a router, trigger and watcher) that talk through Google Pub/Sub, forward messages to BigQuery and export metrics with OpenTelemetry to Google Cloud Metrics, which feeds a Grafana dashboard.
Overstory and Dagster come up together again in my notes from FikaWorks Day 2025, where Overstory presented a Dagster-on-Kubernetes pipeline story.
What came next
The organisers closed with a “Next events” slide: 19 May for the next meetup (topic still TBD), DevOpsDays Amsterdam on 18, 19 and 20 June with a voucher for 25% off valid for 10 tickets, and KCD Utrecht on 3 July. I wrote up the June conference in DevOpsDays Amsterdam 2025.
The Next events slide at the end of the evening.
My take
The slide’s “Software-Defined Assets” point is the real argument for Dagster: if your question is “is this table fresh?” rather than “did this task run?”, an asset model answers it directly. Airflow’s strengths are its huge community and ecosystem, which the slide also gave it. Which one fits depends on whether your team thinks in tasks or in data, and on how many teams share an environment. If you run either on Kubernetes, isolate user code per team, as the workspace.yaml slide showed.
