Apache Airflow

What is Apache Airflow?

Apache Airflow is an open source workflow orchestration platform that lets you author, schedule, and monitor pipelines as Python code. You define workflows as DAGs (directed acyclic graphs) of tasks, and Airflow’s scheduler runs them on time, retries failures, and gives you a web UI to watch every run. It is the de facto standard for batch data orchestration.

What is Apache Airflow best for?

Airflow is best for data and platform engineers who want to orchestrate scheduled, batch workflows — ETL/ELT jobs, machine-learning training pipelines, and infrastructure automation — and prefer defining them in code over a drag-and-drop canvas. It shines when pipelines are mostly static, run on a schedule, and need version control, retries, backfills, and clear observability across hundreds of tasks.

What can Apache Airflow do?

• Define pipelines as Python code — workflows are DAGs you write in Python, so they are dynamic, testable, and version-controlled in Git.
• Schedule and trigger runs — cron-style schedules plus event/asset-based triggers, backfills, and catchup for historical runs.
• Scale out with executors — run tasks locally, on Celery workers, or on Kubernetes pods that spin up per task.
• Integrate with almost anything — a large ecosystem of provider packages and operators for AWS, Google Cloud, Azure, databases, dbt, Spark, and more.
• Monitor from a rich UI — Grid, Graph, and Dashboard views show task status, durations, and logs for completed and running work.
• Pass metadata between tasks — XComs let tasks exchange small values, and Jinja templating parameterizes them at runtime.

Where does Apache Airflow fall short?

• It is not a streaming engine. Airflow is built for scheduled batch workflows that are “mostly static and slowly changing.” For real-time or continuous event streaming, a tool like Apache Kafka is the right layer, not Airflow.
• It is code-only, by design. There is no low-code or click-to-build editor — everything is Python. If your team wants a visual canvas, Airflow will feel heavy, and a tool like Kestra (declarative YAML) may fit better.
• Self-hosting the full stack is involved. A production deployment means running a scheduler, a metadata database (PostgreSQL or MySQL), a web server, and — for scale — Celery or Kubernetes workers with a message queue. This is why managed offerings like Astronomer exist.

Is Apache Airflow free?

Yes — Apache Airflow is free and open source under the Apache 2.0 license, with no paid tier or feature gating in the project itself. You can self-host it at any scale at no license cost. You only pay for the infrastructure it runs on, or optionally for a managed provider such as Astronomer or a cloud vendor’s hosted Airflow if you would rather not operate it yourself.

What does Apache Airflow replace?

Airflow is a self-hosted, code-first stand-in for managed orchestration and scheduling products. Teams adopt it instead of Azure Data Factory, AWS Step Functions, enterprise ETL suites like Informatica, and legacy job schedulers such as Control-M — trading a per-run or per-seat bill for full control over an engine you own and extend in Python.

FAQ

Is Apache Airflow open source? Yes. Airflow is a top-level Apache Software Foundation project, released under the permissive Apache 2.0 license, with development happening in the open on GitHub.

Can I self-host Apache Airflow for free? Yes. The software is free to run yourself with no license fee. You provide the compute and a metadata database (PostgreSQL, MySQL, or SQLite for testing); there is nothing to buy from the project.

Is Airflow a good alternative to Azure Data Factory or AWS Step Functions? For teams that prefer code over a managed console, yes. Airflow gives you comparable scheduling and orchestration without vendor lock-in, at the cost of running the infrastructure yourself. If you want managed Airflow, that is what Astronomer sells.

What do I need to run Apache Airflow? Python 3.10–3.14, a metadata database (PostgreSQL 14+ or MySQL 8.0+ in production; SQLite for local trials), and a POSIX system — Linux is recommended for production. Larger deployments add Celery workers or Kubernetes plus a message queue.