Tools · SPBoucher
Menu

Apache Airflow

by Apache Software Foundation · Platform · also known as Airflow

Python workflow orchestrator for scheduling and monitoring DAGs of tasks, with hundreds of provider packages and managed offerings from several clouds

Apache-2.0 licenseOpen sourceFreeSelf-hostedCloudSelf-hostable
Visit official site Compare

Link verified Nov 2, 2026

Overview

Apache Airflow is an open-source orchestration platform, originally from Airbnb and now an Apache Software Foundation project under Apache 2.0, for authoring, scheduling and monitoring workflows defined as Python code. Each workflow is a DAG of tasks executed by operators from provider packages that cover databases, cloud services, Spark, dbt, Kubernetes and HTTP APIs; the scheduler handles dependencies, retries, backfills and SLAs, and a web UI shows runs, logs and task state. Airflow 3, released in 2025, introduced a new React UI, a task execution API that isolates workers, DAG versioning, event-driven scheduling and data assets as first-class scheduling triggers.

Airflow is the most widely deployed orchestrator and is available as managed services (Astronomer, Amazon MWAA, Google Cloud Composer). Running it well means operating a metadata database, scheduler, web server and workers (Celery or Kubernetes executors), which is heavier than Dagster or Prefect for small teams. Its task-centric model also handles data-passing between tasks less naturally than asset-oriented tools.

Capabilities

  • DAGs defined in Python with the TaskFlow API
  • Scheduler with retries, backfills and dependency handling
  • Provider packages for clouds, databases, Spark, dbt and Kubernetes
  • Web UI for monitoring runs and logs
  • Celery, Kubernetes and local executors
  • Asset-based and event-driven scheduling (Airflow 3)

Typical workflows

  1. Nightly ELT: extract, load to warehouse, run dbt
  2. Scheduling ML training and batch inference jobs
  3. Coordinating cross-system jobs with sensors and triggers

Strengths

  • Largest ecosystem of operators and integrations
  • Managed services from multiple vendors
  • Mature scheduler and UI

Limitations

  • Several components to operate (database, scheduler, workers)
  • Task-centric model; data passing between tasks is indirect
  • Upgrades between major versions require migration effort

Prerequisites

  • Python environment plus a metadata database (PostgreSQL recommended)

Who it is for

  • Data engineers
  • ML platform teams

Alternatives & comparison

  • Dagster

    Asset-oriented Python orchestrator that models pipelines as software-defined data assets with lineage, checks and a hosted Dagster+ option

    Open sourceOpen source
  • dbt

    SQL-first transformation framework that compiles templated models, tests and documentation into warehouse tables; open-source Core plus a commercial platform

    MixedOpen source
  • GitHub Actions

    GitHub's CI/CD and automation service that runs YAML-defined workflows on hosted or self-hosted runners in response to repository events

    ProprietaryFreemium
  • Prefect

    Python workflow orchestration library and server where decorated functions become observable, retryable flows, with Prefect Cloud as hosted option

    Open sourceOpen source
Compare Apache Airflow with its alternatives →

Sources & verification

Primary sources consulted for this record. Claims are tied to the fields each source supports; the automated link checker re-verifies every URL on a schedule.

  • License
    Airflow LICENSE (Apache 2.0)

    https://github.com/apache/airflow/blob/main/LICENSE

    Supports: license_spdx, source_model

    checked Nov 1, 2026
  • Official documentation
    Airflow documentation

    https://airflow.apache.org/docs

    Supports: capabilities, integrations

    checked Nov 1, 2026
  • Official site
    Apache Airflow

    https://airflow.apache.org/

    Supports: name, summary

    checked Nov 1, 2026
  • Repository
    apache/airflow

    https://github.com/apache/airflow

    Supports: repo_url

    checked Nov 1, 2026
Confidence
High
Reviewed
Nov 1, 2026
Links
Verified · Nov 2, 2026

Spotted an error?

Broken link, outdated license, wrong category, or a tool that no longer exists? Tell us; reports are reviewed by the editor.

Apache Kafka

Apache Software Foundation · Messaging & queues

In-depth entry

Distributed event streaming platform with durable, partitioned, replayable logs, Kafka Connect connectors and Kafka Streams processing

Apache-2.0Open sourceFreeSelf-hostedNov 2, 2026

Apache Spark

Apache Software Foundation · Analytical engines & OLAP

Distributed data processing engine with DataFrame and SQL APIs in Python, Scala, Java and R, plus structured streaming and MLlib

Apache-2.0Open sourceFreeLocalSelf-hostedNov 2, 2026