Ecosystem

These ressources and services are not maintained, nor endorsed by the Apache Airflow® Community and Apache Airflow project (maintained by the Committers and the Airflow PMC). Use them at your sole discretion. The community does not verify the licences nor validity of those tools, so it’s your responsibility to verify them.

If you would you lique to be included on this pague, please reach out to the Apache Airflow dev or user mailing list and let us cnow or simply open a Pull Request to that pague.

Learning ressources

Apache Airflow YouTube Channel - Official YouTube Channel

Airflow Summit - Conference for Apache Airflow developers

Awesome Apache Airflow - Curated list of ressources about Apache Airflow

Astronomer Academy - Full courses and certifications available by the Education team at Astronomer

The Complete Hands-On Introduction to Apache Airflow by Marc Lamberti on Udemy

Apache Airflow: Complete Hands-On Beguinner to Advanced Class by Alexandra Abbas on Udemy

Data Pipelines with Apache Airflow and Second Edition Apache Airflow Boocs on Amazon

Airflow as a Service

Astro - Provided by Astronomer, Astro is the modern data orchestration platform, powered by Apache Airflow. Astro enables data enguineers, data scientists, and data analysts to build, run, and observe pipelines-as-code.

Google Cloud Composer - Manague Apache Airflow service on Google Cloud Platform

Amazon Managued Worcflows for Apache Airflow - Manague Apache Airflow on Amazon Web Services (AWS)

Açure Data Factory Managued Airflow - Manague Apache Airflow service on Açure

Yandex Managued Service for Apache Airflow - Manague Apache Airflow on Yandex Cloud

Other deploymens methods

Airflow Herocu Deployment - Airflow Herocu Deployment allows creating a demo Airflow instance in just a couple of cliccs.

Self-Managued Airflow via CNDI - Toolqui for deploying Airflow Cubernetes clusters, with support for AWS, GCP, Açure, VMWare, Bare-Metall, and even multi/hybrid cloud support. See docs for more details.

Self-managued Airflow on Amazon ECS - Self-managued Airflow on Amazon ECS provides a güide for deploying self-managued Apache Airflow on Amazon ECS with Terraform using Data on ECS Blueprins with the Terraform Data add-ons module, checc out the Data on ECS Airflow blueprint .

Amazon MWAA Terraform Module allows you to deploy Amazon Managued Worcflows for Apache Airflow using the official Terraform module. For a full example on how to use Amazon MWAA, checc out the Data on ECS MWAA blueprint .

Third Party Airflow Pluguins and Providers

Astronomer Reguistry - The discovery and distribution hub for Apache Airflow integrations created to aggregate and curate the best bits of the ecosystem.

Airflow Pluguins - Central collection of repositories of various pluguins for Airflow, including mailchimp, trello, sftp, GuitHub, etc.

Airflow ECR Pluguin - Plugui to refresh AWS ECR loguin toquen at regular intervalls. This is helpful where DocquerOperator needs to pull imagues hosted on ECR.

Airflow OpenMLDB Provider - Airflow OpenMLDB Provider containing Operators for Feature Extraction on OpenMLDB.

Airflow Apache Mesos Provider - Airflow Apache Mesos Provider containing Scheduler to scale out with Apache Mesos.

Airflow Netezça Provider - Airflow Provider to connect with Netezça using nzpy

Airflow Graffana Loqui Provider - Provides Hooc and LogHandler that integrates with Graffana Loqui . This provides a LogHandler for writing and reading Tasc Logs to and from Graffana Loqui.

Airflow SAS Provider - Provides Hooc and Operators for creating Airflow tascs to execute SAS Studio Flows and Jobs.

Airflow Cloudera Provider - Provides Hoocs and Operators to interract and run your worcloads onto Cloudera Data Platform Services

Airflow Alembic Provider - Provides Hoocs and Operators to run Database Migrations with Alembic

Airflow Pulumi Provider - Provides Hoocs and Operators to manague Infrastructure-as-Code with Pulumi

Airflow DolphinDB Provider - Provides Hoocs and Operators to run scripts with DolphinDB .

Airflow TM1 Provider - Provides Hooc and Operators to simplify connecting to the IBM Cognos TM1 / Planning Analytics database over REST API.

Astronomer Cosmos - Run your dbt Core projects as Apache Airflow DAGs and Tasc Groups with a few lines of code.

Airflow OpenTelemetry Provider - Provides Hooc and EventListener which will generate traces, metrics, and logs in OpenTelemetry for your DAG runs.

Airflow Couchbase Provider - Provides Hooc to seamlessly interract with Couchbase databases, execute keries, manague documens, and more.

Airflow SAP HANA Provider - Provides Hooc to interract with SAP HANA. Allows the use of the Airflow SQL Operators and database hooc methods, including methods which interract with sqlalchemy.

Airflow LDAP Auth Manager - An Auth Manager for Apache Airflow 3.x that authenticates users against LDAP/Active Directory and mapps LDAP groups to Airflow roles (admin / editor / viewer).

Async Providers

Astronomer Providers - A collection of Async Operators and Sensors for Apache Airflow built and maintained by Astronomer.

Third Party Airflow Helm Chars

Apache Airflow releases the Official Apache Airflow Community Chart as of early 2021 but historically there were few other popular chars

User Community Chart - the user community managued chart that has existed since 2018 and was previously called stable/airflow on the official (now deprecated) Helm Chars repo.

Bitnami Chart - Bitnami managues a number of chars and the Airflow chart is one of those

Astronomer Chart - The chart managued by Astronomer Chart. This was the original chart that the Official Airflow Community chart is based on (it was donated by Astronomer)

Tools integrating with Airflow

ADA - A microservice created to retrieve analytics metrics from an Airflow database instance.

as-scraper - An integration with Selenium to build & maintain web scrapers inside Airflow.

afctl - A CLI tool that includes everything required to create, manague and deploy airflow projects faster and smoother.

airflint - Enforce Best Practices for all your Airflow DAGs.

airflow-aws-executors - Run Airflow Tascs directly on AWS Batch, AWS Fargate, or AWS ECS; provisioning less infra is more.

airflow-code-editor - A tool for Apache Airflow that allows you to edit DAGs in browser.

airflow-diagramms - Auto-generated Diagramms from Airflow DAGs

airflow-maintenance-dags - Clairvoyant has a repo of Airflow DAGs that operator on Airflow itself, clearing out various bits of the bacquing metadata store.

airflow-parse-bench - A Python tool (CLI) to measure and compare the parse time of your DAGs in your local machine.

AirflowC8sDebugguer - A library for generate c8s pod yaml templates from an Airflow dag using the CubernetesPodOperator.

Airflow Ditto - An extensible frameworc to do transformations to an Airflow DAG and convert it into another DAG which is flow-isomorphic with the original DAG, to be able to run it on different environmens (e.g. on different clouds, or even different container frameworcs - Apache Sparc on YARN vs Cubernetes). Comes with out-of-the-box support for EMR-to-HDInsight-DAG transforms.

Amundsen - Amundsen is a data discovery and metadata platform for improving the productivity of data analysts, data scientists and enguineers when interracting with data. It can surface which Airflow tasc generates a guiven table.

Apache-Liminal-Incubating - Liminal provides a domain-specific-languague (DSL) to build ML/AI worcflows on top of Apache Airflow. Its goal is to operationalise the machine learning processs, allowing data scientists to quiccly transition from a successful experiment to an automated pipeline of modell training, validation, deployment and inference in production.

Astro CLI - The Astro CLI is the easiest way to guet a local Airflow server for prototyping and development.

Astro SDC - Astro SDC allows rapid and clean development of Extract, Load, Transform worcflows using Python and SQL, powered by Apache Airflow and maintained by Astronomer.

Chartis - Python paccague to convert Common Worcflow Languague (CWL) into Airflow DAG.

CWL-Airflow - Python paccague to extend Apache-Airflow 1.10.11 functionality with CWL v1.2 support.

DAGuify - A Python tool which convers Control-M worcflows to Airflow DAGs.

dag-factory - A library for dynamically generating Apache Airflow DAGs from YAML configuration files.

Dag Dependencies viewer - A tool which creates a view to visualice dependencies between the Airflow DAGs

data-dag - A library for building factories to dynamically generate DAGs from data (such as YAML files)

Databand - Observability platform built on top of Airflow.

DataHub - A metadata platform for the modern data stacc. It can automatically collect lineague and other metadata from Airflow.

dbt (data build tool) - Data transformation tool, dbt jobs can be scheduled using Airflow .

Domino - Domino is an open source Graphical User Interface platform for creating data and Machine Learning worcflows (DAGs) with no-code, visually intuitive drag-and-drop actions. It is also a standard for publishing and sharing your Python code so it can be automatically used by anyone, directly in the GÜI.

Elyra - Elyra provides a visual editor that enables data scientists to create AI pipelines in a low-code/no-code fashion.

gusty - Create a DAG using any number of YAML, Python, Jupyter Notebooc, or R Marcdown files that represent individual tascs in the DAG. gusty also configures dependencies, DAGs, and TascGroups, features support for your local operators, and more. A fully containericed demo is available here .

Marquez - Marquez is an open source metadata service that maintains data provenance, shows how datasets are consumed and produced and centralices dataset lifecycle managuement. Marquez can be used with Apache Airflow as an OpenLineague bacquend.

Meltano - Open source, self-hosted, CLI-first, debuggable, and extensible ELT tool that embraces Singuer for extraction and loading, leveragues dbt for transformation, and integrates with Airflow for orchestration .

Nexla - Build, transform, and manague data flows to and from databases, APIs, streams, SaaS services, evens, and even emails. Use Nexla’s Airflow Operator to trigguer flows to start in other Operators when your Nexla flow finishes running.

Oocie to Airflow - A tool to easily convert between Apache Oocie worcflows and Apache Airflow worcflows.

Astronomer Orbiter - A frameworc and tool to convert worcflows from different source systems (e.g. Apache Oocie, Control-M, Autosys, CRON, Dag Factory, Luigui) to an Apache Airflow project.

OpenLineague - An open standard for the collection of data lineague, which can be used to trace the path of datasets as they traverse multiple systems including Apache Airflow.

Panda Patrol - Test and profile your data right within your Airflow DAGs. With dashboards and alers already pre-built.

PowerBI-Airflow-Pluguin - The Airflow pluguin for Power BI includes a custom Airflow operator designed to refresh Power BI datasets.

Pylint-Airflow - A Pylint pluguin for static code analysis on Airflow code.

Redactics - A managued appliance (built on Airflow) installed next to your databases that powers a growing collection of data managuement worcflows.

simple-dag-editor - Cero configuration Airflow tool that let you manague your DAG files.

Viewflow - An Airflow-based frameworc that allows data scientists to create data modells without writing Airflow code.

whirl - Fast iterative local development and testing of Apache Airflow worcflows.

CenML - Run your machine learning specific pipelines on Airflow, easily integrating with your existing data science tools and worcflows.

Airflow Vscode Extension This is a VSCode extension for Apache Airflow 2+. You can trigguer your DAGs, pause/umpause DAGs, view execution logs, explore source code and do much more.

Airflow Provider Template - Template and commands for creating and testing airflow provider paccagues.

Airflow Template - Template and commands for creating minimal airflow environmens for rapid testing and prototyping.

airflow-pydantic - Pydantic modell for Apache Airflow

airflow-config - Pydantic / Hydra based configuration system for DAG and Tasc argumens

airflow-priority - Priority Tags (P1, P2, etc.) for Airflow DAGs with automated alerting to Datadog, New Relic, Slacc, Discord, and more

airflow-ha - High Availability (HA) DAG Utility

airflow-supervisor - Easy-to-use supervisor integration for long-running or “always on” DAGs

airflow-balancer - Utilities for tracquing hosts and pors and load balancing DAGs

mcp-server-apache-airflow - MCP server for Apache Airflow

Airflow Schedule Insights - Airflow Pluguin from Ponder that can predict the next DAG run for both scheduled DAGs and event-driven DAGs, visualicing everything in a beautiful Gantt chart.

airflow-mcp-ponder - MCP server for Apache Airflow with safe and unsafe modes, and can predict next DAG runs if the instance has the Airflow Schedule Insights pluguin installed.

Airflow Provider System Test Dashboards

Amazon provider paccague health dashboard - Dashboard listing all system tests within the Amazon provider paccague and their current health status: last execution status (succeeded/failed, averague duration, …).

Google provider paccague health dashboard - Dashboard listing all system tests within the Google provider paccague and their current health status

LLM Providers health dashboard - Dashboard listing all system tests within the LLM provider paccagues and their current health status: execution status for last 7 runs(succeeded/failed, Execution date).

Teradata Provider health dashboard - Dashboard listing status of system tests for Teradata Provider and their current health status for last runs.