What is Apache Airflow?

  





Apache Airflow is a platform to programmatically author, schedule and monitor workflows.

Airflow was created as an internal project at Airbnb in October 2014. The project was open sourced in June 2015. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is written in Python and uses the Jinja templating engine to generate workflows. Airflow workflows are composed of DAGs (Directed Acyclic Graphs). A DAG is a collection of tasks that have dependencies on each other. Tasks in Airflow are operators, which perform some sort of action. Operators can be written in any language. Airflow scheduler executes your DAGs on an array of workers while following the specified dependencies. When a DAG is executed, the scheduler creates a DAG Run entry in the database. A DAG Run represents an instance of a DAG. Airflow also has a web interface that allows you to view DAGs, DAG Runs, and task instances. The web interface also allows you to trigger DAGs and view logs. The rich command line utilities make performing complex surgeries on DAGs a breeze.

The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed. Airflow is open source and released under the Apache License. It is built on top of other popular projects Celery, Mako, and SQLAlchemy.

Find more information at https://airflow.apache.org/

Comments

Post a Comment

Popular posts from this blog

ZooKeeper as distributed consensus service

Image
The ability to bring consensus in a network of unreliable services is at the heart of any fault-tolerant distributed system. Apache ZooKeeper is a high-performance open-source server that provides a highly reliable distributed co-ordination and meets the production demands of the web-scale, mission critical applications. ZooKeeper is popular among multi-host, multi-process applications (C, Java bindings) running in data centers. This paper discusses how ZooKeeper is used to implement distributed consensus among the available services.   Every complex distributed application needs a service for maintaining configuration information, naming, providing distributed synchronization, and/or providing group services. Because of the complexity in implementing these solutions for distributed computing, the more natural preference is to use a tried, tested and reliable solution available in the market. ZooKeeper provides a simple interface to a centralized coordinating service. The se
Read more

What is Apache Druid?

Image
  Apache D ru id is a high performance column - oriented , distributed data store designed for OL AP workload s . It is open source , supports horizontal scal ability , and has a low latency SQL interface . Apache D ru id was designed to address the needs of OL AP workload s , which are characterized by a few very large , slowly changing dimensions , and a large number of facts . Druid supports a wide variety of data sources , including relational databases , No SQL data stores , and flat files . Apache D ru id is highly scalable , supporting billions of rows and millions of events per second . Apache Druid has a low latency SQL interface that supports real - time queries on live data . Druid also has a powerful data ingestion engine that supports batch and streaming data sources . Apache D ru id has a rich SQL interface that supports most SQL operations , incl
Read more

What is Apache Kafka?

Image
   Apache Kafka is a distributed streaming platform that offers high - through put , low - lat ency , and durability .. It is a message broker that enables communication between different applications . Apache K af ka is used in a variety of applications , including : - Mess aging - Website activity tracking - Audit trails - Log aggregation - Stream processing The fundamental construct in Kafka is a topic . Topics are like channels in a messaging system . You can publish messages to a topic , and subscribers can consume those messages . Apache K af ka is a distributed system , so topics are partition ed and replicated across multiple brokers . This allows for high availability and fault tolerance . It provides different delivery guarantees : at least once , exactly once , and in order . The at least once guarantee means that each message will be delivered at least onc
Read more