https://www.alexanderjunge.net/tags/airflow/ Recent content in Airflow on Alexander Junge's website Hugo -- gohugo.io en-US Mon, 22 Feb 2021 00:00:00 +0000 https://www.alexanderjunge.net/blog/taskflow-airflow-2/ Mon, 22 Feb 2021 00:00:00 +0000 https://www.alexanderjunge.net/blog/taskflow-airflow-2/ I recently switched to version 2 of Apache Airflow which was released in December 2020. I am a big fan of the new TaskFlow API and want to highlight it here. The TaskFlow API allows users to write DAGs in a much more efficient way, requiring less boilerplate code. Specifying task dependencies and exchanging data between tasks via XComs is also much easier now. A minimal example of a DAG using the TaskFlow API looks something like this: https://www.alexanderjunge.net/blog/arxiv-airflow-fastapi-psql/ Sun, 02 Feb 2020 00:00:00 +0000 https://www.alexanderjunge.net/blog/arxiv-airflow-fastapi-psql/ Querying arXiv preprints using Apache Airflow I experimented with Apache Airflow to schedule hourly workflows fetching recent preprint articles from different arXiv categories via the public arXiv.org REST API. These articles are then stored in a PostgreSQL database via a custom-built fastAPI-based REST API. The setup looks like this: The code is fully dockerized and available on GitHub along with more detailed documentation.