Authoring pipelines

If you only operate pipelines, monitoring and running is the page you want.

A pipeline is Python

A pipeline is an Airflow 3 DAG written with the TaskFlow API: a function decorated with @dag that calls @task functions. Each task does one step; returning a value from one task into another sets the order they run in.

A typical ingestion pipeline has two tasks: one fetches source data, one loads it into an Iceberg table in a namespace, making it queryable from notebooks, SQL Lab and dashboards the moment the load finishes.

import pendulum
from airflow.decorators import dag, task

NAMESPACE = "demo_example"

@dag(
    dag_id="example_pipeline",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    schedule="@daily",
    catchup=False,
    tags=["example"],
)
def example_pipeline():

    @task
    def extract() -> str:
        # Fetch or generate source data, write it to object storage,
        # and return where it landed.
        ...

    @task
    def load(source: str) -> dict:
        # Read the source and write it into an Iceberg table
        # in NAMESPACE, so it is queryable everywhere.
        ...

    load(extract())

example_pipeline()

dag_id is the name shown in the Airflow UI; schedule decides when it runs on its own (None means trigger-only). See the Apache Airflow TaskFlow tutorial for the full model.

Loading data as Iceberg

The load step writes through the catalog, so every engine reads the same table. Two common ways:

• DuckDB with its Iceberg extension, attaching the catalog and running CREATE TABLE ... AS SELECT — suits data already in a parquet file or Arrow table.
• pyiceberg, which talks to the catalog directly.

For extract-and-load from an external source, dlt is a good fit and Airflow can orchestrate it from inside a task.

Shipping a pipeline

Pipeline code lives in a git repository your deployment loads from. Once your DAG is added there it appears in the Airflow UI on its own — nothing to register in the portal. How the repository is wired up is deployment-specific, so ask your administrator where pipeline code belongs.