Authoring pipelines
Write an Airflow 3 DAG that reads and writes lakehouse data through databaas-utils.
This page covers writing pipeline code: the shape of a DAG, the databaas-utils helpers that reach
the catalog and object storage, and which libraries you can use. To ship the code, see
develop and ship a pipeline. If you only operate pipelines,
monitoring and running is the page you want to look at.
A pipeline is Python
A pipeline is an Airflow 3 DAG written with the TaskFlow API: a function decorated with @dag that
calls @task functions. Each task does one step; passing one task’s return value into another sets
the order they run in. Databaas runs Airflow 3.1, so import from airflow.sdk, not the Airflow 2
airflow.decorators path.
The DAG below fetches rows from an API and writes them into an Iceberg table, queryable from notebooks, the SQL Editor and dashboards as soon as the load finishes.
"""Daily forecast summary, written to Iceberg."""
from datetime import timedelta
import pendulum
from airflow.sdk import dag, task
from databaas_utils.airflow import scoped
from databaas_utils.airflow.target import target
NAMESPACE = scoped("open_meteo")
@dag(
dag_id=target.dag_id("open_meteo_forecast"),
schedule="0 7 * * *",
start_date=pendulum.datetime(2026, 9, 1, tz="UTC"),
catchup=False,
max_active_runs=1,
is_paused_upon_creation=target.paused,
tags=target.tags("open_meteo"),
doc_md=__doc__ + target.doc_suffix(),
default_args={"retries": 2, "retry_delay": timedelta(minutes=2)},
)
def open_meteo_forecast():
@task
def extract() -> list[dict]:
import requests
response = requests.get("https://api.example.com/forecast", timeout=30)
response.raise_for_status()
return response.json()["rows"]
@task
def load(rows: list[dict], ds: str | None = None) -> int:
import pyarrow as pa
from pyiceberg.expressions import EqualTo
from databaas_utils.airflow import get_catalog
catalog = get_catalog()
catalog.create_namespace_if_not_exists(NAMESPACE)
batch = pa.Table.from_pylist([{**row, "run_date": ds} for row in rows])
table = catalog.create_table_if_not_exists(
f"{NAMESPACE}.daily_summary", schema=batch.schema
)
table.overwrite(batch, overwrite_filter=EqualTo("run_date", ds))
return batch.num_rows
load(extract())
open_meteo_forecast()dag_idis the name shown in the Airflow UI.scheduledecides when it runs on its own;Nonemeans trigger-only.targetandscoped()make the same file safe to deploy both as production and as a development copy for a branch. See deployment identity below.loadoverwrites the slice of rows its run owns instead of appending, so a retried task replaces its own rows rather than doubling them.
See the Apache Airflow TaskFlow documentation for the full model.
What databaas-utils gives you
databaas-utils is preinstalled wherever your pipelines run. It reaches the lakehouse as your
pipeline’s service account, through an Airflow connection named databaas_lakehouse. Your DAG code
never holds a credential or a storage key. To create the service account and the connection, see
set up a service account for pipelines.
Import everything from databaas_utils.airflow, except target (from
databaas_utils.airflow.target) and FileRecord (from databaas_utils). The full API is in the
databaas-utils documentation.
| You want | Call |
|---|---|
| The Iceberg catalog (pyiceberg) | get_catalog() |
| Files under a generic table | get_store(namespace=...) |
| A Zarr group under a generic table | get_zarr_group("<table>", namespace=...) |
| An Iceberg record of which files landed | get_index(namespace=...) |
| A data name a branch deploy cannot collide on | scoped("<name>") |
| This deployment’s DAG id, tags and pause state | target |
| An access token for Trino, for example for dbt | get_settings().auth_header() |
| Catalog, files and index from one named connection, plus a connection check | LakehouseHook(conn_id=...) |
get_catalog, get_store, get_zarr_group, get_index and get_settings take a connection id
(default databaas_lakehouse) if your pipeline uses a different connection: conn_id=, or the
first argument for get_settings. get_store, get_zarr_group and get_index also take
namespace=, which overrides the connection’s default namespace.
Do not pip install databaas-utils or add it to your pipeline’s requirements. The preinstalled
version is the one your DAGs get; a second copy conflicts with it.
Iceberg tables: get_catalog()
get_catalog() returns a standard pyiceberg RestCatalog. Everything in the
pyiceberg docs applies.
- Always use fully qualified identifiers,
"<namespace>.<table>". The connection’s default namespace does not apply to pyiceberg calls. - Overwrite the slice a run owns rather than
append, and keep the filter on one value per run. - Reading works the same way:
catalog.load_table("open_meteo.daily_summary").scan(row_filter=...).to_arrow().
Files: get_store()
Raw files that are not tables yet, such as a GRIB download, a PDF or a CSV drop, go into a generic table: a named, permissioned location in the warehouse.
@task
def fetch_and_store(run_date: str) -> str:
from pathlib import Path
from databaas_utils.airflow import get_store
local = Path("/tmp") / f"{run_date}_f012.grib"
... # download from the source into `local`
store = get_store(namespace=NAMESPACE)
return store.put("harmonie_raw", f"{run_date}/f012.grib", local)| Method | Does |
|---|---|
put(table, key, source) | Uploads a local file and returns its s3:// URI. Creates the generic table on first use. |
get(table, key, dest) | Streams one object to a local path. |
open(table, key) | Returns a readable file object, without staging to disk. |
list(table, prefix="") | Lists object metadata under the table, in batches: each item is a list of dicts with path, size and last_modified. |
delete(table, key) | Removes one object. |
uri(table, key) | Returns the s3:// URI an object has or would have. Creates the generic table if it does not exist yet. |
put and uri create a missing table. get, open, list and delete never do, so a misspelled
table name there fails instead of silently creating an empty table.
Pass the returned URI between tasks, not the bytes. A task’s return value is stored by Airflow and is fine for a few hundred rows, not for a few hundred megabytes.
Zarr: get_zarr_group()
A Zarr store also lives under a generic table. You get a normal zarr.Group (Zarr 3).
@task
def write_grid(run_date: str) -> None:
from databaas_utils.airflow import get_zarr_group
root = get_zarr_group("harmonie_zarr", namespace=NAMESPACE)
run = root.require_group(run_date)
t2m = run.create_array(
"t2m", shape=(49, 300, 300), chunks=(1, 300, 300), dtype="float32", overwrite=True
)
t2m[:] = ... # your numpy dataPass read_only=True in a task that only reads; a read-only open never creates the table. Pass
path="<subgroup>" to open a group below the root directly, and create=False to make a missing
table an error for writers too.
Record what landed: get_index()
A generic table is a location, not a file list: neither the catalog nor any UI can say which files landed in it. The index is an ordinary Iceberg table with one row per stored object, so you can query it from the SQL Editor, notebooks and dashboards.
@task
def record(uri: str, key: str, size: int, run_date: str) -> None:
from databaas_utils import FileRecord
from databaas_utils.airflow import get_index
get_index(namespace=NAMESPACE).record(
FileRecord(uri=uri, key=key, size_bytes=size, product="harmonie", run_date=run_date)
)- The index table is
<namespace>.filesby default, created on first use. Passtable=for another name. - Only
uri,keyandsize_bytesare required.product,run_date,cycleandforecast_hourare optional, andattributestakes adict[str, str]for anything else. record()takes any number of records and commits them as one append.latest()returns the newestrun_dateandcyclethat actually landed, andscan()returns the whole index as an Arrow table.latest()returnsNonewhen there is no index yet or no row carries a run, and raises when the index exists but cannot be read. It scans the whole table, so call it once per run, not once per file.
Deployment identity: target
The same DAG file runs as production once merged, and as a development copy for every open pull
request, in the same Airflow. target tells the two apart.
| Attribute | Production | Branch deploy |
|---|---|---|
target.dag_id("my_dag") | my_dag | dev_<key>_my_dag |
target.paused | False | True: development DAGs are created paused |
target.tags("a", "b") | ["a", "b"] | ["a", "b", "dev:<key>"] |
target.is_production | True | False |
target.doc_suffix() returns where this copy was deployed from, ready to append to doc_md.
Import it as from databaas_utils.airflow.target import target. from databaas_utils.airflow import target
gives you the module of the same name, and the first target.dag_id(...) fails with
AttributeError: module ... has no attribute 'dag_id'.
Keep a branch off production data: scoped()
Prefixed DAG ids stop two DAGs colliding. They do not stop a branch’s DAG writing into a production
namespace. Pass every name that identifies data through scoped(): namespaces, tables in a shared
namespace, asset URIs, cursors and state keys.
from databaas_utils.airflow import scoped
NAMESPACE = scoped("open_meteo") # "open_meteo" in production, "open_meteo_dev_<key>" on a branch- Call it from the DAG file, at the top or in a task defined there. A helper in your own package
that must call it needs the DAG’s path passed through:
scoped("open_meteo", caller_file=__file__). - Outside a deployment, such as a test on your laptop, it returns the name unchanged.
- Use
databaas_utils.airflow.scoped, not the top-leveldatabaas_utils.scoped. The top-level one can leak a branch’s suffix into production names.
One connection, explicitly: LakehouseHook
LakehouseHook gives you the catalog, the file store and the index from one named connection, and
test_connection() checks that the connection can reach the catalog and warehouse. It has no Zarr,
scoped() or target equivalent; use those helpers directly.
from databaas_utils.airflow import LakehouseHook
hook = LakehouseHook(conn_id="databaas_lakehouse")
catalog = hook.get_catalog()
store = hook.get_store(namespace=NAMESPACE)
index = hook.get_index(namespace=NAMESPACE)Rules
- Import data libraries inside the task body, not at the top of the DAG file. Airflow parses
DAG files in an environment without duckdb, pyiceberg, zarr, dlt or dbt, so a module-level import
of one of them is an import error and the DAG never appears. The
databaas_utils.airflowhelpers are safe to import at the top: they load their libraries only when called. See Python packages for what is available where. - Fail loudly. A missing or misspelled connection raises
ConnectionNotConfiguredrather than writing somewhere else. Do not catch it to keep a task green. - Never put storage keys in code. Nothing in this flow needs an access key. If you find yourself reaching for one, the connection is set up wrong; check it against the service account setup.
- Do not install the upstream
obstorepackage. The preinstalledobstore-databaasshares its module name, and addingobstoreto a task’s requirements replaces it and breaks storage access withRemoteSigningUnavailable. - Keep tasks idempotent. A retry or a cleared task runs again; it must not double the rows.
- Import timetables from
airflow.timetables, notairflow.sdk.from airflow.sdk import CronTriggerTimetablefails on the Airflow version Databaas runs.
Libraries you can use
These are preinstalled and importable from any task. Import them inside the task body, except
databaas-utils: Airflow parses DAG files without the others (see the rules).
Python packages lists every installed package with its exact version.
| Library | Use it for |
|---|---|
databaas-utils | Catalog, files, Zarr, index, deployment identity. |
pyiceberg, pyarrow | Reading and writing Iceberg tables. |
duckdb | SQL over Arrow tables and parquet files. To query Iceberg tables, load them with the lakehouse helpers first. |
zarr | Arrays; open them through get_zarr_group(). |
dlt | Extract and load from APIs. See dlt and dbt. |
dbt-core, dbt-trino | SQL transformations. See dlt and dbt. |
requests | HTTP calls. |
pandas is installed too, as a dependency, so you can import it in a task.
Python packages lists it under other packages.
Add a package that is not preinstalled
For a dependency specific to one pipeline, swap @task for @task.virtualenv. Airflow builds a
virtual environment for the task, installs the requirements into it, and runs the function there.
from datetime import timedelta
from airflow.sdk import task
@task.virtualenv(
requirements=["example-package==1.4.0"],
system_site_packages=True,
execution_timeout=timedelta(minutes=90),
)
def download(plan: dict) -> list[str]:
import example_package
...- Pin every version. An unpinned requirement rebuilds the environment and can break the pipeline on a day nothing changed.
- Keep
system_site_packages=True, so the preinstalled libraries stay visible and you list only what is missing. - It installs on every run. Each task starts in a fresh environment, so
venv_cache_pathhas nothing to reuse. Budget the install time inexecution_timeout, and use a custom image when the install is slow. - Do not list anything that is already preinstalled, and never
databaas-utilsorobstore. - The body closes over nothing. It runs in a fresh interpreter: every import goes inside the
function, module-level constants must be inlined or passed as arguments, and only the arguments and
the return value cross the boundary. To import your own package, pass its directory in and add it
to
sys.pathinside the function. - Private index:
index_urlsandpip_install_optionspoint pip at a private package index.
A package imported at the top of a DAG file must be available when Airflow parses the file, so a virtualenv task cannot supply it. Move the import into the task body. If a dependency must be available everywhere, ask your Databaas team.
For system libraries, command-line tools or a large set of packages, run the task in a custom image instead.
Which identity a pipeline runs as
A pipeline does not run as you. It reads and writes the lakehouse as the service account behind the
databaas_lakehouse connection, with that account’s data grants:
set up a service account for pipelines covers what to grant. A 403
on a table or namespace, or a SignError on a file, means the service account has no grant there.