Develop and ship a pipeline

Pipeline code lives in a git repository. A pull request gives your branch its own copy of the DAGs in Airflow, so you can run it against the real platform without touching production; merging puts it in production. This page walks through that loop.

The repository

A pipeline repository has a databaas.yml at its root and your Airflow code in one directory:

bundle:
  name: my-team
targets:
  dev:
    mode: development
  prod:
    mode: production
artifacts:
  dags: airflow/

• bundle.name names this repository’s set of DAGs. Your Databaas team tells you which name to use; a missing or malformed name fails the deployment.
• targets lists where the code can go: a development target for branches and one production target for main.
• artifacts.dags is the directory that gets deployed.

The repository’s deploy pipeline reads this file.

A typical layout under airflow/:

• Keep the DAG file in dags/ to orchestration only; put the logic in a package directly under airflow/.
• Add every package you create to both pyproject.toml and .airflowignore. Leaving it out of .airflowignore makes Airflow import the package looking for DAGs.

Develop on a branch

git switch -c add-forecast-pipeline

cd airflow
uv sync
uv run pytest

The deploy pipeline

Your repository’s CI deploys the DAGs with the databaas command from databaas-utils. It runs three commands, all driven by databaas.yml:

On a pull request the target is dev and the development copy is named after the branch; on main it is prod. The full command reference is in the databaas-utils documentation.

What validation catches

databaas dag validate fails the pipeline, before anything is deployed, on:

• a DAG file that does not import, including a duplicate DAG id or one longer than 250 characters;
• a repository with no DAGs at all;
• on a branch, a DAG id that does not start with dev_<key>_. Such a DAG would replace the production DAG of the same id; build every DAG id with target.dag_id(...).

Each problem names the file in your repository, and GitHub shows it as an annotation on that file. See validating before a deploy in the databaas-utils documentation for how the check mirrors Airflow.

Validation imports your DAG files, so it needs Airflow and everything your DAG files import at the top. Keep heavy imports inside task bodies, and validation needs little more than Airflow itself. To run the same check before you push, from the environment your tests use:

uv run --with "databaas-utils[deploy]==0.6.0" databaas dag validate -t dev

Set up the repository

Your Databaas team gives you a DAG deployment key, the storage endpoint and the bucket name for your environment. In the repository settings on GitHub, add:

Then add the workflow:

name: Deploy DAGs

on:
  pull_request:
    types: [opened, synchronize, reopened, closed]
  push:
    branches: [main]

permissions:
  contents: read

env:
  DATABAAS_UTILS: databaas-utils[deploy]==0.6.0
  AIRFLOW_CONSTRAINTS: https://raw.githubusercontent.com/apache/airflow/constraints-3.1.8/constraints-3.12.txt
  AWS_ENDPOINT_URL: ${{ vars.DATABAAS_DAGS_ENDPOINT }}
  DATABAAS_DAGS_BUCKET: ${{ vars.DATABAAS_DAGS_BUCKET }}
  TARGET: ${{ github.event_name == 'push' && 'prod' || 'dev' }}

jobs:
  validate:
    if: github.event.action != 'closed'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      # Add a --with for each package your DAG files import at the top.
      - run: |
          uvx --python 3.12 --from "$DATABAAS_UTILS" \
            --with apache-airflow==3.1.8 -c "$AIRFLOW_CONSTRAINTS" \
            databaas dag validate -t "$TARGET"

  deploy:
    needs: validate
    if: >
      github.event_name == 'push' ||
      github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    concurrency:
      group: deploy-dags-${{ github.head_ref || 'main' }}
      cancel-in-progress: false
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.DATABAAS_DAGS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.DATABAAS_DAGS_KEY_SECRET }}
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - run: uvx --from "$DATABAAS_UTILS" databaas dag deploy -t "$TARGET"

  cleanup:
    if: >
      github.event.action == 'closed' &&
      github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    concurrency:
      group: deploy-dags-${{ github.head_ref }}
      cancel-in-progress: false
    env:
      AWS_ACCESS_KEY_ID: ${{ secrets.DATABAAS_DAGS_KEY_ID }}
      AWS_SECRET_ACCESS_KEY: ${{ secrets.DATABAAS_DAGS_KEY_SECRET }}
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - run: uvx --from "$DATABAAS_UTILS" databaas dag destroy -t dev

• Pin databaas-utils and Airflow. Use the Airflow version Databaas runs, with Airflow’s constraints file: without it, a newer SQLAlchemy breaks the Airflow import.
• Pull requests from forks are validated but not deployed: GitHub gives them no secrets.
• Steps that build files into artifacts.dags, such as dbt parse for a Cosmos manifest, go before the databaas command in both the validate and the deploy job.
• uvx installs databaas-utils on the CI runner only, as a tool. It is not one of your pipeline’s requirements, so the rule against installing it still holds.

For Bitbucket Pipelines, see the Bitbucket example in the databaas-utils documentation.

Find your development DAGs

Development DAGs run in the same Airflow as production and are told apart three ways:

• Prefixed: dev_<key>_<name> instead of <name>. The key is derived from your branch name.
• Tagged: dev:<key>. Filter the DAG list on this tag to see only yours.
• Created paused. Unpause yours when you want it to run.

Each development DAG’s docs end with a Deployed from line: the repository, branch, commit and deploy time. Check it when you are unsure whether you are looking at your latest push.

Production DAGs have no prefix and no dev: tag, and are not paused: they start running on their schedule once merged.

The prefix, tag and pause state come from target in your DAG file; see deployment identity.

Keep a branch off production data

Pass every name that identifies data through scoped(), so a branch writes to its own namespace and tables instead of production’s:

from databaas_utils.airflow import scoped

NAMESPACE = scoped("open_meteo")  # "open_meteo" in production, "open_meteo_dev_<key>" on a branch

Creating that branch namespace needs a create grant on the warehouse for your pipeline’s identity — see which identity a pipeline runs as.

Data written under a scoped name is not removed when the pull request closes. Drop the namespaces and tables your branch created once you no longer need them.

Run tests without a deployment

A local checkout is not a deployment. Outside one, scoped() returns the production name, and reading any target attribute raises a RuntimeError. For dag.test() and pytest, put a production-shaped target file next to your DAGs and keep it out of git:

echo '{"mode": "production", "bundle": "local"}' > dags/_databaas_target.json
echo '_databaas_target.json' >> .gitignore

Without a databaas_lakehouse connection, every lakehouse helper raises ConnectionNotConfigured. get_store(local_root="/tmp/lake") and get_zarr_group("t", local_root="/tmp/lake") write to a local folder instead, as an explicit choice. get_catalog and get_index have no local mode: catch ConnectionNotConfigured yourself if a laptop run should skip them.

Mistakes worth knowing before your first pull request