Creating an Airflow service account for running pipelines

A pipeline that touches the lakehouse authenticates as its own identity, not as you, and not as a shared platform credential. That identity is a Zitadel service account, granted explicit permissions on a warehouse and handed to Airflow as a connection. databaas-utils reads that connection and does the rest, so no DAG ever holds a credential.

This page is the whole flow, once, for one pipeline. Every step happens in a browser.

Before you start

• Admin in Zitadel for the environment: https://auth.<your-databaas-domain>
• Admin in the catalog UI: https://catalog.<your-databaas-domain>
• can_admin in Databaas, so you can open Admin → Connections in Airflow. See the permissions reference.
• Airflow running an image with databaas-utils 0.2.1 or newer. It is baked into the image; you do not pip install it and you do not add it to a @task.virtualenv requirement list.

<your-databaas-domain> is the domain you open the portal on. Zitadel and the catalog sit on the auth. and catalog. subdomains of it.

1. Create the service account in Zitadel

1. Open https://auth.<your-databaas-domain> and sign in as an admin.
2. Users → Service Users → New. Name it after the pipeline it will run, for example airflow-open-meteo. Access Token Type JWT. Save.
3. Open the new user → Keys → New → type JSON → pick an expiry → Add.
4. The browser downloads a file once. There is no way to retrieve it again. If you lose it, delete the key and make a new one. You paste its contents into Airflow in step 3.

It looks like this:

{
  "type": "serviceaccount",
  "keyId": "391305913807408172",
  "key": "-----BEGIN RSA PRIVATE KEY-----\n…\n-----END RSA PRIVATE KEY-----\n",
  "userId": "391305903103544364"
}

Copy the project ID

While you are in Zitadel, open Projects → Databaas and copy the project’s ID. Step 3 needs it twice. Every token the pipeline gets must carry this project as its audience, or Lakekeeper and Trino reject it.

2. Grant it read/write in the catalog UI

The account can authenticate but cannot see a single table until you grant it something.

1. Open https://catalog.<your-databaas-domain> and sign in.
2. Go to the warehouse (databaas-lakehouse), or the specific namespace the pipeline owns.
3. Permissions → add the service account by the name you gave it in step 1.
4. Grant what the pipeline actually needs, typically select and modify/create on its own namespace.

Prefer granting on the pipeline’s own namespace over the whole warehouse. A pipeline that can only write its own namespace cannot corrupt another team’s tables through a bad overwrite.

3. Add the Airflow connection

This is what the DAG actually reads.

1. In Pipelines, open Admin → Connections and add a new connection.
2. Set Connection Type to DataBaas Lakehouse first. The form then shows the lakehouse fields below, with their own labels.
3. Fill in every field as in the table. Leave the ones marked empty empty.
4. Save.

Paste the key from the file, never retype it. A mangled newline inside the private key fails as an unrelated-looking signature error.

4. Use it in a DAG

Nothing to install and nothing to configure. Ask for what you need and the connection is resolved behind it.

from airflow.sdk import task


@task
def write_forecast(rows: list[dict]):
    import pyarrow as pa
    from databaas_utils.airflow import get_catalog, scoped

    catalog = get_catalog()
    table = catalog.load_table(f"{scoped('open_meteo')}.forecast")
    table.append(pa.Table.from_pylist(rows, schema=table.schema().as_arrow()))

The entry points:

Each takes conn_id= if you registered the connection under a different name, and namespace= to override the connection’s default.

Failure is loud by design. An unresolvable connection raises ConnectionNotConfigured rather than quietly falling back to a local directory. That was the old behaviour, where a typo’d conn id redirected a production write to an ephemeral pod disk and the task still went green. If you genuinely want a local run, pass local_root= and opt in.

Verifying

If your Airflow shows a Test button on the connection form, use it. It is only there when connection testing is enabled in Airflow’s configuration, which Airflow 3 leaves off by default.

Otherwise, add this DAG to your pipeline repository and trigger it once from the Airflow UI:

from airflow.sdk import dag, task


@dag(schedule=None, catchup=False)
def check_lakehouse_connection():
    @task
    def check():
        from databaas_utils.airflow import LakehouseHook

        ok, message = LakehouseHook().test_connection()
        print(message)
        if not ok:
            raise RuntimeError(message)

    check()


check_lakehouse_connection()

Both run the same check. It fetches a token and reads the warehouse, and reports the first link in the chain that broke: the connection fields, the token endpoint, or the warehouse. Success reads Connected to … — warehouse 'databaas-lakehouse' resolved to ….

The check talks to Lakekeeper only, so it does not catch a missing :aud in OAuth scope. If your pipeline queries through Trino, confirm the scope against the table in step 3.

Troubleshooting

Notes

• One account per pipeline, not one per person. A pipeline authenticating as a human breaks when that human leaves, and its access is whatever that person happened to have rather than what the pipeline needs.
• Keys expire. You set the expiry yourself in step 1, so put it in a calendar. Rotation is: create a new key, paste it into the connection’s Service account JSON key field, then delete the old key in Zitadel. Step 2 does not repeat, because the account identity does not change.
• No S3 credentials appear anywhere in this flow. Object storage is reached by asking Lakekeeper to sign each request, so there is no access key to leak, scope or rotate. That is the point of the whole arrangement.
• Grant the narrowest thing that works. The warehouse is the easy answer and usually the wrong one.