Connections and variables

Credentials never go in your pipeline repository. They live in Airflow connections, and your task reads them when it runs. Small pieces of pipeline state, such as a cursor, go in variables.

The lakehouse connection

The databaas_lakehouse connection is set up for you. It is how databaas-utils reaches the catalog and storage as your pipeline’s identity, so you do not add storage credentials yourself. Do not edit or delete it. See authoring pipelines.

Add a connection

1. In Pipelines, open Admin → Connections and add a new connection.
2. Connection Id: the name your code asks for, for example source_opendata.
3. Connection Type: the matching type (HTTP, AWS, and so on) if there is one, otherwise Generic.
4. Fill only the fields you need. An API key with no username can go in Password alone.

Read a connection from a task

from airflow.sdk import task


@task
def fetch():
    from airflow.sdk import BaseHook

    api_key = BaseHook.get_connection("source_opendata").password
    ...

get_connection(<conn id>) returns the connection; read .password, .login, .host or .extra_dejson["..."] from it. Airflow masks the password in task logs.

Inside a @task.virtualenv the function body closes over nothing, so write the connection id as a literal inside the function, not as a module-level constant.

Variables are for state, not secrets

A variable is a small key/value entry. Use it for configuration or pipeline state, such as the last file a pipeline ingested:

@task
def ingest():
    from airflow.sdk import Variable

    last = Variable.get("open_meteo_last_archive", default=None)
    ...
    Variable.set("open_meteo_last_archive", newest)

Use a connection for anything that authenticates you, and a variable for everything else. Pass variable keys that differ between production and a branch through scoped(), so a development copy does not move production’s cursor.

Who can read a connection

Every user with can_admin can open every connection and variable in the Airflow UI. There is no per-team scoping. Store shared service credentials there, never a personal one, and choose what you store with that in mind.

• Never commit a credential, not even on a branch.
• Use service credentials, not personal ones. A pipeline that uses one person’s key breaks when that person leaves.