Run a task in a custom image

When a task needs system libraries, command-line tools or a large set of Python packages, build your own image on top of the Airflow runtime and point the task at it. The rest of your DAG keeps running in the standard runtime.

Choose between a virtualenv and a custom image

KubernetesPodOperator, @task.kubernetes and @task.docker do not work on Databaas: a task cannot start another container. A custom image gives a task the same freedom, and the task still gets its connections, logs and retries like any other.

Build the image

Start from the Airflow runtime version your Databaas version runs. Find it under which runtime your Databaas version runs.

FROM rg.nl-ams.scw.cloud/databaas-public/airflow-runtime:1.0.0

# System packages need root. Switch back afterwards: tasks run as the airflow user.
USER root
RUN apt-get update \
    && apt-get install -y --no-install-recommends cdo \
    && rm -rf /var/lib/apt/lists/*
USER airflow

# Freeze Airflow's own packages, so nothing in requirements.txt can change the Airflow version.
COPY --chown=airflow:root requirements.txt /tmp/requirements.txt
RUN pip freeze | grep -iE '^apache-airflow[-a-z0-9]*==' > /tmp/airflow-pins.txt \
    && pip install --no-cache-dir -r /tmp/requirements.txt -c /tmp/airflow-pins.txt \
    && rm /tmp/requirements.txt /tmp/airflow-pins.txt

cdo==1.6.1

• Pin every version in requirements.txt, as in a virtualenv task.
• Do not change the entrypoint, the command, the airflow user or the environment variables the runtime sets, such as PYTHONPATH. Airflow relies on them to start your task, load your DAG code and upload the task log.
• Do not reinstall what the runtime ships: never databaas-utils, and never the upstream obstore package, which replaces the preinstalled obstore-databaas and breaks storage access.

Check that the build kept the runtime’s Airflow version:

docker run --rm <your-image> python -c "import airflow; print(airflow.__version__)"

It must print the Airflow version listed for your runtime on Python packages.

Push the image

Push the image to the registry your Databaas team gave you, and give every build its own tag, such as the version of your pipeline or the commit it was built from. Never push a new build under an existing tag: an image is cached by its tag, so tasks can keep running the old build.

Point a task at the image

Pass the image to the task through executor_config:

from kubernetes.client import models as k8s
from airflow.sdk import task

CDO_IMAGE = "<your-registry>/climate-tools:2026.10.1"
PULL_SECRET = "<your-pull-secret>"

in_cdo_image = {
    "pod_override": k8s.V1Pod(
        spec=k8s.V1PodSpec(
            image_pull_secrets=[k8s.V1LocalObjectReference(name=PULL_SECRET)],
            containers=[k8s.V1Container(name="base", image=CDO_IMAGE)],
        )
    )
}


@task(executor_config=in_cdo_image)
def compute_indices(path: str) -> str:
    from cdo import Cdo

    ...

• Name the container base and set only its image, plus the pod’s image_pull_secrets. Anything else you set there replaces what Databaas configures for the task, and the task can fail to start or lose its log.
• The pull secret lets the task pull your image from your private registry. Without it the task never starts and has no log.
• Import your image’s packages inside the task body. Airflow parses the DAG file without them, exactly as for runtime packages. The kubernetes import at the top is fine: Airflow has it.
• Run every task in a DAG in the image by passing "executor_config": in_cdo_image in the DAG’s default_args. Classic operators such as PythonOperator take the same executor_config argument.

Validation imports your DAG file, so it needs the kubernetes import too. Add the Airflow package that provides it to the environment your validation runs in, next to Airflow itself and with the same constraints file:

--with "apache-airflow-providers-cncf-kubernetes==10.13.0"

Rebuild when the runtime changes

Your image keeps the runtime version it was built from. When your Databaas version moves to a new Airflow runtime, change the FROM line to that version, rebuild, and push the image under a new tag. An image left on an older runtime keeps an older Airflow and older libraries than the rest of your pipelines, and its tasks can fail once that Airflow no longer matches.