# Run a task in a custom image (/pipelines/custom-images)

Build an image on top of the Airflow runtime and run one task, or a whole DAG, in it.

When a task needs system libraries, command-line tools or a large set of Python packages, build your
own image on top of the Airflow runtime and point the task at it. The rest of your DAG keeps running
in the standard runtime.

<Callout title="Prerequisite">
  The Airflow runtime image is public, so pulling it needs no credentials. Your Databaas team gives
  you a registry to push your own image to, and the name of the pull secret your tasks use for it.
</Callout>

## Choose between a virtualenv and a custom image

| Your task needs | Use |
| --------------- | --- |
| Only packages that are already installed | A plain `@task`. See [Python packages](/pipelines/python-packages). |
| A few extra Python packages | `@task.virtualenv`. See [add a package that is not preinstalled](/pipelines/authoring#add-a-package-that-is-not-preinstalled). |
| System libraries or tools, compiled dependencies, or a dependency set too large to install on every run | A custom image, as on this page. |

`KubernetesPodOperator`, `@task.kubernetes` and `@task.docker` do not work on Databaas: a task cannot
start another container. A custom image gives a task the same freedom, and the task still gets its
connections, logs and retries like any other.

## Build the image

Start from the Airflow runtime version your Databaas version runs. Find it under
[which runtime your Databaas version runs](/pipelines/python-packages#which-runtime-your-databaas-version-runs).

```dockerfile title="Dockerfile"
FROM rg.nl-ams.scw.cloud/databaas-public/airflow-runtime:1.0.0

# System packages need root. Switch back afterwards: tasks run as the airflow user.
USER root
RUN apt-get update \
    && apt-get install -y --no-install-recommends cdo \
    && rm -rf /var/lib/apt/lists/*
USER airflow

# Freeze Airflow's own packages, so nothing in requirements.txt can change the Airflow version.
COPY --chown=airflow:root requirements.txt /tmp/requirements.txt
RUN pip freeze | grep -iE '^apache-airflow[-a-z0-9]*==' > /tmp/airflow-pins.txt \
    && pip install --no-cache-dir -r /tmp/requirements.txt -c /tmp/airflow-pins.txt \
    && rm /tmp/requirements.txt /tmp/airflow-pins.txt
```

```text title="requirements.txt"
cdo==1.6.1
```

- **Pin every version** in `requirements.txt`, as in a virtualenv task.
- **Do not change** the entrypoint, the command, the `airflow` user or the environment variables
  the runtime sets, such as `PYTHONPATH`. Airflow relies on them to start your task, load your DAG
  code and upload the task log.
- **Do not reinstall what the runtime ships**: never `databaas-utils`, and never the upstream
  `obstore` package, which replaces the preinstalled `obstore-databaas` and breaks storage access.

Check that the build kept the runtime's Airflow version:

```bash
docker run --rm <your-image> python -c "import airflow; print(airflow.__version__)"
```

It must print the Airflow version listed for your runtime on [Python packages](/pipelines/python-packages).

## Push the image

Push the image to the registry your Databaas team gave you, and give every build its own tag, such as
the version of your pipeline or the commit it was built from. Never push a new build under an
existing tag: an image is cached by its tag, so tasks can keep running the old build.

## Point a task at the image

Pass the image to the task through `executor_config`:

```python title="dags/climate_indices.py"
from kubernetes.client import models as k8s
from airflow.sdk import task

CDO_IMAGE = "<your-registry>/climate-tools:2026.10.1"
PULL_SECRET = "<your-pull-secret>"

in_cdo_image = {
    "pod_override": k8s.V1Pod(
        spec=k8s.V1PodSpec(
            image_pull_secrets=[k8s.V1LocalObjectReference(name=PULL_SECRET)],
            containers=[k8s.V1Container(name="base", image=CDO_IMAGE)],
        )
    )
}


@task(executor_config=in_cdo_image)
def compute_indices(path: str) -> str:
    from cdo import Cdo

    ...
```

- **Name the container `base` and set only its `image`, plus the pod's `image_pull_secrets`.**
  Anything else you set there replaces what Databaas configures for the task, and the task can fail
  to start or lose its log.
- **The pull secret** lets the task pull your image from your private registry. Without it the task
  never starts and has no log.
- **Import your image's packages inside the task body.** Airflow parses the DAG file without them,
  exactly as for runtime packages. The `kubernetes` import at the top is fine: Airflow has it.
- **Run every task in a DAG in the image** by passing `"executor_config": in_cdo_image` in the DAG's
  `default_args`. Classic operators such as `PythonOperator` take the same `executor_config`
  argument.

[Validation](/pipelines/development-workflow#what-validation-catches) imports your DAG file, so it
needs the `kubernetes` import too. Add the Airflow package that provides it to the environment your
validation runs in, next to Airflow itself and with the same constraints file:

```bash
--with "apache-airflow-providers-cncf-kubernetes==10.13.0"
```

## Rebuild when the runtime changes

Your image keeps the runtime version it was built from. When your Databaas version moves to a new
Airflow runtime, change the `FROM` line to that version, rebuild, and push the image under a new tag.
An image left on an older runtime keeps an older Airflow and older libraries than the rest of your
pipelines, and its tasks can fail once that Airflow no longer matches.