Monitoring and running pipelines
See whether your pipelines ran, read their logs, and trigger a run yourself.
Opening Pipelines opens the Airflow web interface. A pipeline (Airflow calls it a DAG) is a job that loads data into tables — see authoring pipelines. Visibility is fleet-wide: if you can open Pipelines, you see every pipeline in the deployment, not a subset. An empty list means none are deployed.
Prerequisite
can_view_pipelines opens Pipelines and reads runs, logs and code. can_run_pipelines adds
trigger, clear, mark and pause, and implies view. Missing from your sidebar means you hold neither.
Everything below is standard Airflow; the Airflow UI docs cover each screen.
What to look at
- Pipeline list — the landing screen: every pipeline with its recent runs, schedule, and paused state.
- Runs — select a pipeline for its run history; each run has a state (success, running, failed) and start time.
- Task states — within a run, each task’s state, in a grid and a graph showing dependencies and which task failed.
- Logs — open a task instance for the log it wrote. A failed run tells you what went wrong here.
Operating a run
With can_run_pipelines you operate every pipeline in the deployment:
- Trigger a run on demand (unpause the pipeline first if it is paused).
- Clear a task or run so it runs again.
- Mark a task or run as success or failed.
- Pause and unpause a pipeline’s schedule.
Stop a runaway pipeline
A run that is taking too long or doing the wrong thing keeps going until you stop it. With
can_run_pipelines:
- Pause the pipeline with its toggle, so the scheduler starts no new runs.
- Mark the running run, or its running tasks, as failed. Airflow stops the tasks that are still running.
Pausing alone does not stop a run that has already started. Two limits also end a task on their own:
- A task that uses more memory than it is allowed is stopped, and fails.
- A task with an
execution_timeoutfails once it runs longer than that. Set one on any task that calls an external source; see authoring pipelines.
What needs an administrator
These need can_admin:
- Deleting a pipeline. This removes its run history irrecoverably.
- Airflow’s Connections, Variables, Pools, Config and Backfills, and changes to Assets.
Airflow’s audit log, users, roles, providers and plugins screens are not available to Databaas users; people and permissions are managed in the portal, see managing access. A permission change applies within about ten seconds, without signing in again. If a control is refused, check what it needs against the permissions reference.