Querying data

For the first ten rows end to end, see the quickstart.

Load the catalog

load_user_catalog from the databaas_auth helper returns a pyiceberg catalog scoped to your grants — it fetches a token for you, so there are no shared credentials to configure.

from databaas_auth import load_user_catalog

catalog = load_user_catalog()

table = catalog.load_table("<namespace>.<table>")

Use the warehouse, namespace and table name from the Data Catalog breadcrumb — see browsing the catalog. For a non-default warehouse, name it: load_user_catalog(warehouse="<warehouse>"). A table you have no grant for raises a permission error here — the same boundary the catalog preview enforces, see data permissions.

Scan rows

table.scan() reads rows; convert to Arrow, then pandas to view it:

table.scan(limit=10).to_arrow().to_pandas()

Everything you scan loads into your session’s memory, so an unbounded scan of a large table exhausts it. Narrow first — selected_fields picks columns, row_filter picks rows, limit caps the count:

arrow_table = table.scan(
    selected_fields=("account_id", "amount_eur", "channel"),
    row_filter="channel = 'mobile'",
    limit=100_000,
).to_arrow()

See the pyiceberg documentation for the full scan API.

Query a slice with DuckDB

Register an Arrow table as a DuckDB view and query it:

import duckdb

con = duckdb.connect()
con.register("tx", arrow_table)

con.sql("""
    SELECT channel, count(*) AS n
    FROM tx
    GROUP BY channel
    ORDER BY n DESC
""").show()

This runs entirely in your notebook, so it only sees the rows you scanned in. See the DuckDB documentation.

When to switch to SQL Lab

Notebooks pull data to your code — right for exploration and tables that fit in your session. When a table is bigger than your session, or you want a query to run next to the data and return only the result, use SQL Lab, on Trino.

Refresh an aged-out catalog

pyiceberg holds the token the catalog was built with, so a catalog left open long enough starts failing with a 401. Re-call load_user_catalog() for a fresh one. If that still fails, your sign-in has expired — sign out of Notebooks and back in.