Connect with Iceberg

Iceberg REST · Delta Lake · Parquet

Resources & endpoints

Let DuckDB or PyIceberg discover tables through the Iceberg REST catalog instead of naming S3 paths yourself, or call the catalog directly with the REST example. The catalog is free and needs no API key. The table files are still read over S3 with your Pinax Project ID and API key, exactly as in Connect with S3.

A namespace can exist before its tables are published. Run the listing step first and query only a table it returns; an unlisted table fails with a not-found error. Until your table is listed, read it with Connect with S3.

Iceberg connection settings
Catalog URI	https://iceberg.s3.pinax.network
Catalog authentication	None
S3 endpoint	https://datasets.s3.pinax.network
Region	us-east-1
Addressing	Path style
S3 credentials	Project ID as access key ID, API key as secret access key

Loading your project…

DuckDB 2.0 or newer, iceberg and httpfs extensions

1-- DuckDB 2.0 or newer.
2INSTALL iceberg; LOAD iceberg;
3INSTALL httpfs; LOAD httpfs;
4CREATE SECRET pinax_s3 (
5  TYPE s3,
6  KEY_ID '<your Pinax Project ID>',
7  SECRET '<your Pinax API key>',
8  REGION 'us-east-1',
9  ENDPOINT 'datasets.s3.pinax.network',
10  URL_STYLE 'path',
11  USE_SSL true
12);
13 
14-- The catalog is anonymous; S3 reads use the secret above.
15ATTACH '' AS pinax (
16  TYPE iceberg,
17  URI 'https://iceberg.s3.pinax.network',
18  AUTHORIZATION_TYPE 'none',
19  ACCESS_DELEGATION_MODE 'none',
20  READ_ONLY
21);
22SHOW ALL TABLES;
23 
24-- Query a table that SHOW ALL TABLES lists. S3 reads bill your project.
25SELECT block_num, block_id, timestamp
26FROM pinax.eth.blocks
27WHERE date = DATE '2020-01-01'
28ORDER BY block_num
29LIMIT 10;

PyIceberg with PyArrow file IO

1# pip install "pyiceberg[pyarrow]"
2# Set these in your shell, not in a shared source file:
3# export PINAX_PROJECT_ID='<your Pinax Project ID>' PINAX_API_KEY='<your Pinax API key>'
4import os
5from pyiceberg.catalog import load_catalog
6from pyiceberg.expressions import EqualTo
7 
8# The catalog is anonymous; S3 reads use your project credentials.
9catalog = load_catalog(
10    "pinax",
11    **{
12        "type": "rest",
13        "uri": "https://iceberg.s3.pinax.network",
14        "auth": {"type": "noop"},
15        "header.X-Iceberg-Access-Delegation": "",
16        "py-io-impl": "pyiceberg.io.pyarrow.PyArrowFileIO",
17        "s3.endpoint": "https://datasets.s3.pinax.network",
18        "s3.region": "us-east-1",
19        "s3.access-key-id": os.environ["PINAX_PROJECT_ID"],
20        "s3.secret-access-key": os.environ["PINAX_API_KEY"],
21        "s3.force-virtual-addressing": "false",
22    },
23)
24print(catalog.list_namespaces())
25print(catalog.list_tables("eth"))
26 
27# Load a table that list_tables returns. S3 reads bill your project.
28table = catalog.load_table("eth.blocks")
29rows = table.scan(
30    row_filter=EqualTo("date", "2020-01-01"),
31    selected_fields=("block_num", "block_id", "timestamp"),
32    limit=10,
33).to_arrow()
34print(rows)

curl, catalog metadata only, no API key

1# Anonymous, read-only metadata.
2CATALOG=https://iceberg.s3.pinax.network
3curl --fail --silent --show-error --max-time 5 "$CATALOG/v1/config"
4curl --fail --silent --show-error --max-time 5 "$CATALOG/v1/namespaces?pageSize=100"
5curl --fail --silent --show-error --max-time 5 "$CATALOG/v1/namespaces/eth/tables?pageSize=100"
6curl --fail --silent --show-error --max-time 5 "$CATALOG/v1/namespaces/eth/tables/blocks"
7 
8# No Authorization or X-Iceberg-Access-Delegation header is needed.
9# Metadata references S3 objects. Use your project credentials to read them;
10# never treat a metadata-location or manifest URI as a public download link.

The catalog never receives or returns S3 credentials. Each catalog request is listed in the Iceberg REST API reference in the sidebar.

Connect with Iceberg

DuckDB example

PyIceberg example

REST example