§1 Chunk 00 — DATA
File-native at scale — the DATA chunk carries the workflow's source data by reference or by value, so a multi-gigabyte Parquet behaves like a small one. Designed, not shipped.
DATA0x0100
In development. This format is designed, not shipped. Nothing to install yet — join the waitlist.
The DATA chunk carries the workflow's source data — by reference when the source is a file, by value when it's small. dat0 reads a Parquet, a CSV, or another data file directly where it sits on disk: there is no load step, no upload, no import. You point at the file and start asking questions.
Why it matters
The size of the file stops being the thing you plan around. A 12-gigabyte Parquet is meant to behave like a 5-megabyte CSV: you scroll it, filter it, and aggregate it at the speed you'd expect from something small. The work happens on your machine, against the bytes already on your disk — nothing leaves, nothing waits on a network round trip. In the concept story, the DATA chunk references a 12.4 GB Parquet source.
Reading a file directly
A query names the file as if it were a table. No schema to declare first, no ingest job to run:
SELECT count(*) FROM 'events.parquet';The same shape works for a slice or a rollup — group, filter, and order straight over the file:
SELECT country, count(*) AS events
FROM 'events.parquet'
GROUP BY country
ORDER BY events DESC;The file header
Every artifact opens with a fixed header — the file magic, a draft version, the chunk count, a flags word, and a pointer to the chunk directory. The DATA chunk follows, opening with a magic of its own:
The header fields, field by field:
| field | type | notes |
|---|---|---|
| magic | 4 bytes | "DAT0" — the file identifier (the Parquet PAR1 / PNG signature convention) |
| version | u16 LE | format version — 1 (draft); bumps on any breaking change |
| chunk_count | u16 LE | chunks after the header — five in the draft anatomy |
| flags | u16 | bit 0 = sealed — a sealed artifact is complete and replayable |
| reserved | 2 bytes | zero — readers ignore, writers zero |
| directory_offset | u32 LE | absolute offset of the chunk directory (the ZIP central-directory convention) |
Every chunk header is the same fixed shape — magic, payload length, encoding, reserved — so the reader contract never changes from chunk to chunk. The DATA chunk's payload is the source reference itself: an encoding byte selecting parquet-ref, csv-inline, or pg-ref, plus the source-footer fingerprint. Lineage entries in chunk 04 address ranges of this chunk by that fingerprint and a byte range, so provenance survives rewrites of the artifact — §5 picks up that thread.
Continue to §2 Chunk 01 — TRANSFORMS.
§0 Introduction
What dat0 is, the artifact metaphor, and how the .dat0 format spec is organized — §0 through §6. Designed, not shipped.
§2 Chunk 01 — TRANSFORMS
A reproducible workflow — every transform is sealed into the file with its SQL — and compute portability — local by default, cloud only when you choose. Designed, not shipped.