DataHaskell’s dataframe library now writes Parquet files — D.writeParquet with sensible defaults, writeParquetWithOptions for control. The stated motivation is interop: Haskell’s data-serialization options had been CSV, JSON, or a raw ByteString, and CSV and JSON get slow and awkward once the volume grows, while homegrown formats talk to nothing else. Parquet trades simplicity for compressed storage and, more usefully, for reads that skip data no one asked for.

The format primer is short. A Parquet file is a series of row groups, then metadata at the end recording the offset and size of each row group and column chunk:

  • A row group is a set of column chunks that all hold the same number of rows
  • A column chunk is a series of data pages: page metadata, definition and repetition levels, then encoded and compressed values
  • Encoding is chosen per page, compression per column chunk
  • Statistics and bloom filters in the metadata are what let a reader prune whole row groups

The implementation walkthrough is the part worth keeping, because it is really a list of forced trade-offs:

  • Size targets have to be best-effort. Columns encode at different densities, so a byte target implies a different row count per column — but all column chunks in a row group must agree on row count. The writer iterates in row batches and checks buffered size afterwards, which is why row groups are integer multiples of batchRows and pages integer multiples of subBatchRows.
  • Sub-batching at page level cuts IORef bookkeeping after every write, for a claimed significant speedup.
  • Buffers are pinned ByteArrays so they can be handed out as Ptr Word8 on flush. The cost is that Data.Primitive’s grow is unusable: growth means allocate, copy, and let the old array be collected. The author reasons explicitly about heap fragmentation (one pinned object can pin a whole 4 KB GHC block) and decides the buffers are large enough and grow rarely enough for it not to matter.
  • The core loop is an effectful fold over the dataframe — advance a row cursor, write a batch, flush the row group when buffered size passes the target — with a scratch buffer assembling values, definition levels and page metadata from separate buffers before they land in the column chunk.
  • The flush function led to the OS. Writes land in the page cache; when the dirty-page ratio crosses a kernel threshold the writer is throttled while writeback catches up, and because row groups flush synchronously the process just waits. Fix: fewer, larger write(2) calls — 256 KiB chunks, chosen after dd experiments — with O_DIRECT named as the deeper path.

The scope statement at the end is unusually honest, and it is the reason to read the post: Snappy and uncompressed only, plain encoding only, flat schemas only, single-threaded, no statistics or bloom filters yet — and every buffer in memory, so an entire row group must fit in RAM. That last one has a planned fix (two-pass with on-disk temporaries). Note the shape of the gap: the features still missing are exactly the ones that make Parquet fast to read, so files written today are bigger and less prunable than the format allows. Correct first, prune later.