Xe Iaso (Tigris) wanted a Git server backed directly by object storage. The obvious approach — pretend object storage is a filesystem and let stock git talk to a shim — works on toys and falls over on real repositories. This post is the performance analysis of why, and the on-disk format he built to fix it.

The core mismatch is timing, not throughput:

  • Filesystem reads are ~10 nanoseconds at most; an object-storage round trip is ~10 milliseconds at minimum
  • That is “at least a million times slower because of how reality works”
  • Git writes a file and immediately reads it back to compute a hash, which object storage cannot do — you cannot GetObject something whose PutObject has not finished
  • Packfiles are mmapped, so the kernel does the paging for you. That model does not exist over HTTP

Then a second problem, which is the one that forces a new format:

  • A packfile index tells you the decompressed size of an object but not the compressed size
  • Without the compressed size you cannot build a Range request for one object
  • Objects sit at arbitrary offsets inside one large packfile, so there is no way to ask for exactly the bytes you need

His replacement is modeled on CD .bin/.cue backups, of all things: objects are concatenated into a container (normally up to 128 MiB), with metadata in a separate fixed-width binary cue sheet — a 16-byte header, then 58-byte records carrying hash, type, compression, offset, compressed and decompressed sizes, and delta base. Record N is at 16 + N*58: one seek, one ranged GET. Deltas became their own objects instead of being appended to their base, and compression moved from zlib to zstd.

The read path is a four-tier ladder: the container downloads to a temp folder in the background while ranged GETs race that download for objects at the far end. Numbers against three repos, run on Wi-Fi on purpose: push wall time 8.7s → 2.2s, 3m29s → 14.3s (14.6x), 2m13s → 26.5s; S3 requests 231 → 18, 9,236 → 30, 3,324 → 136. Clones: 11.8s → 2.6s, 3m23s → 54.4s, 2m23s → 1m22s.

The argument for rolling your own format at all is the interesting part: Git is distributed, so every clone holds the full history. If the hand-rolled format turns out to be a mistake, you push it again. That is a narrow escape hatch most storage layers do not have.

Two things worth carrying away. First, the format was not replaced until measurement showed packfiles were the bottleneck — the filesystem shim worked “ok, I guess” and the numbers said what was actually wrong. Second, the caveats are stated plainly: no auth, no authz, no API, no rate limiting, packfiles accumulate forever, and exposing it to the internet means anyone who can connect can push or pull anything. A format designed for a local filesystem does not survive network round trips; it has to be redesigned for them.