Reconstructing large real-world environments as 3D Gaussian splats runs out of GPU memory quickly. Ka Heng Shiu hit that wall scanning Hong Kong villages and buildings for The Lost Metropolis, turned it into a dissertation, shelved it for a year and a half, and came back with ABCD — Alpha-Composited Block Coordinate Descent — presented at SIGGRAPH this year.

Why the obvious approach fails:

  • Split the scene into regions, train each one separately, merge them — and every region is optimised blind to the rest.
  • Split a room in half and the left optimizer will still place Gaussians to explain pixels that belong to the right half. After merging, those mistakes appear as floating geometry and seams.

How ABCD avoids it:

  • Train one spatial chunk at a time, with the rest of the scene frozen but still visually present.
  • For each camera, the inactive Gaussians are pre-rendered into a foreground and a background RGBA image; the active chunk renders between them and the three layers are alpha-composited.
  • Only the active chunk stays in VRAM. The remainder has collapsed into images whose memory cost does not scale with the number of Gaussians they represent.
  • The optimizer still sees how its chunk occludes, blends, and sits in front of or behind surrounding geometry, so those relationships are preserved during training instead of stitched back afterwards.

What it costs today:

  • GPU memory is traded for system RAM, disk, and training time. RAM caches views for cameras that can observe the active chunk; disk stores the larger set of pre-rendered camera/chunk combinations.
  • Peak VRAM is set by chunk size and render resolution, not total scene size. Even a scene that only barely overflows can roughly halve its Gaussian VRAM by splitting into two chunks.
  • Visibility testing is currently frustum-only. Occlusion-aware culling, compression better than lossless PNG, and smarter caching strategy are named as clear wins that have not been evaluated yet.

Evidence and caveats:

  • Tests used Mip-NeRF 360 scenes, some split into almost 30 chunks to create maximum opportunity for partitioning errors. Reconstructions showed very little visible artifact and converged similarly to baseline splatting.
  • The test scenes are small, so fixed overheads partly mask the memory advantage; city-scale captures would demonstrate it more clearly.
  • ABCD is not specific to Gaussian splatting — anything spatially partitionable and recombineable by alpha compositing should fit, including NeRF-like representations and triangle splatting.

The most interesting part is what happens at scale. Once memory stops being the constraint, the bottleneck becomes scheduling many local optimisation problems: switching between chunks, refreshing cached context, revisiting regions. Shiu’s reframing is that the question stops being which GPU is big enough and becomes how to use whatever compute is available — single-GPU today, but structurally friendly to heterogeneous machines and workers added or removed dynamically.

For scan-heavy pipelines — photogrammetry, digital twins, virtual production sets, large game environments — this is an out-of-core training pattern that keeps the whole scene in context while bounding VRAM. The paper is arXiv 2608.27735 and the code is public on GitHub.