Uber developer-platform engineers Preetam Dwivedi and Manjari Akella explain how their merge queue adapted to AI-driven code volume in this AI Engineer talk.

Faster code creation exposed the merge bottleneck

  • Two changes can pass tests separately and break the main branch when combined.
  • SubmitQueue tests likely future combinations, separates independent build targets, and lets small changes bypass slower ones.
  • The speakers report arrivals rising from roughly 90 to 130 requests per hour as AI-generated changes grew.
  • The 95th-percentile landing time rose from about 30 minutes to three or four hours while the system kept trunk green.

Three interventions changed the queue

  1. Admission gating: keep expensive, broadly overlapping changes out of peak-hour contention to protect smaller changes.
  2. Deferrals: persist those expensive requests and release them automatically when permitted, removing manual resubmission but retaining the wait.
  3. Conflict relaxation: rank dependency overlaps and drop weaker links so more groups of changes can build independently.
  • Classify expense by affected build targets, rather than source-line count alone.
  • Assign conflict relationships when a request arrives, preserving existing speculative builds instead of repeatedly reshaping the whole graph.

More parallelism changes the guarantee

  • Admission gating reportedly increased the land rate from about 130 to 160 per hour.
  • Conflict relaxation then brought it to roughly 250 or more, but allowed some untested combinations to land together.
  • The latest main branch can therefore break; the design needs explicit post-merge checking and recovery.

Keep a separate verified-good state

  • Stovepipe validates landed commits and maintains a verified-main pointer distinct from the latest trunk head.
  • Failed validation leaves that pointer at the last known-good state; a later passing state advances it.
  • On-call engineers currently prepare fixes or reverts, which pass through SubmitQueue before Stovepipe validates them again.
  • The final recap reports roughly 300 lands per hour at peak and a return to 30-minute 95th-percentile landing times.

What remains planned

  • Batch related changes to reduce redundant builds.
  • Track verified state per service or project so an unrelated failure blocks less work.
  • Automate more repair and revert actions to shorten recovery time.

“Some combinations of changes will land untested.” — Manjari Akella

Caveat: These are Uber’s reported operating results. The talk does not quantify the post-merge failure rate or the added on-call burden.