In May 2003, the University of Wisconsin-Madison’s public time server started taking hundreds of thousands of packets per second — enough to break campus routers. The obvious response was the one for a DDoS: block it upstream and wait for it to subside. A month later it was still running, and bigger.

The traffic was not malicious. It was roughly 700,000 Netgear residential routers, each of which had ntp1.cs.wisc.edu’s IP address compiled into its firmware, each polling that one host once per second until it got a reply.

The forensic part of Dave Plonka’s writeup is the good part:

  • Every packet came from the same UDP source port, 23457 — a hard-coded constant. That single fact turned an anonymous flood into a countable, attributable client population, and let the university filter just that traffic while still answering legitimate NTP requests.
  • Peer universities traced two of the top talkers to physical routers; strings on the firmware images printed both on 23457 port. and the university’s IP address.
  • Netgear reported 707,147 affected units shipped. Plonka counted over 500,000 unique Netgear source addresses querying in one day — an undercount, because NAT rewrites source ports and addresses.

The failure mode generalises well beyond NTP: the bug was not the hard-coded address, it was retrying faster when the server didn’t answer. Hundreds of thousands of clients across the internet synchronised into a flood aimed at one machine. The fix spec the review team produced says it plainly — exponential backoff on failure may, a shorter retry interval on failure must not.

The tail of the story is about how little leverage you have after shipping a bad default. Netgear’s support queue answered 23 days later with a form letter; a direct email to executives got a call back in three days. A recall was judged impossible — the routers appeared to work, few customers had registered them, and almost nobody knew the device did time sync at all. The university’s two network-level options were both costly: anycast the abused address to redundant servers inside WiscNet and eat the residual flooding from “zombie” clients that never receive replies, or stop advertising 4,096 addresses around the server and let the internet return ICMP unreachable.

Worth reading as a template for the whole genre: measure first, assume misconfiguration before malice, and treat a retry policy as a load-balancing decision with global consequences.