On 8 July 2026 a large part of Telstra’s mobile network stopped working — voice calls, SMS, even some calls to the emergency number, plus trains, payment terminals and EV chargers. No attack, no cut fibre, no power loss. One GPS receiver in one chassis in Melbourne came back from scheduled maintenance believing the year was 2006, and talked the rest of the timing hierarchy into agreeing.

Netnod’s Svenne Ebenhag read the independent TAP report Telstra published and turned it into a general lesson about timing infrastructure. The verdict is uncomfortable: the protocol worked. Everything around it did not.

Time is load-bearing in ways that are easy to forget:

  • Most modern 5G spectrum is TDD — one block of spectrum alternating transmit and receive in very short intervals
  • Every cell on the same frequency has to switch direction in step, or the network jams itself
  • The industry traded spectrum efficiency for a hard dependency on every node agreeing about “now”

Whether that dependency was wise is settled. Depending on time is now a design choice, and time deserves to be treated as critical infrastructure.

The 2020 hardware upgrade removed the redundancy that protected against exactly this:

  • The new chassis could not feed a stratum 2 and stratum 3 server in the same box, so the two sites were cross-wired — one usable source per site instead of two
  • In the same change, client/server mode was replaced with symmetric peering, so servers could negotiate their own replacements. The topology became emergent rather than designed
  • Five years later Melbourne started taking time from a node beneath itself. Accurate time still arrived by other paths, so nobody opened a ticket
  • In October 2025 engineers activated the GPS card that had been sitting unused since 2020 and wired it to the stratum 3 server. The alarms stopped

Adding that card promoted the Melbourne server from stratum 3 to stratum 1 — a new source at the very top, the same rank as Australia’s national reference. It became the most authoritative server in the hierarchy for the country’s largest mobile network, installed as a workaround nobody reviewed.

The last ingredient was a well-known property of GPS. A receiver that stays powered keeps counting weeks and never has to figure out which epoch it is in. Powered off, that knowledge is gone, and on restart it falls back on whatever its firmware assumes. The card’s firmware had not been updated in six years, so it placed the date 1,024 weeks — 19.6 years — in the past.

Both of NTP’s defences against bad sources then failed, not by malfunctioning but by having nothing to work with:

  • Lower stratum carries more weight, so the protocol ranked Melbourne highest and steered clients toward the source that was wrong
  • Outliers are voted down, but NTP does not ask whether a date is plausible — only whether sources disagree. Melbourne’s counterpart had been switched off by the same chassis replacement, and everything left was downstream of Melbourne, so the wrong date spread out and came back as agreement
  • Making it worse, the two engineers who did the change were both on mandatory stand-down before the consequences were understood

The report’s recommendations are worth stealing wholesale: classify timing as critical infrastructure, document every change, keep golden configurations under version control checked automatically against what is deployed, give vendor firmware bulletins a named owner, put timing alarms in 24x7 monitoring with consequence-based severity, and prefer documented point-to-point links over self-arranging peering.

The line that stays with me is that architectures like this do not degrade gradually. They work, and keep working, right up until they stop. The same shape can exist quietly in any organisation that acquired its NTP hierarchy by accident.