Incident sheet · SIG-1521 · /infrastructure

Signal
SIG-1521
Filed
Read
3 min · 596 words
Severity
Routine

Data Center UPS Units Identified as Source of Global Grid Anomaly

A 60-second grid frequency sag measured worldwide has been traced to synchronized UPS state-of-charge checks. Ireland's dip quadrupled from 3 to 12 mHz in five years.

Cam feed · SIG-1521INF
Coordinated disturbances haunt grids worldwide. Here’s how researchers solved the mystery.
FeedCoordinated disturbances haunt grids worldwide. Here’s how researchers solved the mystery. — AI-generated

Event log

  1. Ireland's once-per-minute grid frequency dip quadrupled from 3 mHz in 2016 to 12 mHz in 2021; recent dips exceeded 20 mHz.
  2. Data centers consumed 18% of Ireland's grid load by 2022, each running fleets of UPS units.
  3. Researchers installed over 80 power quality meters on the Jülich campus, sampling every three seconds.
  4. A recent study found less than 3% of IoT devices run the latest firmware.
  5. The proposed fix is randomizing UPS state-of-charge checks, a software-only change.

A frequency sag hitting power grids every 60 seconds, on multiple continents at the same moment, has been traced to internet-connected UPS batteries checking their state-of-charge in unison. Researchers at Forschungszentrum Jülich in Germany published the finding after years of modeling and field measurement. The scale is modest today. The trend line is not.

Germany's largest transmission system operator, TenneT Germany, brought the puzzle to physicist Dirk Witthaut. The same once-per-minute pattern appeared in grid data from Europe, Australia, the U.S. and Malaysia. Synchronized dips of this kind can cause blackouts if operators respond incorrectly, and they can damage equipment in power plants — especially hydropower turbines — if left unchecked.

Why did investigators suspect hydropower first?

Hydropower plants have historically driven ultra-low frequency oscillations (UFLOs). Every turbine vibrates, and vibrations at a resonant frequency can magnify to dangerous levels — the wine-glass-before-an-opera-singer effect. Witthaut's team modeled a hydropower control system on the Scandinavian grid.

The hypothesis half-fit. The model produced a resonance peaking once per minute. It could not explain persistence. Mechanical resonances come and go; this signal fired every 60 seconds without fail. The team concluded that anything running on clockwork needs a clock — and started hunting for one.

What did 80 meters on one campus reveal?

The researchers installed more than 80 smart power quality meters across the Jülich campus, which runs medium- and low-voltage distribution lines, transformers, solar panels, a battery energy storage system serving as a UPS, and both high-performance and office computing on smaller UPS devices. The meters sampled frequency, voltage, current and power once every three seconds.

The transformer feeding the battery storage system peaked every minute — exactly like the continental grids. Many stationary batteries, from data center UPSes to grid-scale storage, poll their state-of-charge every minute. Internet connection keeps their clocks nearly synchronized. One check is invisible. Millions at once are measurable.

How big is the measured impact?

Grid operator data supplied the confirmation, and Ireland supplied the numbers. Data centers consumed 18% of Ireland's grid load by 2022, and each facility runs a fleet of UPSes. Between 2016 and 2021, the once-per-minute dip on the Irish grid quadrupled, from 3 milli-Hertz to 12 mHz. Some recent dips exceeded 20 mHz.

Context matters here. Ireland's 50 Hz grid tolerates swings of plus or minus 200 mHz, and the operator prefers to stay within 100 mHz. Twenty mHz is small in isolation. But data centers are not the only load; if other loads sag frequency simultaneously, the collective UPS check could contribute to blackout conditions. Witthaut's team did not claim an imminent failure — they quantified a compounding risk.

What is the fix, and who applies it?

The remedy is a software change: randomize the state-of-charge schedule instead of polling exactly on the minute. Witthaut's team recommends UPS and grid-scale battery manufacturers ship that randomization. No hardware retrofit is required. No vendor has committed publicly to anything, at least in this account.

IoT devices extend the risk beyond UPSes. The researchers detected periodic power fluctuations in devices as different as a smart oven and Raspberry Pi clusters. Patching rates are the weak point: a recent study found less than 3% of IoT devices run the latest firmware. Regulators can mandate changes within their own jurisdictions, but the phenomenon is global and synchronization crosses borders. The mystery is solved; the remediation, on current evidence, will take years and coordination that no body currently owns.

via techcrunch.com (Source)

Filed under

  • data-centers
  • ups-systems
  • power-grids
  • battery-storage
  • iot-firmware
Share this article:

More from Daniel Okafor

Daniel Okafor

Show full bio

Staff writer covering technology and product news in enterprise information technology and the leaders who run at CIO Monitor.

134 articles

The newsroom digest

One email a day with the stories that matter. No noise.

By subscribing you agree to the privacy policy.

Cross-references

‹ Previous articleNext article ›