A downed power line exposed a growing AI data center problem. How to fix this.


There was a power outage outside of Washington this week. Normally, the network only needs a few seconds to recover from such an event. But it took more than 10 minutes, because more than 3 gigawatts of data centers stopped drawing power at about the same time.

According to reports, the incident caused a spike in PJM’s network from Northern Virginia to Chicago. Ting Laboratoriesa startup running an IoT sensor network from people’s electrical outlets.

Although the incident did not cause a power outage, it did cause lights to flicker in the region. The incident demonstrated the impact data centers can have on the network — an outcome experts say will happen more often.

Located in PJM, Northern Virginia is home to the highest concentration of data centers in the world.

“It’s the canary in the coal mine,” Ricardo de Azevedo, CTO ON. Energytold TechCrunch. Such incidents involving large loads such as data centers are “happening more and more,” he said.

The incident echoes one that occurred two years ago, also on the PJM grid, and could be a harbinger of larger incidents if data centers are not set up to handle power outages more gracefully. PJM Interconnection operates networks from New Jersey to Illinois and serves 67 million customers, making it the largest network operator in the United States.

When a power line went out this week, it prompted data centers to switch to backup power, and about 3.1 gigawatts of load disappeared in about 30 seconds. PJM data. The network recovered somewhat, but after a short time additional loads fell. PJM’s grid had an additional 3.49 gigawatts of electricity at its peak. It took another 11 minutes before it stabilized. Disconnected data centers accounted for about 3% of PJM’s total demand at the time. according to to Reuters.

A few percent may not sound like much, but the power grid must operate in a state of perfect balance, with supply and demand closely matched. Otherwise, the voltages may decrease or increase. A network and the devices connected to it can tolerate small fluctuations, but if these fluctuations become too large, they will cause faults within the network or in individual objects, causing them to disconnect.

When data centers in Northern Virginia experienced surges caused by a faulty power line, they switched to backup power that took the load off the grid. As more data centers switch, they take more load off the network. What started as a relatively small drop in supply turned into an even bigger drop in demand, causing supply to surge and the light bulbs to blink.

Most data centers make decisions in a matter of seconds, and this week’s outages are no different. Ali Zain Banatwala, chief market model specialist at the Independent Power System Operator, told TechCrunch that when the voltage drop hit them, they all decided to disconnect within seconds of each other.

“We have to find a way for these loads that are next to each other to either disconnect or reconnect in sequence,” he said. A more streamlined process will allow network operators to develop more robust procedures in advance.

Alternatively, data centers can be built to absorb disruptions and not turn their backs on them. One startup, ON.Energy, is working on a product to help data centers and the grid get through events like the one that happened this week.

The company has developed an uninterruptible power supply for the entire data center campus, which includes not only servers but also refrigerators and other equipment. The company essentially hides its data center behind a bank of batteries connected to sophisticated power conversion equipment. The entire network “sees” a consistent, well-designed load rather than peaks and valleys from each individual part of the data center. ON.Energy’s system allows data centers to ramp up and down compute workloads, including AI training, without disrupting the network.

Perhaps more importantly, it means that data centers can absorb power surges from the grid. Instead of disconnecting from the grid, the ON.Energy system can use any extra power to charge its batteries, and if the flow drops, the system can send power to the servers. Plus, it can follow the lead of the grid within milliseconds, avoiding kinks or surges like the ones that caused problems for PJM this week.

ON.Energy is currently installing a total of 3 gigawatts of systems across four different data center campuses, de Azevedo said.

Network managers also woke up to the problem.

De Azevedo said ERCOT, for example, would require large loads such as data centers to “get through” the outages.

The clock is ticking though. This week’s massive outage was twice as large as a similar event in 2024, when 60 data centers were simultaneously disconnected from the grid, draining 1.5 gigawatts of load. At the time, data centers accounted for about 6% of PJM’s load. according to For Synapse Energy Economics. By 2040, they are expected to make up 24%. If the problem is not solved soon, the situation may worsen.

When you purchase through links in our articles, we may earn a small commission. This does not affect our editorial independence.



Source link

Leave a Reply

Your email address will not be published. Required fields are marked *