Edit: It's interesting the GPU's are causing issues on the grid before they cause issues with the data center's power.

I suspect it's more of a problem for the data center's energy bill. My understanding is that large electric customers pay a demand charge in addition to the volumetric charge for the kWh's use at whatever rates given time of use / wholesale rates. The demand charge is based on the maximum kW used (or sometimes just the connection size) and may also have a penalty rate if the power factor is poor. Smoothing over small duration surges probably makes a lot of things nicer for the rate payer, including helping manage fluctuations from the utility.

There's probably something that could be done on the individual systems so that they don't modulate power use quite so fast, too; at some latency cost, of course. If you go all the way to the extremes, you might add a zero crossing detector and use it to time clock speed increases.

Yes its a problem for the grid and the power companies don't allow large clusters to oscillate their power like this. The solution that AI have to do during their training big runs is to fill in the idle time on the GPUs with dummy operations to keep the power load constant. Having capacitors would be able to save on power usage.

Those loads aren't nearly as intermittent. Your furnace likely runs for tens of minutes at a time. These datacenters are looking at second-to-second loads.

Drawing high intermittent loads at high frequency likely makes the utility upset and leads to over-building supply to the customer to cope with peak load. If you can shave down those peaks, you can use a smaller(cheaper) supply connection. A smoother load will also make the utility happy.

Remember that electricity generation cannot ramp up and down quickly. Big transient loads can cause a lot of problems through the whole network.

They service 'spot demand moderation' as an extension of UPS and power smoothing. In this case it's flattening out spikes to smooth slopes.

Pretty much. From the article:

> Another solution is dummy calculations, which run while there are no spikes, to smooth out demand.

Paraphrasing this[1]?

[1] https://github.com/pytorch/pytorch/pull/132936/files#diff-98...

In addition to what you said, nothing is forcing or even encouraging anyone to use lithium-ion batteries in fixed service, such as a rack full of computers.

Eh, I think part of the problem here is the speed of load switching. From the article it looks like the loads could generate dozens to hundreds of demand spikes per minute. With most battery operated loads that I've ever messed with we're not switching loads like that. It's typically 'oh a fault, switch to battery' then some time later you check the power circuit to see if it's up and switch back.

This looks a whole lot more like high frequency load smoothing. Really it seems to me like a continuation of a motherboard. Even if you have a battery backup on your PC you still have capacitors on the board for voltage fluctuations.

in a properly designed install you can actually use the compressors and fans for smoothing load spikes. won't be much, but why not.

edit: otherwise I'm not getting what the entire article is about. it's as contrary to what I know about datacenter design as it can get.

it's.. just wrong.

that would be a blackhole bomb.

You might need more memory for this secondary training workload. But yeah, donating/selling the 'network' time for high-intensity, low memory footprint workloads (thinking number crunching, monte-carlo stuff, maybe brute-force through a series of problems...) might end up making sense.

If the cores go idle (or just much less loaded) in between steps because they're waiting for network communication that would cause the problem.

Bad input pipelines are a big cause of spikiness - you might have to wait a non-trivial fraction of a second for the next batch of inputs to arrive. If you can run 20+ training steps per second on. adecent batch size, it can take some real engineering to get enough data lined up and ready to go fast enough. (I work on audio models, where data is apparently quite heavy compared to images or text...)

Oh man, I really, really wish that you hadn't said this and also that you were wrong.

YCW27

Dummy calculations aren't even needed, if you allow the LLMs to pre-compute on the given context before inference:

https://arxiv.org/abs/2504.13171

It should be noted that this type of inference is less useful on time-sensitive tasks, but most tasks truthfully don't require such time sensitivity (there exists slack time between when the task is given & when questions are asked).

>implying its not already happening