Texas is the perfect example of how not to run an electrical grid by not allowing other states to assist in an emergency.
In this case it's not really Texas 'not allowing' other states to help but the other states not allowing Texas. Conceivably federal law could be updated to remove those regulations and Texas would absolutely connect to the interstate grid at that point.
That, of course, ignores the fact that those regulations are in place for a reason. Texas refuses to play by the rules, and the impact of that is that they don't get help when it's important. It is unfortunate, but a direct consequence of the choices they made.
Super abusive. Let’s do away with safety systems that literally save human lives. Heck Texas doesn’t like them so let’s do away with them for the whole nation? How many people died during the last couple heat and cold induced grid outages in Texas? I lost count after a couple dozen. But those people were weak or poor anyway right? Texas strong!
No, it isn't. Any decent datacenter will have on-site generation in event of power grid failure, anyway. When I was an intern, the company I worked for would routinely go off grid during the summer at a call from the electric company. The electric company actually gave us significant incentives to do so, because us running on our own 12MW generator was effectively like the grid operator farming out a 12MW peaker unit.
The power company has a long list of who has backup power. I know of one factory where the generator was installed in the 1920s on a boiler from the 1880's - it is horribly inefficient, but the power company still gives the owners incentive to keep it working because for 4x the normal cost of power and 12 hours notice that generator can run the entire town it is in, which they do every 5 years when things really go wrong with the grid.
What is your definition of regularly, and what qualifies as getting it fixed? I know lots of places that had things scheduled, but on the day of, something "came up" that the test was pushed. I've seen others where they tested by only firing up the generator, but didn't actually use it to power the facility. I've also seen repair tags that sat "unlooked" at for years.
Not every facility is managed/financed the same for such a blanket statement as yours.
Ok!
Agreed, the datacenters need to be extremely durable. What's more durable than proving you're able to withstand a power outage event? The grid does go down from time to time; they need to be ready to handle it. That's not a Texas-only kind of thing; power outages happen all over the US.
If the datacenter can't handle the outage that was announced as a probability ahead of time, they have no business running critical applications.
> We live in the digital age, so almost everything depends on data
> As in all mechanical devices, generators have a tendency to either break down(Let's not forget about associated switching gear.) or become subject to maintenance cut backs instituted my buffoons in management
> We cannot depend on human errors, we can depend on the electric grid, if properly handled and maintained
> Depending on generators just adds another link in the failure chain.
> Texas is the perfect example of how not to run an electrical grid by not allowing other states to assist in an emergency.
Their "free market" real time power auction makes no allowances for long term concerns like reliability. Any provider who spends money to address these sort of issues is immediately priced out of the market.
Some things are too important to leave to the "free market".
It's a holier than thou thing for the other states. "Free market" buying from Texas when they need it, when they don't it's "fuck the free market" and cut texas off via legislation and refuse to provide the same assistance that Texans provided.
> without a problem
That doesn't really line up.
What is the alternative, have a state imposed monopoly with a single power company like PGE who I would not see as an ideal operator either. Same can be true for a lot of other similar generators across the company.
You’ll probably bring up the winter storm outage which is inexcusable but their neighbor to the north SPP had very similar failings in being prepared and only faired better because they have interconnects.
Texas has had some of the fastest adoption for wind and solar. It is far from perfect but I also think there is benefit to having multiple generation companies supplying to the grid. You have companies with different expertise and perhaps innovation.
The government can structure a public market in many different ways (they do this in many aspects of the economy). It’s not limited to real time auction vs single provider.
Except then they’d be subject to regulations. And we can’t have that now can we?
Why so salty? There are n>0 things the federal government does that I disagree with. On a subject I know nothing about why would I assume that Texas is wrong to avoid it's rules?
When I lived in the capital, we got our power from the Lansing Board of Water and Light, which is 100% publicly owned. Their rates are still some of the lowest in the Midwest. The main downside is that until recently their main energy source was coal. (We used to live downwind of the smokestacks. You couldn't smell it, but lung cancer was definitely in the air.)
It does the citizens of TX a disservice and has resulted in deaths of many of them.
EDIT: s/rooting/rooted/
It’s really cheap. Ive done it for a grand total of $2000 most of which was to get a real beefy generator so I could just power my whole house instead of only a few circuits. Most people think an installed appliance like Generac or some battery/solar option are the only options, and those often run $15-20k and up. We don’t always need instant switchover, but if it doesn’t come back on in a couple hours I pull out the generator.
Apartments and other MF properties will need to approach it differently, but I don’t think it’s possible and reasonable to just let the property owners take ultimate responsibility. After all, most my outages aren’t grid failures they’re some localized wire/transformer issue that is unavoidable.
So my general albeit cold sounding response is “Doesn’t matter.” We should have the expectation that it’s owner responsibility first. After that, we can devise subsidies and such to ensure everyone can retrofit their house. There’s a ton of levers to work with once you admit that the grid and power transmission isn’t some god like thing that never fails
You can’t hinder progress because someone can’t afford it. They maybe did have the money if it meant a few bucks a month on their bill, but they were never told this risk existed, we all thought we lived in a modern enough country that we would never be without power for an entire week. But we also have never seen freezing temperatures for a solid week either, not in anyone I knows lifetime including some 90 year olds.
Once I know the problem exists, I’d rather spend the $2k and have a solution at hand than take on the full system costs of winterizing/prepping for a once in a century(?) snow storm. That would perpetually make my energy cost go up by 10% or more. It’s the smarter solution with better ROI if people DIY the contingency.
Are you talking about something like a 7.2kW portable with a 60A manual transfer switch? I could see that costing around $2000, which is substantially cheaper than a Generac. I found a portable Kohler with 7.2kW for ~$1500.
Instead of a transfer switch, you could shut off the main circuit breaker or pull the meter and backfeed the panel through a 60A 2P breaker, that would save some dollars.
Just make sure that you disconnect from utility power before backfeeding and be absolutely certain to disconnect the generator before switching back to utility power, you don’t want to find out what happens when a generator isn’t in sync with the utility frequency :) Rapid unintentional disassembly, lol.
My generator is the biggest one harbor freight sells. Might not be that beefy but I also don’t need much electricity during these events. I wouldn’t run laundry or my central AC but it’s enough to keep my pool pump running (we don’t winterize here, run nonstop during freezes as they are short enough the pool water never drops enough). Also, this model runs of natural gas which is a big plus for me so I don’t have to keep up with fuel. Keeping fridge running and furnace running is my highest priority but since all my lights are LED I don’t have to think about them. I’m giving the harbor freight one a shot first, my hope is that it last a long time since I’m putting little time on it. If longevity is ab issue I’d probably spring for a kohler or Honda next time.
I may add a soft start kit to hvac as last summer we had some outages long enough to be uncomfortable, twice cost us our fridge contents which is expensive and annoying. But primarily, winter protection is highest priority for me as the risk is highest
Lack of reliability is not innovation. This is something the Texas "real time power auction" has created and continues to promote.
Measures like the one the governor just signed are bandaids meant to cover up the fundamental lack of reliability.
There is nothing left to innovate in terms of reliability. The solution really is very simple --- build additional capacity to cover worst case scenarios.
This can't happen if providers are forced to continually offer a rock bottom price or lose out in the auction.
Some things are too important to leave to the "free market".
What is the most cost-effective and reliable way to administer something that every single person in a country requires? Taxes and the public sector.
It's not rocket science.
It's just like Visa and other payment processors. What do you think that 3% on every single transaction is? That's a tax. The difference is you're being taxed and you're buying someone's third private jet. If everyone is gonna be taxed regardless, just nationalize it. At least then we won't have to pay the burden of profit.
> ...no allowances for long term concerns like reliability. Any provider who spends money to address these sort of issues is immediately priced out of the market.
...is a bit of an exaggeration.
When Texas has those cold-snap/freeze days a few years ago, wholesale rates went up to $9,000 per megawatt-hour. So $9/kWh. Wholesale.
A large amount of energy suppliers went out of business because they didn't properly hedge for such an event.
You can bet those who are left have started to react to market price signals like that. Whether it's through financial engineering or boots-and-poles engineering is a fair discussion to have, but to say "no allowances for reliability signals" is a bit disingenuous.
When someone says "It's all really very simple...", it's almost certainly not.
Blaming the "free market" is an ignorant kneejerk reaction.
There is a whole discipline revolving around designing the incentives for electricity markets - countries pay for consultants from companies like NERA Economic Consulting to create or improve their electricity markets to achieve the goals needed by that country.
https://duckduckgo.com/?q=wholesale+electricity+market+desig...
The problem is that wholesale electricity markets need to be designed so that the incentives of the market participants are aligned with desired outcomes. Use game theory so that selfish participants are rewarded for good outcomes in the market, and are penalized for bad outcomes in the market.
When there is a failure (e.g. Spain) the responsibility lies at the feet of the regulators.
Unfortunately after a failure we all want to blame someone, and it is easiest to blame the market participants or the market.
Read how Enron manipulated the market to the detriment of California and its residents. The answer isn't to Blame Enron. Blame the people that set up shitty incentives.
The big issue is ensuring that large generators/retailers don't twist the market rules to their advantage or obtain regulatory capture. The large players tend to have a knowledge advantage and are very experienced at manipulating their regulatory authorities.
Of course, it is easy enough to deal with curtailment for many services. But, it should be on the table, either way.
It is really strange to see them categorized under
Data centers and other large, non-critical power consumers Region: us-south1 (Dallas)
Zones: us-south1-a, us-south1-b, us-south1-c
Availability: Dallas, Texas is also listed as a location for Vertex AI.
Region: "South Central US" (paired with "North Central US")
Azure Government: Region in Austin, Texas
Availability: Azure has a physical presence in Texas.
This sounds like someone was trying to mitigate the "AI is eating our small town electricity" and threw the baby along with the water.
Under national fire and electrical codes hospitals have to have at least three completely isolated power systems (these are the different colors of electrical outlets you see).
The life safety branch is the most critical and powers things like fire alarms, exit signs, stairwell lighting - anything required to evacuate patients.
The critical branch is patient care like ICU beds, ventilators, and operating rooms.
Lastly is the equipment branch which is essential to patient care but won't kill people immediately: HVAC, kitchens, and sterilization/cleaning equipment.
There is usually a fourth system for everything else, but these are the required ones. Each has its own requirements for battery and generator backup (life safety can't take more than 10 seconds to cut over), and usually the systems are wired so that they can draw power from lower priority circuits if needed.
And prioritizing humans (home heating, food freezing, etc ) over servers is a good thing.
https://www.spglobal.com/automotive-insights/en/blogs/2025/0...
[1] https://www.climateandcapitalmedia.com/35-gas-turbines-no-pe...
If they’re that important, shouldn’t be a problem right? AI makes so much money right?
People come first. Not virtual hype driven land grabs.
Not that businesses aren't abusing the process, either. The article mentions that 80-90% of planned data centers won't be built, due to duplicate applications. They duplicate both to secure a "phantom" slot for power, and to get localities to compete with incentives for business. It's hard to plan a grid when financial speculation and gamesmanship is driving the planning.
The worst aspect is that Texas corruption is explicitly being offered as a model for other states in the PJM interconnect.
I wonder how much of the high economic growth rate in the 1950's-1980's came from the lack of gamesmanship and political franchises. But it's probably unrealistic to think professionals could step back from the very maximalist positions that are selecting them as leaders.
When I saw that it only applies to datacenters using more than 75 MW, my first thought was who are they writing the rule around?
I imagine we will see things like "xAI Under 75 MW Datacenter III, LLC", "xAI Under 75 MW Datacenter IV, LLC", etc.
https://energyeducation.ca/encyclopedia/Energy_density
https://www.linkedin.com/posts/bjornlomborg_there-are-no-low...
Bender•5mo ago
Shank•5mo ago
I could see operators of datacenters in Texas wondering about this. Also, it's underrated how much critical infrastructure is dependent on datacenters running. Like, are you going to pull someone's EHR system down that serves a local hospital, while keeping the local hospital on a critical circuit?
Bender•5mo ago
Knowing when the power will be cut will not help unless I am misunderstanding you. If the data-center loses power for even a minute the generators will all fire up and then every ATS will count-down and transfer in 30 seconds. Battery backup only lasts just long enough to do a quick return to service on a failed generator and even that is sketchy at best. A properly engineered data-center can run on reduced generator capacity.
Some data-centers are indeed on circuits deemed to be critical but I could see regulations changing this so that they are "business critical" vs. "life support critical" and some changes could be made at substations so that data-centers could participate in shedding. I think you are right that they will be thinking about this and adding to this probably filing preemptive lawsuits to protect their business. Such changes can violate SLA contracts businesses have with power companies and Texas is very pro-business so I can not compare it to California.
toast0•5mo ago
If you know when the power will be cut, you can start the generators before the cut, and depending on your equipment, you may be able to synchronize the generator(s) with the grid and switch over without hitting the batteries. I assume big datacenters are on three phase, can you switch over each phase as it crosses zero, or do you need to do them all at once?
Bender•5mo ago
I know what you mean though, the generators I worked with in the military had a manual frequency sync that required slowly turning a dial and watching light bulbs that got brighter with frequency offset. Very old equipment for Mystic Star, post-WWII era equipment. 50's to 90's
jimmygrapes•5mo ago
Bender•5mo ago
dylan604•5mo ago
Bender•5mo ago
Agreed. Even my home computer and networking equipment is 100% in-line with inverters and never see commercial power. PG&E in California got me into this habit with all the Planned Safety Power Shutoffs, wildfires, surges from really old transformers and unplanned outages. Now each of my tiny indoor rings of power have 200 to 800 amp-hour capacity each and over-sized inverters. I put the whole-house inverter plans on hold for now.
dylan604•5mo ago
bluGill•5mo ago
Synchronizing generators is a thing, but it isn't useful for this situation since they need to be able to handle the sudden without warning power loss where generators cannot be synchronized anyway.
stevetron•5mo ago
Bender•5mo ago
hdgvhicv•5mo ago
Of course that doesn’t help for fire/flood etc which is why we have critical workloads in two dcs.
bluGill•5mo ago
hdgvhicv•5mo ago
imglorp•5mo ago
https://en.wikipedia.org/wiki/2021_Texas_power_crisis
doodlebugging•5mo ago
I'm not sure that this is correct. I was initially worried about how Mom would fare since she lives alone and is over 80. During the entire one week period of power problems in Feb. 2021 my Mom never lost power, not even a quick brown-out. Her home is within a half mile of a local hospital which also never lost power. The area around the hospital did not lose power so businesses and homes close by had no issues with heating, cooking, bathing, etc during the cold blast. That fact allowed me to stay here at my place a couple hours away and manage my own situation which was fairly easy compared to many others in the state.
Your other statements are quite true and to date no one who played a part in mismanagement of utility power in Texas has been held accountable nor will they ever be in a libertarian state where regulations exist only to guarantee a profitable situation for a commercial entity. In fact, most electricity customers in Texas ended up paying for the huge cost increases that occurred as those in charge tweaked the system in real time to maximize their own profits.
Texas needs regulations worse than most other states. Grifters, fraudsters, and thieves have filled too many critical positions for too long.
grepfru_it•5mo ago
doodlebugging•5mo ago
I have more family up there where Mom lives and they lost power for all or most of the week so they all shuffled operations to the homes that had the most reliable power and pooled resources so no one had to be hungry or cold.
grepfru_it•5mo ago
bsder•5mo ago
First, that was the big manufacturers. ERCOT couldn't force big companies off the grid, and they didn't go off grid until the press noticed and started complaining.
Second, the Texas grid has insufficient granularity to actually shed enough non-critical load to do rolling blackouts. There are too many "critical" things connected to the same circuits as non-critical ones, and it would cost money to split those loads (something Texas just ain't gonna do).
Third, the base production got hit because fundamental natural gas infrastructure wasn't winterized, froze and exacerbated the whole situation. It would cost money to fix. (aka: something Texas just ain't gonna do)
Finally, when you don't have big industrial consumers defining your power grid (aka massive overprovisioning), you can't "shed load" your way out of trouble.
The fundamental problem is that, like so many things in the US economy, personal consumption is so low that it doesn't help when the problem is systemic. We've optimized houses with insulation, LED lighting, high-efficiency appliances, etc. Consequently, the difference between "minimal to not die" and "fuck it, who cares" in terms of consumption differential isn't sufficiently large to matter when a crisis hits.
buerkle•5mo ago
doodlebugging•5mo ago
opo•5mo ago
I don't think any organization that considers themselves to be libertarian has ever called Texas a "libertarian state". For example:
>...Texas’ institutions and policies continue to bear something of an old statist legacy. In the Cato Institute’s Freedom in the 50 States study, Texas scores a mere 17th, behind even the southern states of Florida (#2), Tennessee (#6), Missouri (#8), Georgia (#9), and Virginia (#12).
https://www.cato.org/commentary/texas-really-future-freedom
Are there any Texas national or state politicians who are members of the Libertarian Party or even refer to themselves as Libertarian?
const_cast•5mo ago
thegreatpeter•5mo ago
grepfru_it•5mo ago
vel0city•5mo ago
bluGill•5mo ago
nradov•5mo ago
mschuster91•5mo ago
Particularly the "last mile" often enough is only equipped with batteries to bridge over small brownouts or outages, but not with full fledged diesel engines.
And while hospitals, at least those that deal with operating patients, are on battery banks and huge ass diesel engines... private small practices usually are not, if you're lucky the main server has a half broken UPS where no one ever looked after that "BATTERY FAULT" red light for a year. But the desktop computers, VPN nodes, card readers or medical equipment? If it's not something that a power outage could ruin (such as a MRT), it's probably not even battery backed.
There's a German saying "the emperor is naked, he has no clothes". When it comes to the resilience of our healthcare infrastructure, the emperor isn't just naked, the emperor's skin is rotting away.
nradov•5mo ago
marcosdumay•5mo ago
I really doubt the classification of "data centers and other large, non-critical power consumers" extends to telecom infrastructure.
mschuster91•5mo ago
That's the point. Okay, cool, the datacenter is highly available, multiple power and data feeds, 24/7/365. But that highly available datacenter is useless when it cannot be reached because its data feed elements or the clients don't have power.
tw04•5mo ago
vineyardmike•5mo ago
EMRs contain a record of when patients last took some critical but dangerous drug, what their allergies and reactions are, and many other important bits of information. When one of the patients starts to exhibit some new symptom or reaction (very stressful situation!), doctors and nurses look at the EMR to understand the best course of treatment or intervention.
When the EMR goes down, doctors and nurses revert to pen and paper. It’s very slow, and requires a lot of human handoff - which, critically, they’re less practiced in.
tw04•5mo ago
Which you got to after spending 3 paragraphs talking about what an EMR is for.
lowwave•5mo ago
lokar•5mo ago
Bender•5mo ago
more_corn•5mo ago
dylan604•5mo ago
bilbo0s•5mo ago
Until there's new management.
You can't run a business by seesaw.
Best to just count on that rule being enforced and place the necessary battery backups and wind or solar in place to backstop the diesel. Then make any users who need to use those data centers eat that extra cost. There's no problem with us-east costing less than us-west, and us-texas costing most of all. That's how markets work.
dylan604•5mo ago
xxpor•5mo ago
dylan604•5mo ago
xxpor•5mo ago
abeppu•5mo ago
How long can most DCs run with just the fuel onhand? Have standards around that changed over time?
stogot•5mo ago
jabart•5mo ago
The whole area lost power for weeks but gym was open 24/7 which became very busy during that time.
stevetron•5mo ago
abeppu•5mo ago
kalleboo•5mo ago
Bender•5mo ago
There really is not a universal answer for this. Every generator will have what is called a "day tank" that as you might guess lasts for one day under a nominal load.
The day tanks are connected in pods to large diesel fuel tanks. Every {n} number of generators get a main tank. Those tanks vary in size depending on how much the company wishes to spend and how resilient they need to make their data-center excluding fuel trucks. Cities have regulations about how much fuel can be above or below ground at each location. My main tanks were 10K gallons. Each generator used over a gallon per minute under load.
And you are right, during a regional or global disaster fuel trucks will be limited. They who bribe the most get the last fuel but that too will run out. Ramping up production and distribution takes weeks and that assumes roads are still viable and the internet outside of the data-center is still functional.
chaz6•5mo ago
Bender•5mo ago
Cat has since added options for hydrogen [1] but I have no idea how many people have bought them.
[1] - https://www.cat.com/en_US/by-industry/electric-power/electri...
f1shy•5mo ago
doubled112•5mo ago
Also, the data center I use has run from generator power for days after storms with NO quality of service loss. Nothing like some real world testing to remind me they have this figured out.
Is a hospital in the same situation? Or is only part of the hospital on those generators?
f1shy•5mo ago
We had batteries and 2 generators, and once we were minutes from blackout, as the primary generator failed, and the secondary was not dimensioned to cope with the load of AA of a 45 celsius day
quickthrowman•5mo ago
Critical branch is defined as loads that are used for direct patient care, plus ‘additional task lighting, receptacles, and circuits needed for effective hospital operation.’
Life safety branch is the fire alarm system and emergency lighting, plus elevator lights and controls, medgas alarms, PA/notification systems used during building evacuation, and some generator accessories.
Equipment branch has some require items including OR cooling, patient room heating, and data rooms. Some hospitals will add MRIs, non-patient care HVAC, and chillers (for air conditioning) on the generator backup system as well.
There’s typically a fourth system for everything else (‘normal’ power) that is not backed by a generator. Non-emergency lighting, convenience receptacles, and other non-essential loads are on this system.
stronglikedan•5mo ago
Maybe they can cap it for most cases - we'll turn it off for n number of days max - so that companies have a target to prepare for (or maybe that was already mentioned). Of course, no one can prepare for anything, but if it's longer than that then most other things are probably also affected anyway and no one is worried about their favorite data center.
duxup•5mo ago
They were legit tests at the offline site too too, they'd power down equipment, and power it up and we'd fix what didn't come back up. Even the data centers would be fully powered off to test.
At least at that time those banks did not skimp on those big tests and it was a big effort and pretty dang well run / complete.