Hack the Plant Episode 12: ERCOT and the Texas Power Outage
Today on Hack the Plan[e]t, Beth Garza, a senior fellow with the Energy & Environmental Policy Team at R St and former director of the Electric Reliability Council of Texas, aka ERCOT explains the power outage that crippled the state of Texas back in February during a week of record-cold temperatures. Three severe winter storms and an electricity generation failure left almost 5 million without power, and gave rise to shortage of heat, food and water.
Over the course of her 35-year career in the electric utility industry, Beth Garza has held a variety of leadership roles in generation and transmission planning, system operations, regulatory affairs and market design for both regulated and competitive entities.
In this episode, we discuss how ERCOT manages the flow of electric power to more than 26 million Texas customers – and how the massive power failure happened.
What does this power outage suggest about the resilience of our critical infrastructure? Join us for an in-depth discussion.
(Subscribe to Hack the Plant on Spotify or Apple, by RSS feed or search for it wherever you listen to podcasts.)
Transcript:
Joshua Corman:
Our dependence on connected technology is growing faster than our ability to secure it, especially in areas affecting public safety and human life.
Bryson Bort:
I’m Bryson Bort. And this is Hack the Plant. Electricity, finance, transportation, our water supply. We take these critical infrastructure systems for granted, but they’re all becoming increasingly dependent on the internet to function. Every day I ask and look for answers to the questions. Does our connectivity leave us more vulnerable to attacks by our enemies? I’m a senior fellow at the R street Institute and the co-founder of the nonprofit ICS Village, educating people on critical infrastructure security with hands-on examples, not just nerd stuff. I founded GRIMM in 2013, a consultancy that works the front lines of these problems every day for clients all over the world.
Bryson Bort:
I’m Bryson Bort and this is Hack the Plant. For today’s episode, I’m joined by a colleague of mine at R St: Beth Garza, who is a senior fellow with the Energy & Environmental Policy Team. Over the course of her 35-year career in the electric utility industry, Beth Garza has held a variety of leadership roles in generation and transmission planning, system operations, regulatory affairs and market design for both regulated and competitive entities. Her previous employers include Nextera and Austin Energy. Prior to joining R Street, Beth Garza served as the director of the Electric Reliability Council of Texas, aka ERCOT- which gives her a unique perspective on the power outage that crippled the state of Texas back in February. Three severe winter storms and an electricity generation failure left almost 5 million without power, and gave rise to shortage of heat, food and water. Beth, who lives in Texas now, shares her experience, and explains how this infrastructure failure happened.
Beth Garza:
The proximate cause was weather. It was pervasive, widespread, very cold weather. And for listeners who may live in New England or Montana, they would laugh at what we in Texas would call cold weather. But the fact of the matter is that here in Austin, we set a record that week for the number of hours below freezing. And that that record was 168 hours. So we were a full week of freezing weather, not rising above freezing, which is very, very unusual. So what happened? Lots of power plants became unavailable in rapid succession, likely due to a combination of cold weather and freezing components at the power plants.
Bryson Bort:
The massive power failure was a cascade of events within ERCOT which manages the flow of electric power to more than 26 million Texas customers — representing about 90 percent of the state’s electric load. As the independent system operator for the region, ERCOT schedules power on an electric grid that connects more than 46,500 miles of transmission lines and 710+ generation units.
Beth Garza:
All types of generation that are part of the ERCOT mix suffered or none of them were at full capacity. All types of generations suffered some sort of outage due to the weather. And so what happened is that in the very early morning, just after midnight on Monday morning, in a world where demand has to equal supply all the time and it has to meet exactly, the situation was that demand was exceeding the diminished supply. Power plants were tripping offline. There just was not enough supply to meet customer demand.
Bryson Bort:
How does ERCOT work? What does this power outage suggest about the resilience of our critical infrastructure? Join us for an in-depth discussion.
Bryson Bort:
To orient our listeners, if you recall in February, Texas suffered a crippling major power crisis with results of the winter storms and millions of folks went without power for several days. In that context, what was ERCOT’s role? Why is the setup with ERCOT and the grid in Texas particularly unique and what contributed to the loss of power in Texas?
Beth Garza:
So it’s easy to forget, certainly now as I sit here in Austin, Texas, and it’s 75 degrees right now, we have to remember, it was bitterly cold with icy, snowy precipitation starting a couple of days before the electricity crisis. So I like to describe the event as it started as a weather issue, and when I say weather issue, traffic accidents, fatalities in… There’s a great picture of over 100 vehicles involved in a massive pile up in the Fort Worth area, the weekend before the electricity crisis.
Beth Garza:
And so weather precipitated electricity crisis, that really started very early Monday morning, February 15th. And lasted for the bulk of that week. On the electricity side, things were pretty well back in shape by Friday, but that pervasive lack of electricity for that week in very cold weather certainly had dire outcomes. I’ve seen numbers of 100 to 200 deaths specifically attributed during the winter storm. And that lack of electricity then led to a water crisis that lasted for at least another week.
Beth Garza:
And in some cases, several weeks afterwards. Burst pipes, inhabitable structures, just really a problem on the water side. And now two months later, we’re dealing with the financial fallout from the storm. And so, that’s the sequence of events. The proximate cause was weather It was pervasive, widespread, very cold weather. And for listeners who may live in New England or Montana, they would laugh at what we in Texas would call cold weather. But the fact of the matter is that here in Austin, we set a record that week for the number of hours below freezing. And that that record was 168 hours.
Beth Garza:
So we were a full week of freezing weather, not rising above freezing, which is very, very unusual. So what happened? Lots of power plants became unavailable in rapid succession, likely due to a combination of cold weather and freezing components at the power plants. There certainly were also limitations of fuel. And before I get too focused on natural gas, and it’s easy to get focused on natural gas because that’s the largest fuel source here in ERCOT, let me just say that every type of generation suffered some form of limitation during the week.
Beth Garza:
And when I say all types of generation, nuclear, coal, natural gas, wind, solar, all types of generation that are part of the ERCOT mix suffered or none of them were at full capacity. All types of generations suffered some sort of outage due to the weather. And so what happened is that in the very early morning, just after midnight on Monday morning, in a world where demand has to equal supply all the time and it has to meet exactly, the situation was that demand was exceeding the diminished supply. Power plants were tripping offline. There just was not enough supply to meet customer demand. And this is the middle of the night, when most people are asleep.
Beth Garza:
So bringing it back to ERCOT and their role in all of this, the ERCOT organization, the ERCOT operators are the ones who are responsible for matching load and supply exactly and continuously. And when they can’t do that on the supply side, when there’s no more supply, no more resources to come in, the only way to keep those two balanced is to curtail load. And so it was ERCOT operators that issued the order for distribution companies, what folks may think of as traditional utilities, to actually interrupt, to turn off customers.
Beth Garza:
ERCOT issues the order, but which specific customers and how each distribution company implements its obligation to curtail is left to that distribution company itself. Each distribution company determines which of its own customers will be curtailed and is responsible for executing on that.
Bryson Bort:
So ERCOT is managing at this higher level, and then we have technology down at specific consumers called AMIs, advanced metering infrastructure, which is managing and tracking that load at that level. Can you explain that technology little bit more?
Beth Garza:
So advanced meters are, in my perspective, kind of a catch-all phrase for electronic meters. And what I mean by that is I’ve been around long enough to remember the meter that hung on your house was an electromechanical device. It literally spun, there was a disc that spun in it and you read the meter based on how that spinning disc spun various dials that allow you to interpret consumption. AMI are electronic meters, and with the electronic meters, they can be remotely read.
Beth Garza:
And that’s the primary benefit, I think, that most most entities have used as justification for implementing advanced metering. You don’t have to have a person that walks the neighborhood, reading meters. You have an electronic device that captures data and sends it in. Some of the challenges are that the choices about how much data or the granularity of how detailed the data is captured, and then how long you assemble that data before transmitting it, those parameters are user-defined, if you will, and therefore can be different for different utilities and utility implementation.
Beth Garza:
I offer that just as an example. I am a customer of Austin Energy, which is a municipal utility, right in the middle of the state. Austin Energy has advanced meters. During this storm, in a press conference that I listened to, the general manager for Austin Energy was asked this question, “Well, why can’t we curtail customers using advanced meters?” And her answer was, “We can turn the meter off remotely. We can’t turn it on remotely. We have to send somebody out. That’s unique to Austin. That may not be the case in Houston or Dallas area.”
Beth Garza:
And so there are other aspects of the advanced meters themselves and reasons why when those distribution customer or distribution companies are ordered to curtail customers, the action they take is at the substation. They interrupt hundreds, if not thousands of customers at the same time. And they generally choose those circuits they’re interrupting based on which kinds of customers are fed by those circuits. And they generally are focusing on residential customers to the detriment or avoiding more critical loads like hospitals and fire stations.
Beth Garza:
But the speed with which those orders have to be implemented is rapid. When ERCOT issues an order that says, “We’re going to firm load shed,” and what ERCOT does is they define the total block. And then every distribution company in Texas knows their percentage of that total block. And so then for example, Austin Energy then, if ERCOT says, “We’re curtailing 1000 megawatts of load,” Austin Energy says, these are rough numbers, “I’m 4% of the total. I’m going to curtail 40 megawatts and to get to 40 megawatts, I need to interrupt this circuit and this circuit and this circuit,” and then from their control room, they can push a button and a switch in the substation will open and stop flow of electricity to all of those customers.
Bryson Bort:
Pivoting from the technical to the people’s side, I think that one of the things that’s neat is there are a lot of talking heads who spout about this. They haven’t actually been down there in the trenches. Beth Garza, you have. What can you tell us about the culture and people in the industry and sharing some of the more colorful expressions for how they describe themselves?
Beth Garza:
Well, it’s been many years since I’ve been anywhere near a trench, let me say it that way. Not that they would allow me anywhere near an actual physical trench, but I did manage the Austin Energy System Control Center for a period of time, a long time ago. And the tools and techniques we had at that time were not what’s available to the folks today.
Beth Garza:
But managing these types of force curtailments is done in a very similar manner now. You identify the circuits in sort of a priority level, and you hear the order from ERCOT as to how much you have to curtail, and you actually implement that curtailment. Some of the things that have come out from this, what was so drastic or so dire about this storm was that the amount of load that needed to be curtailed was so significant.
Beth Garza:
And to put it into context, the last time there was a forced curtailment of customers in ERCOT was 10 years ago, in 2011. And the magnitude of that curtailment was probably six to 8% of anticipated demand and it lasted for four hours. This time, we curtailed as much as 30% of anticipated demand. And I can speak from my personal experience, I was out of electricity for 81 hours due to that forced curtailment.
Beth Garza:
Again, we have to remember, this was a winter storm. And so there were customers that had been out prior to this event on early Monday morning, because of icy limbs falling and interrupting distribution circuits. Even the restoration of power after there was sufficient supply was delayed because of ice on tree limbs that had to be… fallen tree limbs that had to be cleared out before circuits could be put back in service. And so that is all part of the normal, if you could use that word, normal winter storm operations for any distribution utility.
Bryson Bort:
So tell us about the folks at ERCOT.
Beth Garza:
So ERCOT employees as a group, it’s an interesting organization. I was part of it for a brief time, about 12 years ago. And it’s interesting because on the one hand, you have basically utility operators and I would put myself in that camp, people who grew up in the electric utility world, operating and planning of the systems, and we would refer to those folks as the DOUGs, the dumb old utility guys. And I absolutely put myself in that camp.
Beth Garza:
The other thing that ERCOT does is they have a massive and very significant IT structure and organization, because they’re dealing with high volumes of data and reporting, and lots of information is flowing through there. And as somebody who is a self-described DOUG, that was just a different culture on the IT side. And I would hope that at this point where the organization’s been around long enough that the there’s less obvious divisions between the two sides of the house, but it does create a challenge for a unified organization when you have two very different structures, structures and background of the employees and trying to find a unifying culture throughout the organization.
Bryson Bort:
Talking to you even further, what is the view of security from the front lines? And security of course, cyber security, physical security, the whole shebang.
Beth Garza:
So again, it’s been a very long time since I’ve been close to any of the physical aspects, but we can start with just physical security of the system, because there are physical elements to it. In your neighborhood, there are distribution lines that are either underground or going down the down the street, and they connect to your house. And from your house up to a substation, and at a substation, then you have transmission lines that are big and visible and connect each of those substations and the network and incorporate power plants that are sort of really big things that you can’t miss. You know where they are.
Beth Garza:
And so from a physical security perspective, each of those elements, it’s hard to hide them. And bad actors can certainly be disruptive if they wanted to. And it’s dangerous. Those are dangerous facilities with high voltage and if you don’t know what you’re doing, you can kill yourself. And what’s the term of art? The attractive… I’ve lost it, it doesn’t matter.
Beth Garza:
So that’s the physical stuff. The cyber stuff, I have even less knowledge about. I’m just one of those frustrated users that has to take cyber or had to take cyber training every year to make sure I didn’t fall victim to a phishing scheme or some other nefarious way to try to get into systems. And why is that? Because these are all systems that are connected, and increasingly connected via the internet in an electronic way, in ways that potentially could be exposed to bad actors. Actors that either by intention, mal intention or by accident, can do great harm, either financially or physically, possibly.
Beth Garza:
Recognizing how critical it is, and electricity’s one of those things that it’s very easy to take for granted. Every day, we wake up in the morning, you turn the light on, you turn your computer on. It just works. That’s just life as we know it right now, until you live without it in your same house for a period of time, and then you recognize how pervasive it is as just the way of life in society right now. And the disruption that can come from not having electricity highlights how important it is to do everything we can to continue to have it.
Bryson Bort:
Security has been in the news a lot lately, but there are a lot of challenges to actually doing it. One of them is that a lot of these asset owners, the utilities themselves, are limited by rate-based infrastructure. Can you talk about that and some of those challenges?
Beth Garza:
So one of the differences in ERCOT, in the ERCOT structure is that the bulk of the system is not limited by rate-based structures. They are even more limited based on competitive forces. So power plants and retail providers, those entities actually billing individual customers and receiving money from individual customers, none of those entities have a guaranteed rate of return or guarantee of revenues of any sort. They are competing against others of similar type.
Beth Garza:
And so if you want a power plant, a generator to implement appropriate cybersecurity safeguards in ERCOT, those have to be mandated. It has to be structured such that if you want to operate in ERCOT, here’s what you have to do. And the costs of doing so have to be such that that generator owner can recover them as part of their competitive supply. And that’s easier to do if every generator is having to do the same thing. If every generator has to do the same thing, then just the overall cost across the board is increased. And you’re not creating a differentiated response, which could preference one asset owner versus another.
Beth Garza:
So that on the generation side, similarly on the competitive retail side. And then on the wires, the transmission and distribution companies, they are regulated. And my pretty blase view of regulation is regulated entities will do whatever regulators tell them to do and whatever they’re allowed to recover their costs of. And so that’s from my perspective, regulation is a much more straightforward mechanism for ensuring whatever requirements or standards it’s decided needed to be imposed are actually imposed and implemented.
Bryson Bort:
And if you were to make your druthers, where should we be investing?
Beth Garza:
Okay, yeah. I guess if I only have one area, a pot of money in one area to be spending it on or figuring out how to better invest and strengthen just the overall grid, my attention would absolutely be drawn to the transmission system. I view high-voltage transmission as just the great enabler to allow customers access to the lowest cost supply resources. But of all of the aspects of the utility system, transmission is probably the hardest to understand and hardest to actually implement, but I think it provides the highest value. So maybe that’s the right trade off there.
Bryson Bort:
Looking into the crystal ball for a five-year prediction, what’s one good thing and one bad thing that you think is going to happen in the US electric grid?
Beth Garza:
That’s a very intriguing question. One good thing, I believe, will be the continuing de-carbonization of the electricity system. And I think that’s a good thing. I think we’re at a great juncture point as older, more carbon-emitting facilities are reaching the end of their economic life. Clearly, the new wave of resources is less carbon-intensive.
Beth Garza:
And I think as an industry, we will be up to the challenge of effectively integrating smaller and more variable resources throughout the grid. I think it will be harder, but I think we’ll be up to the challenge. The bad thing that could happen is that somehow, we find the problem to be too hard to manage somehow. And that we, and that’s a broad we, that’s a national or a societal we, decide all these little things, smaller resources that move up and down more frequently or vary with the weather, that’s not what we want.
Beth Garza:
We want big stuff. And that would be a bad outcome in my view. That would be a bad outcome in my view. I’ll just stop there.
