Hack the Plant Episode 16: Cyber Challenges to Securing Our Electric System
“Initially it was looking at specific types of attacks and thinking how those could be utilized against our systems, but then it became more sophisticated in thinking of how these attacks could be coordinated together by larger actors? …. I think that regulation’s role is more to draw attention and provide you with a base minimum, and then from there, it’s the responsibility of those industries of those actors to step up and design the systems and implement true security.” – David Coher
How can our electrical grid system anticipate cybersecurity attacks? What is the nature of its vulnerability to attack, and what role can regulation play in securing our future?
In this episode, we hear from David Coher, leader of Southern California Edison’s (SCE) Energy Contract Management team, which manages their long-term energy procurement contracts (approximately $4 billion, annually). David is an attorney, who moved from real estate litigation to SCE where he established programs for cybersecurity, participation in California’s Greenhouse Gas emissions Cap & Trade market, and Dodd-Frank compliance.
We discussed how the power grid works and the changing landscape of keeping our energy grids safe from cyber attacks. We also explored the challenges of establishing a regulatory compliance program – in particular how to anticipate cybersecurity threats.
What is next for SCE? What are some potential opportunities and threats on the horizon for the safety of our electric grid? Join us to learn more.
(Subscribe to Hack the Plant on Spotify or Apple, by RSS feed or search for it wherever you listen to podcasts.)
DISCLAIMER: “The opinions expressed by David Coher are his own and do not necessarily represent the positions, strategies or opinions of Southern California Edison, its parent company Edison International, or any of their affiliates.”
TRANSCRIPT
Joshua Corman:
Our dependence on connected technology is growing faster than our ability to secure it, especially in areas affecting public safety and human life.
Bryson Bort:
I’m Bryson Bort. And this is Hack the Plant. Electricity, finance, transportation, our water supply. We take these critical infrastructure systems for granted, but they’re all becoming increasingly dependent on the internet to function. Every day I ask and look for answers to the questions. Does our connectivity leave us more vulnerable to attacks by our enemies? I’m a senior fellow at the R street Institute and the co-founder of the nonprofit ICS Village, educating people on critical infrastructure security with hands-on examples, not just nerd stuff. I founded GRIMM in 2013, a consultancy that works the front lines of these problems every day for clients all over the world.
[SPEAKER]:
It’s playing out in Israel right now where hackers have been going after Israeli water systems. Again, not to steal information from them, to change the setting on the chemicals in Israeli water.
Bryson Bort:
Each month, I’m going to walk you through my world of hackers, insiders, and government working on the front lines of cybersecurity and public safety to protect the systems you rely upon every day.
[SPEAKER]:
If you think that the small town water authorities and the mom-and-pop-sized companies have better cybersecurity in the US than the Israelis do, I have really really bad news for you.
Bryson Bort:
An attack on our critical infrastructure, the degradation to the point that they can no longer support us means that we go back to the stone age literally overnight.
[SPEAKER]:
If we think the government’s going to solve it for us, we’re wrong. We have to help them.
Bryson Bort:
This is not a podcast for the faint of heart. If you want to meet those protecting the world and what problems keep them up at night, then this is the podcast for you.
I’m Bryson Bort and this is Hack the Plant.
For today’s episode, I’m joined by David Coher, leader of Southern California Edison’s (SCE) Energy Contract Management team, which manages their long-term energy procurement contracts (approximately $4 billion, annually).
David is an attorney, who moved from real estate litigation to SCE where he established programs for cybersecurity, participation in California’s Greenhouse Gas emissions Cap & Trade market, and Dodd-Frank compliance.
We discussed how the power grid works and the changing landscape of keeping our energy grids safe from cyber attacks.
David Coher:
It’s really quite incredible when you consider all of the millions of similar transactions that are occurring simultaneously. The thing is that when you see these things in the movies, and I know you know this and most of your listeners are probably aware of this, it’s not something where you just push one button and the whole thing turns off.
So it’s not about guarding this one red button on one control panel on the proverbial Homer Simpson-type screen, it’s much more complicated than that. Which I say partially to indicate of how tough the job is, but the flip side of that is also that that’s a great opportunity, because that means that there’s plenty of redundancy and plenty of resiliency in the system, and that’s the part that gives me greater hope.
Bryson Bort:
We also explored the challenges of establishing a regulatory compliance program – in particular how to anticipate cybersecurity threats.
David Coher:
It’s not just an n − 1, normal with one thing going wrong, but thinking n − 2, n − 3. What happens if we have a series of wildfires and that limits which lines we can have operational? And we’re in the middle of a global pandemic, so we’re limited in our ability to interact physically. It used to be when you tried to get creative, some of the scenarios you would dream up, people would roll their eyes, “This could never happen, this could never be.” Maybe one silver lining out of the COVID pandemic has been that people are more open to more creativity in this role-playing because they have a greater appreciation of yeah, some crazy things really can happen and they can come together and all happen at once.
Bryson Bort:
What is next for SCE? What are some potential opportunities and threats on the horizon for the safety of our electric grid? Join us to learn more.
DISCLAIMER: “The opinions expressed by David Coher are his own and do not necessarily represent the positions, strategies or opinions of Southern California Edison, its parent company Edison International, or any of their affiliates.”
David Coher:
My name’s David Coher, and I am an attorney by training and kind of accidentally cybersecurity attorney. I worked for Southern California Edison for about a decade in the cybersecurity space initially in the law department where I was supporting the implementation of the NERC regulations – NERC meaning the North American Electric Reliability Council. NERC is an industry trade group that has a unique relationship with FERC, the Federal Energy Regulatory Commission, whereby NERC, that industry trade group, gets to establish the regulations for cybersecurity for the energy, in particular, the electric energy industry in North America. It’s a unique relationship because they have a similar partnership with the Mexican and Canadian counterparts of the federal government’s FERC. The reason for that, that’s kind of an outgrowth of the energy policy act of 2005. And what happened in the late ‘00s was that what were voluntary guidelines of this industry trade group became regulatory requirements with the force of law. Utilities and others in the electric industry had to start implementing them. We began to do that. I initially was supporting the first few versions. The regulations have version numbering similar to software, which is also something very unique in the regulatory world. But in any case I supported the first few from the law department, then in later versions in the past few years here it has become something where it requires a more robust compliance program, and I stood up the first cybersecurity compliance program at Southern California Edison. From there really helped to build out the controls across the company to consider all of the cybersecurity implications.
Initially it was about the regulatory requirements and what the letter of the law said, and then thinking, okay well, here’s the base minimum, what are we doing next? What else are we doing? Let’s think about potential threats. How do we then guard against those? What do we need to be worried about? What is a little too Hollywood script for us to be worrying about, and where’s that sweet spot in between? So it is a pretty interesting role with thankfully a lot of great opportunities to really dig into the systems we had across the company.
I’ll tell you a little bit about Southern California Edison: it’s electric utility covering about a third of the landmass of California, but about half of the population, so it’s serving a total of somewhere under 15 million individuals. But in addition to that, operating that portion of the transmission grid that’s running through that third of the landmass of California, that is the service territory for Edison.
What that means, and one other thing is the way that California’s utilities are set up, it’s not fully integrated, meaning a traditional utility would own everything from the generation power plant, the wires all the way to the meter on the back of the customer’s house. Because of energy deregulation in the early part of this century in California, Edison owns very little in the way of generation facilities. There’s still a few that we have, but at the end of the day, it’s mostly owned by third parties, those generation facilities, so it’s a pretty good mix. Taking a look at what the cybersecurity landscape is for a company like Edison, you have some generation facilities where you need to be thinking about it. You have many generation facilities that you’re dependent upon that are operated by others, and so you’ve got to be working with them. You have the transmission grid that both you’re utilizing and the entire Western United States is utilizing, to some extent. Then you have the traditional customer relationships, the retail customer relationships, that customer information, so it’s quite a wide open… There’s a lot of room to play. I’ll put it that way, in a lot of different areas.
Bryson Bort:
So to clarify, because the phases of electricity are: generation, the production of electricity; transmission, the moving of that electricity down and breaking it into the different regions for different kinds of consumers; and measuring and monitoring that consumption part. So what exactly does Southern California Edison do on that latter part because it sounded like that’s also a mixed ecosystem?
David Coher:
Here in California, we have deregulation on the generation side of the wire or the power plant side of the wire; we do not have deregulation on the consumer side of the wire. So those who are familiar with Texas will know, in Texas you have a system where you get to pick the company that you buy from. Here in California, it’s assigned to you; it’s either one of the investor-owned utilities or IOUs, they’re called. Edison’s one of those. Or in many other cases, it’s your local city. I actually, ironically, I live in the City of Pasadena, which has a municipal utility, so I don’t get the discount as an employee.
Bryson Bort:
How did you accidentally get into cyber? You were lawyering along and suddenly boom cyber?
David Coher:
My career as a lawyer, prior to joining Edison I was actually handling real estate litigation here in Los Angeles at the time, in the LA area, and I had an opportunity to join Edison because Edison was having a lot of issues. This was back in the 2006, 2007 period; a lot of things were being built right before the great recession. And Edison, it’s one of the larger land owners in the state of California because of all the transmission line property. But there’s a lot of legal issues in and around that, and so I ended up joining Edison and handling litigation with various land owners, or those who owned land adjacent to the wires. Basically working with folks up and down the state as to what you can build under the wires, what you can’t build under the wires, what you can build next to the wires, what you can’t build next to the wires, why that is.
For example, just a quick example: you can’t have certain kinds of storage underneath wires, because if something catches fire and the smoke burns too thickly, think like a tire fire, that thick smoke can actually carry electricity to ground from the large transmission line if you think of the large towers, mostly outside of urban areas. So there’s a lot of safety issues in and around those wires and what can be built on that land or next to that land.
I was dealing with that initially, that work dried up when people stopped building things because of the great recession. At that time there was a big push in California, there still is, but to build a lot of interconnections for new energy facilities, solar and wind farms. This is in the late ‘00s, early teens, and those relationships with the… If you want to build a solar farm out in the desert, oftentimes there won’t be a wire going to the place where you want to build the solar farm, which is you want to build it where it’s very sunny.
Generally speaking, when the electric grid was being built in the early part of the 1900s across California, the wires were put in places where they’d be protected from wind and protected from the sun and the exposure. Well, turns out all of a sudden we wanted to start putting our power plants, if you will, in the places where it’s windy and sunny, so that meant building new wires. And all of the wires, those relationships were governed by FERC, the Federal Energy Regulatory Commission, and so given that I’d been working on transmission line issues, I moved over from the real estate side of things to FERC regulatory work.
As I was talking about it a little earlier, about that time FERC also got the assignment to start adopting these cybersecurity regulations. So it was really being in that group within the law department where I was handling these transmission line issues, and then this other, smaller thing came in and they said, “All right, well, we got this new set of regulations,” and I had an interest in technology myself, so I took it on and it really grew from there into what we see today, and I’m sure there’s a lot of growth opportunity even beyond.
That came about from when we established a separate group to be handling cybersecurity and handling the compliance side. It was A, an acknowledgement that there was a large volume of work in terms of traditional compliance work: documentation and just making sure you had all those pieces of evidence and had them in an organized fashion. But also at that time, that came about from a reorg, and I think it was great because in that reorg, I had an opportunity to really have conversations with the executives about what it was that we were doing, and those conversations then naturally transitioned from, “Okay, well that’s great, that covers X, but what about Y? What about Z? What about all the other various things?”
Those were some pretty fun conversations, because it really got the executives engaged in thinking about the value proposition from both the threats and the opportunities. So we established a group and was able to staff it up in a way where we had that compliance piece to it, but also could then start to explore what are the other things that could be happening here? In other words, a recognition that regulation is going to be your base minimum of what you should be doing. Regulation’s what you have to be doing, and depending on the utility that you’re operating, it may be appropriate to be at or slightly above the regulatory requirements. But if you’re a large utility, an entity like Southern California Edison, one of the largest utilities in the country, you should be thinking about, what else can you be doing? What else do you need to be doing? How should we be leading the industry in this space, similar to how we’re leading the industry elsewhere in other segments of the electric utility industry?
That was really a great opportunity because what that meant was now I could go and have some fun. Now I could go with the team and let’s start exploring what’s out there; let’s delve into this world and see what are the threats that are on the horizon. Think about how we apply or would apply those threats if we were kind of putting on the black hat, if you will, and thinking about how could that be utilized here on this wide variety of systems that we have? From everything from the industrial control systems in power plants, for the generation facilities that we do own and operate, to your traditional concerns about data exfiltration and the like, and really everything in between. So that was a fun opportunity because there’s a lot of room for creativity there.
Well, it’s especially nice when you get to play the bad guy, you put the hat on, play the role, and then you take the hat off. You don’t have to deal with the consequences, because you’re not actually doing the damage.
Really on two levels, initially it was looking at specific types of attacks and thinking how those could be utilized against our systems, but then it became more sophisticated in thinking of how could these attacks be coordinated together by larger actors? Now the concerns are primarily in the nation state or similarly organized organizations. Obviously we have some threats in terms of our customers and the tremendous volume of PII (personally identifiable information) that we have as a company with millions of customers. But really I think the greater concern, frankly, is the potential threat for our infrastructure role and what that means. And I don’t want to limit it necessarily to just turning off the lights, so to speak, but then what is the plan beyond turning off the lights? How much further could that go, and what purpose would turning off the lights serve? Obviously our role in that would be to not have the lights turned off, and so a key component of the plan fails. But it’s not like that’s the end goal alone, for much of what we’re thinking about.
Bryson Bort:
How do you play the bad guys? It’s a fine line between testing your systems and, of course, not affecting the delivery of electricity to those customers. How do you do that?
David Coher:
It’s really more running a lot of tabletop exercises, going through the process and stepping through it and thinking, okay if this happens, then what next?
You’re right, you want to shy away from anything that’s going to create any operational risk in and of itself, but I think that there’s plenty of room just in that space alone, to be able to think through what are the possibilities here? What could happen? Okay, if X happens, then how would we react? What are our options to react? A key part of that is just being aware of your options, of how you would react so that if an incident does occur, you know what to do or, or not do.
Bryson Bort:
And sometimes those incidents have nothing to do with any particular third-party human trying to do something malicious; sometimes it’s the environment. So I’m not drawing comparison to the environmental challenges that ERCOT faced in Texas that we covered in a previous episode.
David Coher:
Absolutely, yeah.
Bryson Bort:
But California does face its own unique environmental challenges.
David Coher:
No, that’s absolutely right. That’s part of the exercise planning, of thinking through what would happen in your… It’s not just an n − 1, normal with one thing going wrong, but thinking n − 2, n − 3. What happens if we have a series of wildfires and that limits which lines we can have operational? And we’re in the middle of a global pandemic so we’re limited in our ability to interact physically.
It used to be when you tried to get creative, some of the scenarios you would dream up, people would roll their eyes, “This could never happen, this could never be.” Maybe one silver lining out of the COVID pandemic has been that people are more open to more creativity in this role-playing because they have a greater appreciation of yeah, some crazy things really can happen and they can come together and all happen at once.
Bryson Bort:
So what’s next for Southern California Edison?
David Coher:
Well I think in this space, I know Edison has a team in place. I’ve actually moved to a different role myself, but Edison has a pretty robust team in place. It’s been growing both in terms of size and capabilities. The concerns I think are now much more prevalent, so when you look at industrial control systems, I think are really having a moment in cybersecurity, both in terms of an awareness on the executive levels, as to what the potentials are. Also the policy makers have shown up at the party, which has both good and bad aspects, and we can talk a little about that if you’d like. But frankly, I think it’s an area that’s just going to continue to grow and mature. As things become more and more integrated, that means that infrastructure becomes a much more critical part of the national security conversation.
So it’s not just necessarily about the business of keeping the generation flowing so that we can still collect and pay the bills, we’ve moved well beyond that as an industry, really. I think the next steps are to figure out how to balance that role and responsibility that the utility industry holds in the national security conversation with the governance structure we have here in the United States of how we have this set up in terms of our regulation. I think that that’s also a conversation that’s occurring across many other industries, and that’s why ICS (industrial control systems) are having this moment in cybersecurity.
So if you look at the history of these regulations in the electric industry and how we’ve gotten to the place where we’re at currently, or similarly, cybersecurity regulation in the finance industry, and then turn and look at what is going on with industrial control systems across manufacturing and other important segments of the economy, I think that you’re going to see those other industries are going to be following behind where finance and the utilities are at. I think we’ve seen…
It’s some more troubling examples in terms of manufacturing and healthcare. For example, some of the recently reported attacks, it gets people concerned, rightfully so. So there needs to be a system to address those concerns, and that’s where regulation can step in. I’m not saying it’s a panacea, it’s actually going to solve your problem and provide you with security. I think that regulation’s role is more to draw attention and provide you with a base minimum, and then from there, it’s the responsibility of those industries of those actors to step up and design the systems and implement true security.
Bryson Bort:
If you could wave a magic air-gapped, of course, wand what is one thing you would change?
David Coher:
If I could wave my magic air-gapped wand and change one thing, I think I would change the perception that automation is an automatic cost savings. I know that’s somewhat amorphous, it’s not very concrete, certainly. But what I mean by that is I think that in a lot of businesses, automation, taking what was separate and disconnected and putting it onto the “cloud” so as to provide cost savings, to cut down on labor, to consolidate, all of these things we’ve seen as, “Hey this is great. This is great for the bottom line, and we get all these cost savings.” What I think happens is the costs of security do not get factored in until after the fact. Then the execs are upset, “Well I was supposed to save 20% by doing this. Now I’m only saving 10%,” but just in the same way that previously you had physical security needs, or you had costs in the physical realm, now you have those security and cost needs in the cyber realm, if you will. I think it’s the overvaluing of automation and undervaluing of the cost of doing business in a new way.
Bryson Bort:
Alright, you waved your magic wand. Now looking into the crystal ball for a five-year prediction: one good and one bad thing.
David Coher:
Well, since you gave me a very precise time period to analyze for, five years, I’ll say on the good thing, I think that we’re going to continue to see maturity and growth in this space, both in terms of on the regulatory side, where I think we’re going to be getting some new regulations. I don’t think it’ll be happening, quite frankly, until 2023 because of the political machinations, but nonetheless, we will be getting a new series of legislation and regulations that expand into more industries for industrial control systems. I think that’s a positive. That is going to further drive the development that has been occurring in terms of vendors out there who are servicing these needs, both in terms of products and service offerings.
In terms of the one bad thing, I think five years is about the right timeline, frankly. I think that we’re going to see there’s going to be some incident. I don’t necessarily want to say the feared cyber Pearl Harbor, per se. I don’t know what the standard is for meeting that level anymore, but there will be an incident similar in the way that the Target hack, for example, really opened a lot of people’s eyes to the risk for their information being held by all these various companies. Not that the Target hack itself was the most sophisticated, not that it was the biggest, even at the time, but maybe it was because of a popular brand. Maybe it was because of timing of when the news was announced, or any of a variety of factors that I may not even be considering, that happened to be what perked people’s interest.
There was a political scientist, James Q. Wilson, who talked about something called policy windows, and it’s the idea of, you know how, when you go to launch a rocket or the shuttles, they would have these launch windows: a specific period of time when everything was aligned perfectly so that you can execute the launch within this set period of time, and then the weather changes on you, and the window closes? I think it’s a similar idea here for the policy windows and for that large incident that we’re going to be seeing, that’s going to push things forward. I’m not hoping for it, and I’m certainly hoping that it’s a minimum of impact, but a maximum-of-awareness opportunity. I don’t know which industry it’s going to be in, but I do see something like that coming and driven by a nation-state actor in support of other policy goals. That’s the one bad thing I see in the crystal ball five years out.
Bryson Bort:
So, grab bag: anything we didn’t cover that you want to?
David Coher:
I’ll take the opportunity here to talk a little bit about the grid and frankly, what an impressive machine this is. So the way the US is set up, you basically have three, really North America, because it operates in the US, Mexico, and Canada, and you have three interconnections. They’re called the Eastern Interconnection, the Western Interconnection, and then ERCOT, which I forget exactly what ERCOT stands for, but it covers most of Texas, although not all. So you’ve got most of Texas covered by ERCOT. The Western Interconnection covers El Paso and a little bit into Texas there, covers New Mexico, Colorado, Wyoming, I’m trying to go up in my head. And then I think most of Montana, Alberta and British Columbia. And then Baja down south. This Western Interconnection, it’s the world’s largest machine, it’s often referred to as. And if you think about what’s taking place, you have a windmill on the Columbia River that is spinning and generating electricity. That’s moving energy through wires down to a substation, up to Bulk Electric Transmission, B.E.S. as it’s called, or the BES, but the Bulk Electric System, on through over thousands and thousands of miles. Then coming to a substation here in beautiful Southern California, stepping down at that substation to a wire that’s running through my backyard and down to a panel on the back of my house. Then from there, of course on into the plug in the wall and then up through my switch box, my USB router, into this microphone.
It’s really quite incredible when you consider all of the millions of similar transactions that are occurring simultaneously. The thing is that when you see these things in the movies, and I know you know this and most of your listeners are probably aware of this, it’s not something where you just push one button and the whole thing turns off. That’s good and bad, right? I mean, sometimes you wish you could just push one button and make the thing work, but there’s lots of interconnected and moving parts. And frankly sometimes, as you mentioned earlier, sometimes those pieces will fail on you on their own, regardless of what you’re trying to do.
So it’s not about guarding this one red button on one control panel on the proverbial Homer Simpson-type screen, it’s much more complicated than that, which I say partially to indicate of how tough the job is. But the flip side of that is also that that’s a great opportunity, because that means that there’s plenty of redundancy and plenty of resiliency in the system, and that’s the part that gives me greater hope. If you just had to protect that one proverbial button, it’d be a lot more difficult of a job that you’d be talking about. And so if you carry that example out, once you understand how complicated all of this is, it’s both positive in the sense of there’s a lot of opportunities for folks to step in and protect things. But the flip side of that is the old joke of the internet really is held together by chewing gum and twine.
Bryson Bort:
Alright, and I think that is a wrap.
