Hack the Plant Episode 11: Department of Defense Policy and ICS Security
This is from today’s guest, Daryl Haegley the Director of Cyberspace Mission Assurance and Deterrence at the Department of Defense. Daryl oversees cybersecurity efforts to secure control systems (ICS) and operational technology (OT), and focuses on bringing awareness to the ever-increasing cyber threats.
Today, we’re here to talk about the challenges of securing OT systems. Daryl draws on his 30 years of military, civilian and commercial consulting experience to discuss the kind of public and private solutions we need to prevent the worst case scenario.
(Subscribe to Hack the Plant on Spotify or Apple, by RSS feed or search for it wherever you listen to podcasts.)
Transcript:
Joshua Corman:
Our dependence on connected technology is growing faster than our ability to secure it, especially in areas affecting public safety and human life.
Bryson Bort:
I’m Bryson Bort. And this is Hack the Plant. Electricity, finance, transportation, our water supply. We take these critical infrastructure systems for granted, but they’re all becoming increasingly dependent on the internet to function. Every day I ask and look for answers to the questions. Does our connectivity leave us more vulnerable to attacks by our enemies? I’m a senior fellow at the R street Institute and the co-founder of the nonprofit ICS Village, educating people on critical infrastructure security with hands-on examples, not just nerd stuff. I founded GRIMM in 2013, a consultancy that works the front lines of these problems every day for clients all over the world.
For today’s episode, I’m joined by Daryl Haegley, the Director of Cyberspace Mission Assurance and Deterrence at the Department of Defense, where he oversees cybersecurity efforts to secure control systems (ICS) and operational technology (OT). Daryl has 30 years of military, civilian and commercial consulting experience, and now focuses on bringing awareness to the ever-increasing cyber threat to unprotected connected OT devices.
We’re here today to talk about the challenges of securing OT systems.
Daryl Haegley:
….in January, the water company in Florida, they had somebody notice someone was trying to take control and modify the controls and the mixture within their system. …… the department has about 500 installations, 250,000 buildings, over 200,000 structures and another 200,000 linear structures.
…..We almost have to adopt the same kind of mentality for the folks who are making sure these systems work as to ensure they’re secure. And that’s been a challenge on what cyber requirements should we expect of the people who have been normally managing HVACs.
Bryson Bort:
We also discuss why cybersecurity for control systems in particular (not just for IT) is so important….
Daryl Haegley:
…..I think we’re going to see despite investments, despite technology, we’re going to see some ransomware on some of these critical infrastructure systems. And I think people are going to get hurt. Things are going to stop operating. Things are going to explode and there’s going to be some serious consequences. Worse would be if it’s not even malicious. If it’s somebody went out to try and do something and thought they could scare somebody by coming close to creating chaos, but then it just spills away because they don’t understand the full breadth of the operation or what’s involved with the engineering. ….
Bryson Bort:
What kind of public and private solutions do we need to prevent the worst case scenario? Join our discussion for an in-depth analysis.
*music*
Okay. Daryl, what’s your background? How did you get into industrial control systems and cybersecurity?
Daryl Haegley:
I started out, I was in the Navy after college for 20 years and got into what was called the operation security and learn how to do some assessments of information we were freely giving out, got out of that and then got back into OSD after I retired into the assistant secretary of defense for energy installations and environment. And after a year, they had gotten reports that thousands of smart meters were purchased in the DOD, but then the IT people wouldn’t let them on the network. And so, “Daryl go figure that out.” I had no prior training in cybersecurity, had really no insight or past to control systems and just through working group together with DHS, with the Department of Energy Homeland security, we had some of the labs, and a bunch of stakeholders from the army, Navy, air force and Marines.
And we recognized right away that there were no policies in place, and there were no common procedures in place, not a lot of information on threats or vulnerabilities, DH, Department of Homeland Security was just getting started and publicizing information about vulnerabilities in this space. So, that’s where I got started and just learned that there was a lot of things that needed to get fixed from starting with policies and procedures and working with industry to figure out the best way to get a lot of this stuff solved.
Bryson Bort:
So that’s really interesting because smart meters are something that are quite prevalent now, they’re at most residences to help manage electrical consumption. And in some cases returned to the grid. Why did the IT teams freak out about putting those on DOD networks. And how did you convince them and get past that?
Daryl Haegley:
Congress had asked the Department of Defense to start measuring their energy consumption because we were a little bit behind. And again, this was around 2010, 2012. All the other government agencies were asked to measure their energy consumption and then look for ways to reduce it. And we were encouraged to buy meters because on a lot of these bases, all we had was one meter at the front gate and all the energy that was being consumed was coming around to this meter. And it’s turning at the speed of sound. And we were just paying bills and not checking anything. So rightly so we had to do a little bit of our own oversight. So you can’t manage what you don’t measure. So, we started buying smart meters.
Now the people to buy the smart meters were the engineers who manage these systems. And, there was not a good communication between the traditional IT people and the control systems folks. Even though those networks were separate, they still were not given permission to put them on. So it took a couple of years of educating both sides of the IT folks and the control system folks to show how you could put some cyber security mechanisms on some of the meters, some of the meters you couldn’t, they weren’t designed to do that. They had Bluetooth, IRR ports, USB options. So a lot of those were just thrown out. And unfortunately a lot were never used and they’re in the same building as the Ark of the covenant right now.
But eventually, if the IT people were part of managing it, they would start allowing those on there. If they met through the DOD’s risk management framework requirement, but still a lot of them still have just disabled the function of connectivity, and we go around with a clipboard and collect the data. And then we report it instead of having that information automated. That problem has not been solved, that continues on today, especially, for four deployed sites.
Bryson Bort:
So what is your current role and the scope of that within DOD?
Daryl Haegley:
About two years ago, I shifted from the energy office focus to an OSD policy office, that was established by Congress in 2014, called the office of the principal cyber advisor. And that was set up. So it was be the one place where Congress could go to find out how the department’s doing, performance measures on US Cybercom and give advice to the secretary on cyber, from everything from policy, to procedures, to workforce recommendations, Intel support and essentially the track implementation of the Department of Defense cyber strategy. So now within this office, I work with about two dozen folks. And my role specifically is director of mission assurance and deterrence in cyberspace and control system cyber security.
So I look at the policies and processes and solutions to ensure that our missions are able to go off without being impeded, that is looking at all the things that make up a mission and what our mission depends upon, looking at what the activities that we are doing that denies, degrades and deters adversaries from continuing and and certain places with cyberspace.
We all agree that deterrence as in the nuclear sense is not possible for cyber. Cyber is going to always continue. But can we stop them from doing some of the real harm or bad things that we do not want them to do? And then I’m still continuing the role of control system cyber security. Ensuring that all the army, Navy, air force, Marines and the agencies are making progress and investments in doing inventories of the systems, training their folks to learn how to manage them and working with industry and the other government departments to figure out how we best can work together.
Bryson Bort:
So going back to that original smart meters story, this is where you first started to build that relationship with the IT security side of the house that says no, and had to start to learn that control systems and how they work, have different requirements and the ways that they’re going to do their thing. How have you navigated that to this point and how would you like to improve it?
Daryl Haegley:
Well, one of the first things I had to do was learn from the system owners and operators what their environments were and what their day-to-day challenges were with regard to cybersecurity. And so that was really fun is I went to several buildings on different installations and for medical buildings, logistics areas, and regular facilities, and just talk to the folks who run those. So I got to learn what they do and that they manage these systems. These networks can or cannot be connected. And then a lot of these same places, I would try and talk to the IT people and try and learn what their roles or responsibilities are, especially related to these systems.
And consistently learned that there was just this complete separation and that the IT people were not responsible at all. And, that is still fairly prevalent today. It has taken years to get policies in place to direct. Someone has to be in charge. Someone has to report the status of a control system network and somebody has to look at it from a cyber perspective. It’s taken years to get contract requirements to state that they need to have cyber security built in to these control systems. There’s a very standard processes in place. If you’re going to add some software onto your email network. There’s places to test it. There’s people who are certified. You’ve probably seen this before, that just does not exist in the control system space.
And we’re learning every year as the army, Navy, air force, Marines and these agencies go out and look at what they own and operate, what agreements they have. Learning where they’re taking risks right now, because they don’t have anything, and they’re just starting to make some small investments. So, I think there’s a tremendous opportunity that’s still open is to solidify whatever that partnership is going to be between the IT and the control system folks. Some places are more evolved than others. A lot of them though, still have yet to even partner. One really good story is the general services administration. They forced their CIO to sit next to the chief engineer. So their offices were moved right next to each other. So as they moved forward and we’re trying to enact change and security across their department, they were forced to work together. I think that’s a good model.
Bryson Bort:
So there are DOD assets that depend on the private sector. For example, there are community utilities that might be providing power to a post. How do you manage those relationships? Is that in scope for what you do?
Daryl Haegley:
It is. Every installation is supposed to map the dependencies that they have for water, power, wastewater, any other services and understand those dependencies to their missions. We are tracking the completion of that as each of the services move out and do that. A lot of these dependencies have been mapped in terms of, “Hey, we get power from here and water from here.” That next level down on, well, what kind of systems are they using? Are they still using Windows 7? Are they still using 20 tents servers? What processes do they have in place to prevent exploitation of their system that could impact us? So those are the next steps that we are beginning to reach out on. We’ve asked the services to go do that and we have teams that when we go to an installation and we might do a cyber security assessment, we also would look at that and establish those communications.
A lot of the utility services are privatized that are on that support the basis. They’re like 40 a year contracts. And so, we’re very familiar with who the folks are and the services they provide. It’s making that distinction on, in order to launch this missile, which of those services do I depend and which of those services have integrated cyber as part of their protection? Does that make sense?
Bryson Bort:
Yeah, it does. And that’s got to be a real challenge to navigate that between those sides. Particularly when you’re now getting into contract management and certain cycles, service level agreements. And then of course, tied to the fact that the DOD is purpose is national security.
Daryl Haegley:
You’re absolutely right. So we do have good partnerships with Department of Energy and Department of Homeland Security. And we try and work out with each other, the information of what these utilities have because the Department of Energy has got a good handle of what a lot of utilities are doing. And also there are groups at the Department of Homeland Security work with, and so we convey what our missions are and what those key components are. And then, that’s again, that next step. And then, if there are any vulnerabilities, then we have to work together to figure out well, is this in scope for the current contract? Does it need to be modified? Who are the best folks to do it? Do we accept the risk? Do we wait till the next upgrade, which is six months from now or is that something we need to fix right away?
One of the things that occurred, I think it was in January, the water company in Florida, they had somebody noticed someone was trying to take control and modify the controls and the mixture within their system. One of the things we’re working on is to ensure that a utility like that then notifies the department and key stakeholders that, you know, this has happened. And then we all work together to do the forensics and make sure that the lessons learned are shared with others. Because, the department has about 500 installations, 250,000 buildings, over 200,000 structures and another 200,000 linear structures. Like a shed or a pipeline, all these things, typically have something that communicates with something to let its know its status.
And so, that’s been a real fun challenge, getting everybody on board and thinking in terms of security. I’ve likened it to, we understand safety, we understand typically. We look at something and it almost looks unsafe. We almost have to adopt the same kind of mentality for the folks who are making sure these systems work as to ensure they’re secure. And that’s been a challenge on what cyber requirements should we expect of the people who have been normally managing HVACs.
Bryson Bort:
Security has different aspects. Part of it is asset inventory, what do I have? Part of it is understanding what’s the threat. What kind of threats do you model for DOD and what can you share from what you’ve seen in the real world?
Daryl Haegley:
Well, what’s the control systems we have in the Department of Defense are the same that are outside of the Department of Defense. What a Department of Homeland Security puts out, what we learned from the FBI or some other agencies that collect information on this, and then whatever can get released or the public generally sees is that these are now controlled systems are just becoming this target of opportunity. And as people are learning that they can access them and that they generally don’t have security in place. I think just the other day, there is like I mentioned earlier, the Florida water system, there is I think, another water system in Nevada.
We’re going to see more and more of these, we’re going to see ransomware to these systems, just like we’re seeing them in schools and in hospitals. We recognize that there are investments by these, by a number of the countries.
Russia, China, Iran, North Korea, and they have been doing their best to get a foothold in places that are easy to get to. One interesting story I heard was, an intelligence analyst was telling me viewing the different ways in which these countries act. So for example, imagine your room right now with all the things you see in it and you walk outside the door and then, North Korea has done a cyber attack in your room. You’d open up the door and you’d see spray paint all over everything and stuff knocked over and knocked and knocked around and things like that.
Everything gets reset. You leave the door and then you come back in or before you come back in, China had conducted a cyber attack and you open up the door and everything’s gone, even the screws and the light switch, everything’s gone. So now you put the room back together, leave and then Russia does its cyber attack. And you open up the door and you have to recognize that your bottle of Blanton’s has been turned 180 degrees. Now, I don’t know if that’s exactly the way it is, but those are sort of things that we try and determine. Again, not just determine who might be behind some attacks.
Everyone’s getting better and better as we learned with solar winds of hiding their tracks and making it harder too, but we recognize that everyone’s going to try and get into places that are not secure. Almost all of these systems that have been designed years ago were not built with security. I think another interesting example is the Pentagon is going through a renovation of its control systems. It’s a couple of hundred million dollars, and it’s going to take 10 years to do. What year do you think, the control systems were installed and now are outdated and the vendor doesn’t support?
Bryson Bort:
If I were to have to guess, I would probably assume the 1970s to the 1980s.
Daryl Haegley:
Well, that’s a good guess. And so they were updated since then. And so these were installed in 2001. So now-
Bryson Bort:
Oh 9/11, of course.
Daryl Haegley:
Well, but, really even before then, they were already going through the upgrade then. But to me, that’s a lot, what you said, Bryson Bort is exactly what I tend to think with a lot of these systems is that they’re 40, 50, 60 years old and yet we got to replace them and it’s going to cost that. Well, here’s an example of these things are just 20 years old just, and the vendor no longer supports them. So you got to rip out an infrastructure and you got to put it back in, and then you have to put cyber into it from the design, construction and sustainment of this now. And now you’ve got to have somebody to manage that and be responsible for that. And that’s like a whole new way. That’s what made is making this very exciting is we’re going to have folks that are in this space smart to understand that what’s going on is not a regular failure of equipment, but it is something cyber induced.
I would like to give a shout out to Sans as I mentioned earlier, I had no idea about any of this stuff. So I went through the certification for the global industrial cybersecurity professional, and just renewed that. And that really opened my eyes on all the things to consider and there was a lot to learn. Just again, it was very exciting. And right now I’m the only person in the secretary of defense staff that has that certification.
Bryson Bort:
That’s kind of scary if you think about that out loud.
Daryl Haegley:
Well, my point is, yeah, I think we should have more people that do it. We should have more people who are familiar about this in the space. Again, if you have people who’ve been in the department and in headquarters positions for 10, 15 years, typically they just don’t stop what they’re doing and learn something new, but that’s almost what we’re doing with this field. People who might be working in a traditional CIO shop, somebody is typically going to be tapped on the shoulder to now care about this and champion for it.
And I think a podcasts like this, the sands of summit, the ICS Village, the hack, the planet and others that you guys are, the hack the city, those are just tremendous events for people to get an appreciation of what’s out there, what they should learn and resources.
Bryson Bort:
What other kinds of things would you like DOD to do to help achieve that? At the end of the day, this is supporting mission assurance and our reaction time.
Daryl Haegley:
I am very excited to see that the vendor community and the innovative industry folks out there are coming up with a number of solutions that can help us identify, something trying to probe the network, get on the network, detect it, actually, and provide a response to it and even recover from it. I think that’s great. However, if we just spent millions of dollars buying all that, we don’t have the people who have this competencies, skills, have been in internships or apprenticeships to know how to run all that. And some of it sure we could outsource it, cyber is a service for control systems, certainly. Are we going to do that across the board? No way. I mean, we’re not doing that for IT. We’re not doing that for our weapon systems. We’re not doing that for our most sensitive networks and things.
So we do need, I just think, not just the Department of Defense, but all of government and industry, we need to come together and work with academia to get more courses in place that start all the way at the steam level. When you watch kids participate in at that low level, and that’s a great awareness. They’re still just using windows machines or, and going after each other just like a traditional IT network. How do we get it injected to look at infrastructure type kind of networks. And then, what courses in high school, what courses in college and in advanced degrees do we get engineers to learn about cyber? Do we get some of the cyber people to learn about engineering and bring those together?
And where do we have these places to test and train, do we have cyber ranges that enable this and competitions in this space? I know that we now have a cyber competition that we host within the government, but right now, most of that stuff is again, focused on the IT side. I would really like to see more of it on the control system side. So I think, what really would like to see across the board is a focus on training people, promoting this as a professional opportunity, job descriptions need to, include, control systems, competencies, and, more certifications and, opportunities to train and get internships.
Bryson Bort:
For the capsule, the flat we’re you talking about the all army cyber stakes competition?
Daryl Haegley:
No, there was a recent one that’s been set up from 2019. There is an executive order on workforce cybersecurity that there was DHS and DOD was hosting the-
Bryson Bort:
The president’s cup?
Daryl Haegley:
The president’s cup. That’s correct.
Bryson Bort:
Yeah.
Daryl Haegley:
And I guess, I’m not going to lie. I think it’s good news that we’ve had some DOD people in the top slots. I think that’s great. I would like to see more than just one competition a year and again, integrate some of the control systems networks. I recently briefed or gave a presentation to George Washington University and Georgetown university. And you got folks in there who are now studying automated vehicles and I think that’s another place along the lines of control systems of those communications and making sure that we’re focused on protecting those. I think there’s no shortage of opportunities for folks to learn. I think we need do need a concerted effort, and it looks like the potential with this new administration from Ms. Newberger’s comments recently at the sands conference, that they’re going to work on some focus in that area.
Bryson Bort:
Sorry, I just lost my notes. There we go. If you could wave a magic non-internet connected wand, what is one thing you would change?
Daryl Haegley:
Well, if I could wave that magic non internet connected wand, I’d make control systems, cybersecurity on par with IT cybersecurity across the board. And I think that would impact the workforce, the training, the solutions, how we train for offense, how we train for defense, war game scenarios would include this, funding would be appropriate and be matched to suit what the requirements were. I think if we could make just a, the concept of controlled system cybersecurity already accepted and on par as something that you should be paying for something you should be training for something you should be working toward, with what we’ve already instituted over the last 20 years with it, that’s what I do.
Bryson Bort:
All right. You waved your magic wand. You got that. Now let’s look into the crystal ball. Five-year prediction, one good thing, and one bad thing that you think will happen.
Daryl Haegley:
Okay. Let me address what I think the bad thing would be. So I think we’re going to see despite investments, despite technology, we’re going to see some ransomware on some of these critical infrastructure systems. And I think people are going to get hurt. Things are going to stop operating. Things are going to explode and there’s going to be some serious consequences. Worse would be if it’s not even malicious. If it’s somebody went out to try and do something and thought they could scare somebody by coming close to creating chaos, but then it just spills away because they don’t understand the full breadth of the operation or what’s involved with the engineering. So I’m concerned from a bad thing that we’re going to see some really adverse consequences from folks messing around in this infrastructure space.
The really good thing is, I just am so excited with what I see on the industry side. A lot of these tools and techniques they’re being either converted or modified that we’ve learned, got a great deal of learning from the IT and cyber side, we’re pulling them over. And then there’s just whole new ways of thinking about how do we secure these systems. So I’m very excited that we’re going to have some tools that are going to be very effective in being able to identify and either stop or give us the heads up so that we could wind down a system and bring it back up and recover in a way that is not adverse. So, I think those are two there.
Bryson Bort:
All right, grab back. Anything you would have liked us to cover that you want to talk about?
Daryl Haegley:
Well, I would like to know from your perspective, from what you think industry’s perspective is what do they think they need from the Department of Defense?
Bryson Bort:
What do we need?
Daryl Haegley:
Because I’ll give a short story. We’ve been in some discussions with industry and other departments, and it would be well if we have a big cyber attack when are you going to call the Department of Defense? And a lot of the folks in natural gas companies, utilities, other agencies like, “Well, we’re really not going to call you because we know our systems better than you do.” And then we say, “Oh, well, that’s kind of surprising. Well, what if there’s a major event in Baltimore and a major event in Washington DC?” “Well, that’s pretty good and pretty close. We still think we might be able to handle that. We’d like you to be ready, but we think we can handle that.” “Oh, well, what if something happens in Savannah and in Washington DC and in New York city?” “Oh, we’re going to need you then.”
I think in the beginning of standing up US cyber command and training a bunch of forces that the Department of Defense was going to come in and rescue every issue that happened in the United States. And we’re not set up to do that. We’re not trained to do that. what we are is defending our critical assets, our war fighting assets, and to ensure that we’re able to wage diplomacy when needed. I think that was enlightening to learn as a lot of people initially thought that the Department of Defense would be called in to fix everything.
Bryson Bort:
Yeah. I think that’s certainly ties to a policy level understanding and thinking, when you first asked me that question, I was thinking more tactically. I work in research and development, and I think that’s an area where the DOD requirements can help drive a lot of this change, where we needed at a broader scale. And then again, with everything being interconnected at all of these levels, including critical infrastructure, that DOD has a leadership position if they wanted it.
Daryl Haegley:
Certainly, and a lot of the stuff that we need to be able to do is we need to be able to use cyber as a weapon. And that’s a unique thing that we develop and we train to do, one of the things that has been, I’m sure there are a lot of folks who are really good and excel at that. At the same time, we need to develop cyber for defense. A lot of times there’ll be a defensive, the DOD conference and day one will be offensive tools and techniques, and day two will be defense. And I get invited to speak on day two and day one. It’s standing room only, and day two, there’s a tumbleweed that goes across my podium, because there’s just not that much excitement.
And there’s this challenge of who’s going to pay for it? Are your facility people whose, accounts have been essentially postpone for eight years of doing maintenance, to pay for cyber securing of these HVACs and other utility systems, or is it going to be the IT budget, which didn’t budget for these non IT systems? Or does there need to be a separate pot of money, designated specifically for it? Who should pay for it? Should there be a set aside for mitigation? I don’t know about you, we’re all supposed to have a rainy day fund in case something breaks in the house, breaks in the car or breaks in your life.
Well, when we go out and do assessments of places, we will find vulnerabilities. And a lot of times we did not have a complete set aside budget to pay for all these mitigations. So we spent a lot of time figuring out prioritizing and getting the most important ones done first, which everyone would do, but in the grand scheme of things, is there something that should be set aside from an overarching policy level to fix things, that’s certainly an option worth considering.
Bryson Bort:
Well, look at that. Well, one little simple question that you mostly directed at me, and we ended up getting a whole lot more out of it.
Daryl Haegley:
All right. I do. I really do want to know what does industry think they need from DOD. I get your point, helping drive some of the R&D, and I’ve been in a lot of some conversations where they’re like, “Well, we need you to tell us what the bad guys are doing.”
Bryson Bort:
No, you don’t know. That I don’t agree with. The innovation tied to priorities is always helpful. I think where you were getting that, and this is what I was trying to talk about earlier with, there really is a natural requirement for public and private collaboration. And I think we, as a country are grappling with that on a larger level already with what happened with something like solar winds and then the exchange attack where our adversaries are using the asymmetric benefit of being able to cause massive amounts of economic disruption and loss with no clear penalty. I personally have been saying this for years, that the soft underbelly of the United States is the private industry and our economy.
And sure enough, guess what’s being hit. That’s where this is now started taking its toll. And this is the kind of thing where 50 to 60 years ago, it would have been very natural for DOD to have had a presence in that, because the way that would have been done would have been through a kinetic effect. Somehow when we’ve gone to computers and I don’t know.
Daryl Haegley:
Right. No, that’s a great point. One of the things I find very interesting is, and I really love when I had the opportunity to get in front of audiences and I would just ask, “Okay, we’re having a war game here, left side against the right side. Who’s going to go after the hardened IT systems first?” And then you’ll always have one person raised their hand because they’re smarter than everybody else. And then I say, “Well, who’s going to go after the control systems of the other side first?” And everybody raises their hand. So if this is what we know intuitively without any training from the Department of Defense, that we’re going to go after that soft underbelly you’re talking about, why aren’t we making this more of a priority? Why aren’t we asking how well we’re doing defending it? How well we’re doing resourcing it from tools, processes, and people?
Bryson Bort:
All right. Well, Daryl, thank you very much for joining me today.
Daryl Haegley:
I really appreciate the opportunity Bryson Bort, and this is great.
