Hack the Plant Episode 9: On the Front Lines with Rob Lee
That’s Rob Lee, the CEO and founder of the industrial cybersecurity company, Dragos. They is pioneer in the ICS threat intelligence and incident response community. The Dragos platform provides cybersecurity technology providing comprehensive visibility to assets and a response plan to threats for ICS/OT assets. Before Dragos, Rob served as a cyber operations officer in the Air Force where he helped identify and analyze threats to industrial infrastructure – an issue that leaders in the UK, Saudi Arabia, Australia, Singapore are now wrestling with. He joins Bryson to talk about industrial control systems (ICS) and the key challenges to their security in an increasingly digital, interconnected, complex and automated world.
(Subscribe to Hack the Plant on Spotify or Apple, by RSS feed or search for it wherever you listen to podcasts.)
Transcript:
Bryson Bort:
I’m Bryson Bort and this is Hack the Plant.
For today’s episode, I’m joined by Rob Lee who is the CEO and Founder of the industrial cyber security company Dragos, Inc. He is considered a pioneer in the ICS threat intelligence and incident response community. The Dragos platform provides cybersecurity technology providing comprehensive visibility to assets and a response plan to threats for ICS/OT assets.
Before Dragos, Rob served as a cyber operations officer in the Air Force where he helped identify and analyze threats to industrial infrastructure – an issue that leaders in the UK, Saudi Arabia, Australia, Singapore are now wrestling with.
We’re here today to talk about industrial control systems (ICS) and the key challenges to their security in an increasingly digital, interconnected, complex and automated world.
Rob Lee:
I usually tell people the threat is worse than you realize, but not as bad as you want to imagine. This isn’t like, “Oh my God, we’re all going to die.” But at the same time, yes, it’s getting a lot more real than people realize. We’re seeing state actors breaking into facilities and stealing property and prepositioning, and learning, and doing that type of stuff.
Bryson Bort:
Why are ICS systems such a big deal? They’re everywhere.
Rob Lee:
You probably work at an industrial company. Everything outside of financial sectors is industrial, data centers, HVAC systems, utilities are obvious, mining, rail, oil and gas, chemical. It’s not just the big industrials, you start getting into anything and everything has OT and can be an industrial system.
Bryson Bort:
OT refers to “operational technology” – hardware / software that monitors and controls physical devices and processes within ICS. As Rob points out, this is a critical piece of the day-to-day functioning of our society, business, and government.
Rob Lee:
There was a point in time when our operators and engineers understood everything in and out about that facility and that control environment. But, because of this increasing complexity and because of this increasing automation…..nobody really knows all the ins and outs anymore. They’re lacking in the details, and the data, and the things they need to know what’s really going on. The ability to get things like root cause analysis of what really happened any why so we can do better next time, is damn near impossible in some of these facilities.
To be fair to our asset owners and operators, that’s what’s been pushed on them. They are amazing when you sit down with these folks that are keeping the lights on, keeping the rail on time, et cetera. They just need some more help.
Bryson Bort:
What are the key issues at stake in keeping our industrial control systems safe? Join us for a deep dive.
(music)
Bryson Bort:
All right, Rob. You are one of the cofounders of Dragos…….What do you all do?
Rob Lee:
From a company perspective, our main focus, from a revenue line perspective, from what our company does is our technology. The Dragos platform is technology that we deploy into those operations and industrial networks for the purpose of giving the folks of the company insight to what they have, so it helps you create your asset inventory. It automatically passively and safely discovers the devices on the network, understands the control system communications, figure out what is there. That helps a lot from with everything from supply chain discussions, to resilience, to just needing an asset inventory for pretty much everything that you do in security.
But then, on top of that, where we go a little bit further than what’s normal in the market is we also have our analytics, which are just think of it as advanced queries, looking across our system for various industrial-focused threats that we track and understand. And then, knowing that a lot of companies don’t have ICS security people, we also have playbooks with each one of those detections that pop up and says, “Here, this is what a step-by-step process looks like to investigate and respond to this detection.” That platform, vulnerability, visibility, threat detection response category.
The way that we approached it is really unique where we approach it by the mindset of having a lot of smart people. I’m good with machine learning and stuff like that, and that can be in the product. But I think a lot of times, what we see in product companies is an over-pitching of how, as if that’s the differentiation, like, “Oh, we use blockchain,” or, “We use AI.” For us, we’ll look at any different tech stack and see what makes sense for us. The reality is, hire and employ a bunch of smart folks from the community, that hands on experience, and codify and productize human expertise.
What a lot of folks know us for, beyond the Dragos platform, is we also have an intel team that goes out and hunts those threats, and tracks them, and reports on them. We also have a services team that does everything from red teaming and architecture assessments to instant response and hunting. It’s that combination of services, intel, and product that allows us to be a really good partners to our customers and on their journey of industrial.
Bryson Bort:
You said, “Safely discover assets.” I want to pull the thread on that. I mean, the start in the industrial control systems space, like information technology, was in the 80s and 90s was configuration management. What the hell do I even have? That’s a particular challenge in industrial control systems. Can you talk about that and how we’ve moved from a passive to a hybrid discovery process and why that’s important?
Rob Lee:
Yeah. for decades the strategy for industrial security was prevention only. Segment the power plant. Segment the rail line. Segment the HVAC system. Segment things from the other networks and the internet. And occasionally put up a firewall and to keep it tuned. Then, okay, well let’s also do patch management. Then, okay, let’s also do anti malware systems and antivirus. Everything was prevention, prevention, prevention, with the leading strategy of segmentation.
It’s not that prevention is a bad strategy, it is a starting place. I think it’s inadequate unless you have prevention detection response. But the real weakness of prevention is, if you don’t have detection, your prevention is deteriorating over time. You’re not keeping your firewalls tuned. You don’t really know what’s in your network.
Bryson Bort:
Security through obscurity, you can’t hack what you can’t touch, but… internet connected ICS and not truly air gapped with the enterprise. the perimeter is dead. in security, we call this model, “assumed breach” because a determined adversary will get inside your network. from there, it’s a question of whether you can catch them in time.
Rob Lee:
As integrators and original equipments manufacturers and these vendors of the industrial systems needed more access to those systems to provide more efficiencies and business value, as these companies started down their digital transformation journeys where they are using more cloud, and analytics, and things to increase reliability and safety, but also business efficiencies, as all that’s happened, what we have is a hyperconnected operations technology environment.
The strategy of segmentation, which was the leading one and backed by every framework, standard regulation, et cetera, is definitely no longer sufficient, but it’s still the predominant strategy. Then, you look at the fact, when something happens, we don’t have any ability to detect or respond to it. You can’t be resilient if you don’t know what’s going on. And so, that’s where the community’s starting to push a lot more people. That starts with, “What the heck do I have?”, which goes to your topic of visibility. In that topic of visibility, beyond the people weren’t doing it because that wasn’t the focus, when you actually get down and try to do it, a lot of your operators, and engineers, and folks in that environment, they want an inventory too. They’re cataloging things and Visio diagrams, and Excel spreadsheets, and whatever, just for a general maintenance purpose. But it’s a lot of walking facilities. This isn’t an IT network that’s centrally located. These are plant, by plant, by plant physical locations. Walking the systems, hard hat and knee pads, tracing wires, trying to get access to where the stuff is.
On top of that, then gets into the passive discussion, this entrance into the market of, “Well, let’s have ICS, or OT, or whatever you call it, industrial specific technologies that understand the protocols, and the communications, and the networks, and the ICS equipment, so that I can automatically start discovering that.” That was great. That’s that passive discussion of we’re not going to send data to controllers. We’re not going to send queries. They’re not going to scan stuff. We’re going to listen and analyze the traffic and come to those discussions.
What then started to happen is where you’re going to is some companies wanted to start positioning this active scanning. They tried to pitch it in various ways, which probably seem friendlier, like, “It’s not active scanning, it’s friendly polling,” or safe polling, or whatever else. But they’re still talking about sending data to these systems across the network in such a way to try to profile them even more, or find firmware versions and similar. What we’ve seen time and time again is the broader market talked about it for two to three years as, “You have to be doing this.” At Dragos, we never did. We never went that route. We always stayed in passive. I’m not saying it’s impossible to do. I think it’s actually perfectly fine in an assessment, in a maintenance period of the plant, et cetera, to go through and with training, with the right tooling, start querying data actively.
What I have found time and time again is all the people that say, “Of course you can do this,” they don’t know every industrial environment and how it’s configured. I have seen numerous times where that active scanning, or polling, or whatever, takes down those sites. And so, it’s not like the scary boogeyman that’s like, “Somebody said this might happen.” We have dozens of cases where this happens, where somebody deploys a product that can actively interrogate or discover these devices, and it starts tripping up a controller, or causing latency on the network, or the system doesn’t know how to handle it. Even using native industrial protocols, they may not be completely integrated correctly. Or you use a wrong native protocol on the wrong system, and the next thing you know your power plant is going offline, because you made a mistake in the controllers.
On top of that, and I don’t want to go on too long of a tirade here, but on top of that, the first incident that happens at a site, regardless if it was your product’s fault or not, when the engineers and the vendors start looking at that site, and they go, “Hey, what’s that box hanging off the network?” “Oh yeah, it scans the network.” “Well, I’m not touching this until you remove that.” It gets blamed, even when it’s not actually at fault. And so, what we’ve found is that this let me actually query kind of data on the system actually poses risk, has been the result of more power outages than Russia, China, Iran combined, and culturally is a good way for security to get kicked out of those facilities and not invited back. So, I just generally don’t encourage it at all, unless you’re doing it from an assessment perspective with permission and a maintenance or similar. What we found is passively doing it gets 80%, 90%, of what people are wanting. That last 10%, then they can just load it into the system themselves without ever risking operations.
Bryson Bort:
How do folks not know what they have on their network?
Rob Lee:
You see these industrial environments that change quickly. And so, there’s this idea that these industrial environments are super stable and they never change. That’s just not the reality. We don’t have all the Facebook, and YouTube, and Google connection that you would have in an enterprise network. Integrators, equipments manufacturers, et cetera, are constantly moving the needle on what can be done in these environments. That means that changes in inventory, and updates, and new equipment, and similar happens. When you do that at not one plant, but you do that at a company that has 3,000 substations, or 100 transmission substations, or 16 water treatment facilities, or whatever else, it just becomes a very complex problem at scale. Knowing every piece of network equipment and how it’s operating and how it’s working, it’s just very difficult to do.
What we tend to find is there was a point in time when people had an understanding of what was in their environment, not how it was communicating, not how it was connected, but had an inventory, not a topology, as we would call it, of what this really looks like in real life. They might’ve had an inventory at one time, but usually it’s in an Excel spreadsheet, or a Visio diagram, that’s five, 10 years old, and it’s wildly out of date with what the reality is there. That also goes back to my comment on prevention. The value of prevention decreases unless you’re also doing the other things as well.
Bryson Bort:
Yeah, I like to describe that as security is a constant fight against gravity. I want to make sure I’m clear on this. You’re saying that industrial control systems, that those environments should not allow access to Tik Tok.
Rob Lee:
Yeah. That’s generally the policy. I would tell you that there are plenty of places that we go, that, of course, is not the reality. Maybe not Tik Tok, but I’ve seen a lot of sites you go, and they’re like, “We’re air gapped.” And I’m like, “You’re streaming Hulu.” They’re like, “Oh, yeah. No, I guess we have internet.” Okay, guys.
Bryson Bort:
And air gap is not an air gap, is not an air gap. This is something we’ve covered on the show a few times, but I think it’s worth bring up yet again. What is an air gap? What should an air gap be? And what really is an air gap?
Rob Lee:
Yeah. A real air gap is when there is no ability for communications to flow in and out of an environment. It’s actually disconnected from all these other networks. There are very few places you’ll ever find a real one. In the nuclear power industry, you will find real ones, where they truly have, in their nuclear network, as it relates to operations and control, zero interactive connectivity. What we then see as a step out from that is people go, “Well, I don’t want interactive, but I want to be able to see what’s going on, and so I’ll deploy a data diode, this product that will allow a one-way transmission of data.” At first, that was okay for nuclear power. It’s not realistic for these other companies that move data in and out of environments though.
The next thing they do is, “Oh, well I want a diode on the way in too.” It’s like, you’ve expanded that communication, not minimized in the way you really thought you did. The next level that people claim are air gaps is when they have a well-tuned firewall in front of their plant. That’s definitely not an air gap. We still see people call it that.
And so, I guess my point is a true air gap is zero connectivity. What we see people pitch air gaps as is various levels of connectivity still calling it an air gap. There’s not problems with reducing the amount of connections in and out of those environments. I think, creating choke points, these natural places where there’s limited connectivity, but you can monitor what’s going on, that’s perfectly reasonable and a good way to approach connectivity. But, when you don’t allow connectivity or you’re very restrictive on it, it’s very Jurassic Park. Life finds a way. And so, we’ll say operators and engineers setting up their own wireless access cards. You’ll see people bypassing the things you put in place, because they’re actually adverse to operations. It’s adverse to the business operations as well. It’s not very realistic to do. It is okay to limit connections, but you will always have connections as long as you’re, again, outside of the nuclear power industry.
Bryson Bort:
Let’s get into your background. What led you here? Rumor is you went to another service academy. I’ve heard there’s one besides West Point?
Rob Lee:
I started my education at the Air Force Academy. I didn’t really know what it was about and didn’t really have any intentions of going there. I didn’t really fully understand it. Both my parents were enlisted, senior master sergeants. My dad was an old Vietnam guy. I was going to go to Auburn. I’m from Alabama originally. I was going to go to Auburn or Alabama. He poked me in my junior year, and was like, “Yeah, you couldn’t get into the Air Force Academy anyways.” I’m like, “What’s the Air Force Academy, dad?”, like this naïve crap. He’s like, “Oh, no, no, no.” He just schooled me, reverse psychology, and was like, “No, no, no. You couldn’t in anyways.” I’m like, “All right, well, I’ll try,” and I got in. I was like, I guess I’ll go give it a shot. I’ll go for six months to a year, and if I don’t like it, I’ll leave, which is not a real thing.
And so, when I went there, and I get off the bus, and they’re yelling, and shaving their head, and PT, and physical training, and beating the crap out of you and stuff, you’re like, “What is this?” I don’t know. I don’t know that I perfectly fit in, but I was too stubborn to leave. And so, I ended up graduating. Along the way, probably what the most formidable thing for me was, on summers, you could get to go fly planes, or you get go home for once. Neither of those things were interesting to me. They force me to go fly in an F16 in Korea for training purposes and crap. I was like, “Okay, that’s nice, but this is not what I want to spend my life doing.”
Most of my summers, I was like, “Crap, can I go somewhere cool?” They’re like, “Oh yeah, there’s this Engineers Without Borders. Do you want to go do engineering humanitarian work in Africa?” I’m like, “Yes, that sounds great.” So, I would spend time in places like Cameroon building water filtration units, and wind turbines, and things like that. Mostly, there was so much good work going on, well beyond us, but we would contribute a little. It was very cool to see these things that we call control systems and the value they bring to society, and keeping the lights on, and creating a little micro-economy is to help communities. Yeah, that kicked off my love for control systems. When I found there were people that were interested in hurting people and attacking these things, it just seemed really offensive. I was like, “Dude, I’m going to go protect these things, because that’s what’s going to protect people.”
So yeah, after I left the Air Force Academy, they kicked me off, they said, “Hey, do you want to be a pilot?” I’m like, “Not at all.” They said, “Well, you go be cyber then.” I’m like, “What’s cyber?” They’re like, “I don’t know. Go do it.” They put me through the schoolhouse, put me through the ringer, et cetera. I picked up a lot of stuff, and then got shipped out. Long story short, landed at that NSA. They were like, “What do you do?” I’m like, “I don’t know, cyber.” They’re like, “That’s not a thing.” I’m like, “Okay, what do you want me to do?” They’re like, “Find the new threats.” I’m like, “What are the new threats?” They’re like, “I don’t know.” I’m like, “Okay, this is really well formed out.” I was like, “Why don’t we just focus on control systems?” They’re like, “What are control systems?” I’m like, “Oh, god. Okay. We’re going to look for threats against control systems.” Along that path, I ended up building out and leading the US government’s mission on identifying state actors breaking into industrial sites around the world. That’s what kicked it all off.
Bryson Bort:
Any particularly interesting stories or clients, of course anonymized, that you can share about?
Rob Lee:
From a Dragos perspective, we always go into the coolest places. All these different industrial operations are just so beautiful and so wonderful to see what they’re doing in their communities and impact the global in-trade and everything else. Yeah, there’s plenty of craziness. We had an instant response case, as an example, where the … change across the system that caused enough noticeable change from a physics perspective, consider electric power outage type stuff, but not outage, maybe voltage change. That it was a big enough thing to scare company and probably something that was going to be reportable. They had no understanding, none. They don’t have the forensics in the environment, they don’t have logs, they didn’t have visibility like most of these places. The company wasn’t doing anything wrong, that’s just been the strategy.
And so, they were really thinking this could be a state actor or something, like, “Oh my gosh, what is this? We did not send those commands across the system to those controllers. Something is going wrong across multiple systems. This is exactly like we’ve seen in different attacks before.” As we go in and just deploying our tech and getting in front of visibility [inaudible 00:21:52], what we were seeing was there’s some old, crazy, bug in the software that just never met all the right conditions before that was causing voltage change across electric systems, across multiple substations, just off of an issue in software.
It’s not a crazy story. I mean, it is crazy when you think of the impact. But it’s not so much a crazy story from that end. What’s crazy to me and what’s the mattock of what we’re seeing is there was a point in time when our operators and engineers understood everything in and out about that facility and that control environment. But, because of this increasing complexity and because of this increasing automation, and this increasing things that drive so much value, what we’re seeing is nobody really knows all the ins and outs anymore. They’re lacking in the details, and the data, and the things they need to know what’s really going on. The ability to get things like root cause analysis of what really happened any why so we can do better next time, is damn near impossible in some of these facilities. We’ve seen that type of case multiple times every year, huge, huge, things. And it’s not state actors, it’s complexity.
Now, of course, we do see the state actors way more. I usually tell people the threat is worse than you realize, but not as bad as you want to imagine. This isn’t like, “Oh my God, we’re all going to die.” But at the same time, yes, it’s getting a lot more real than people realize. We’re seeing state actors breaking into facilities and stealing property and prepositioning, and learning, and doing that type of stuff. Those cases, to me, aren’t as crazy as, “We just don’t understand our system anymore. I think it might have societal impact. What should we do?” That, to me, is just otherworldly. It’s pretty awesome. Again, to be fair to our asset owners and operators, that’s what’s been pushed on them. They are amazing when you sit down with these folks that are keeping the lights on, keeping the rail on time, et cetera. They just need some more help.
Bryson Bort:
You recently announced that Malcolm Turnbull, from Australia, is investing. It seems like we’ve hit a turning point in how everybody’s starting to get that this is truly important to think about.
Rob Lee:
Wildly so. When I started my career in industrial control systems, I think it is fair to say that there were other industrial security people that had spent time and effort to really raise the discussion. There was a budding community forming, still you can name everybody in it kind of feel, but it was budding. Outside of that very small group of people, and we all knew each other, that was it. This idea of, what’s industrial? What’s OT? What is this stuff? That was the common question across everywhere, from policymakers to businesses, everybody. Even when I started on the agency side of the house, it was, “Why are you doing that? What is the value? There is ICS specific threats,” or, “They’re all disconnected systems anyway. Dude, why are you wasting your time?”
As we’ve started to collect in those environments and take a look, and go, “Okay, well, let’s look inside. What’s actually going on?” We start seeing all this activity. It’s raised the discussion of, “Oh, there’s non-security value and there’s security value to this. And, there are real security risk. They journey we’re on is that digital transformation convergence with ICS threat landscape convergence, to IT, OT convergence. That’s already happened. This is a path that’s worrisome.”
I think it’s fair to say and I hope it’s not arrogant, but I’ve spent a lot of my career trying to educate and help people. And so, sure, I do my stuff over at Dragos, but I still teach at SANS, for example. I have spent years just going to the Hill in Congress, and taken a whiteboard, and talking to staffers. And “Hey, you know there’s not one electric grid, right? There’s multiple ones. Here’s how electrical power works in the United States.” Or, “Hey, I know you think our asset owners aren’t doing a whole lot, but here’s actually all they do. And here’s the challenges they face. You know what? The government doesn’t even have its answers together. And here’s all the crazy stuff all of your agencies are saying to these CEOs and stuff.” You just spend a lot of time educating as others have done.
Over that time, with a very community-wide approach, we’ve seen the raising profile of OT. Because, if you look back, even dating to 1998, when President Clinton issued presidential directive 63, I believe it was, talking about critical infrastructure can be vulnerable to cyber attacks, and designating sector specific agencies and all, the discussion was always critical infrastructure. What they were trying to articulate, but didn’t, was, “We really care about health and safety. We really care about these things that can impact national security or kill people. That’s somewhere we should focus.” What that expanded into was everything’s critical infrastructure, and even casinos, now, are critical infrastructure. That critical infrastructure term became super conflated on everybody’s critical if you think about it. We got further abstracted from industrial.
What we’ve seen over the past couple of years, and really, really, in the last two years is a much more hyper-focus on, well, hold on, when we say critical infrastructure, we’re not talking about your email servers and similar. That’s important and we should do that. What we’re really talking about is life and safety, and national security. Can you have an oil spill? Can you have a chemical explosion? Can you have a power outage? There’s a better awareness at the board level and CEO level of these companies, and now in Congress and around the world, that what we’re talking about is OT. That, yes, your enterprise securities and your enterprise systems are important. And yes, enterprise systems can impact OT. We care a lot about it. But we’ve under invested in OT, and that focus is something that we need to do, because that is what we’re talking about when we’re really say and talk about the critical infrastructure component of these industrial companies.
Anyways, so that profile was raised to the point that Anne Neuberger, an extremely impressive individual who is the White House deputy national security advisor for cyber and emergent technologies, so the president’s cyber person, she’s in keynoting the SANS ICS Security Summit talking about how the president is focused on OT security and actually cares about OT. There’s an action plan that they’re preparing about the White House to help the community. You see the national security advisor tweeting out about security. You see multiple congressional hearings where they’re asking DHS and others, “What do you think the top concerns are?” “Number one, supply chain. Number two, control systems, because that’s what can kill people, and that’s where we care.”
Then, you see Singapore standing up their OT master plan. You see Saudi Arabia focusing on it. You see UK focusing on it. You see Australia investing on it. To the point now, we see world leaders. We have Malcolm Turnbull, the previous prime minister of Australia getting up in front and saying, “OT security is necessary. It’s different that IT security, because of the threats, risks, missions, everything around it, and we need to have a dedicated approach.” That is insane. I’m sure there’s new people entering the field. They’re like, “Oh okay, yeah, that’s exciting.” But anybody who’s been here a minute… I’m not trying to separate that out at all. Please come, we need more people. For anybody that’s been here for a minute, that is insane. That is so different from where we were five years ago let alone 10. It’s just so exciting to see that there’s real focus on it.
My advice actually is usually to the CSOs, where the CEO and board are having conversations about OT security at a world leader community, whatever, level, and they’re concerned about it. There’s a number of CSOs who look at that and go, “Oh okay, well, I’m going to take my enterprise security strategy and then copy and paste it into the plants then.” That is a good way to never get invited back and to not do a whole lot at all.
And so, what you’ve seen is CSOs and CIOs, and there’s plenty of them that are actually amazing on this topic, so I don’t want to paint a broad brush here, but there are some who don’t know anything about the operations of business. They know nothing about operation systems. And so, they came from backgrounds of IT. When they look at the problem, they want to talk about patch management, and vulnerabilities, and AV, and things like that. It’s not actually addressing the real problem. And those CEOs are having real conversations of, “Man, are we not going to be able to get this right with people we have? That seems weird. What do we do about this?” And so, it’s just an amazingly interesting conversation’s happening at all levels right now.
Bryson Bort:
You mentioned IT/OT convergence. What is that, and why is it happening?
Rob Lee:
Yeah, I mean, IT/OT convergence is something that gets thrown out a lot. I would note, it’s actually not happening, it’s already done in many cases. But, this idea of IT/OT convergence is, “Oh, we’re getting IT systems in the operations environment. We’re seeing some more connectivity from the enterprise to the ICS.” It’s this idea of it converging together, the enterprise and the ICS. The reason I say that’s already happened is we already… If you dissect the terms OT and ICS, ICS is historically related to the actual control systems instrumentation, really the control elements around a lot of the physics and what was happening out the field of valves, and actuators, and things like that. OT was really mission critical IT. Windows system is populating in a plant, but it was running an energy management system, purpose built software to interact with transmission substations and similar. It wasn’t an IT system. You couldn’t treat it like an IT system. It didn’t have the same risk profile. It had different things that you could do to it. And so, we called it OT.
Now, we really just talk about OT as the whole thing to help, at a macro level, of communicating about the differences about OT. This whole ICS versus OT nuance is really just in the community to debate stuff that doesn’t really matter. Anyway, long story short, the idea of IT and OT converging, to me, is a little silly, because that was the whole point of OT to begin with is, yes, we already have IT in these environments. Again, that’s 15 years ago, plus. And so, don’t view it as if there’s this IT/OT convergence. I view it as that’s already happened.
What I articulate to people is there’s really this digital transformation journey that these companies on. It’s more than just connecting into an enterprise. You’ll have sensors in the production environment connecting directly to cloud. And so, it’s not just about connectivity, though there’s a big of hyperconnectivity, but it’s also about how do we get that data? How do we use that data? How do we increase efficiencies? How do we scale it in a way that impacts the workforce? How do we provide more reliable and safe services to our users and consumers? It’s a whole transformation of the business and the industrial environments. That transformation is introducing complexity. It’s introducing hyperconnectivity. All of that converging with a threat landscape out there of actors, and state actors, and similar that are interested in industrial environments, and focused on them, and learning, and developing capabilities, it’s that convergence that’s causing a lot of risk that is causing a lot of people to get nervous. If prevent only was your strategy you’re going to fail. It’s that mix that we’re seeing.
Bryson Bort:
How do we get more people into the space?
Rob Lee:
Yeah, I mean, first and foremost, you have to make it welcoming. And so, when you look at InfoSec, and you look at a lot of these social media interactions and similar, it’s a surprise that we have any women in this space. It’s get extraordinarily toxic. Women, and people of color, and similar, and diverse candidates go through way too much crap. If you look at that, you don’t want to join. There’s plenty of guys that look at that and go, “Well, if they can’t even treat people respectfully, then I don’t need to be part of that either.” And so, I think InfoSec, from a social media perspective and in the real world too, I don’t want to displace that it’s not there, but I think it just gets amplified on social media, is already causing problems for InfoSec.
When we look at industrial security, first and foremost, we should learn from InfoSec and go, “Hey, if there’s jerks in our community that act like that, let’s kick them out fast, blacklist those folks fast. Because, first and foremost, we want to be welcoming to everybody. Because, it takes everybody.” I mean, you’re going toe to toe with state actors. You’re going into hyper-complex environments. The last thing you want is group think. Diversity and diversity of thought are going to be extraordinarily important to doing that well.
The second piece is you have to make an onboarding path where there has to be training and certifications. There are various thoughts around certifications, but you have to have a professional development track. We’ve spent a long time over at the SANS Institute building out that ICS curriculum as one path. Not the only path, but one path productizing and professionalizing on education for, “Hey, here’s your ICS 410 class to come learn about ICS. You’re from IT, come on over.” Or, “You’re from engineering, come learn about these things.” “Here’s this 515 class about learning how to respond and hunt in these environments.” “Here’s 612 about getting hands on with industrial process and controllers, and really learning.” And so, that path has seen thousands of people get familiar with OT and ICS in ways that never existed before at scale.
But then, there’s got to be something after to go get those folks that are now trained and invested in themselves and opportunity to get into an entry level job into our field. What I see a lot of the conversations on is around training, training, training, training, or so forth. What I hear from a lot of my students is, “I can’t find a job that’ll take me without 10 years of experience.” Normally, what I communicate to folks is go find your local manufacturing facility, go find your local utility, your local oil and gas company. They’ll onboard you. There’s a lot of those jobs that they can’t pay the higher InfoSec salaries of competitive startups, but they’ll invest in you and they’ll train you. You’ll learn industrial operations. You’ll get into those environments. Some of our best community members came from those backgrounds. That’s a great way. But, we need to have a more concerted focus on that.
We talk about government’s involvement in that and similar, there’s got to be impact, and influence, and investment in getting people into those jobs, when those jobs exist. I disagree about this giant workforce gap. “Oh, there’s three million jobs that are unfilled.” No, no, no. There’s three million job posts that are poorly worded with asking seven years in experience on a technology that’s only been around for three, and all these principle jobs when there’s no juniors. That’s a gap we’ve got to take on. It touches from HR systems, to the way people are trying to onboard folks, to how you do professional development once you’re in your job. Long story short, there’s lots of areas to focus, but one of the biggest, in my opinion, is helping people get into the workforce, and helping those companies better understand what good job descriptions and similar look like to onboard those people.
Bryson Bort:
If you could wave a magic air gapped wand, what is one thing you would change?
Rob Lee:
Scoped to the world or industrial security? I mean, world hunger is probably a pretty good one, but what scope would you like?
Bryson Bort:
Considering that the podcast is about industrial control systems, why don’t we limit our magic to that space?
Rob Lee:
You’re like, “Look, I just gave you a magic wand, don’t be greedy.” Yeah, okay. I got, you. On industrial security, I’ll give two different options. Number one, I really think we have a giant collection gap in the community of knowing really what’s happening in these industrial environments. And so, I broadly call that visibility, but it’s not just in inventory, it’s what’s really happening, what’s really occurring? Vendors are abusing their software license agreements. Adversaries are taking advantage of connectivity. How bad is it and how bad is it not?
If I had a magic wand on the security discussion, I would actually start with, let’s expose the real risks. What is really taking place? That’d give us perfect visibility into those environments. And then, companies, and governments, and everybody else can make informed choices, instead of, “We have to worry about this systemic threat.” Do we? Is it actually real? Let’s take a look. Or, “Hey, here are some real threats and we need to address these as a community.” This is a very smart engineering operations focused community that, when they know what the problem is, they’ll get it done. They’ll invest in it. Now, if they understand the real risk, these companies aren’t shy about hiring people, and training them, and doing it. They just have to understand it. I think that starts with visibility not a risk management discussion. You really got to know what’s really going on. So, that’s the technical one.
The personal one, I see a number of folks that get looked up to as ICS folks, people that have been around for a while, that for whatever reason have become fairly gate keeping/toxic. Like, “Let me tell you why everyone else’s stuff sucks instead of telling you why mine is good.” It goes back to that diversity in my toxic discussion. I would like to see, especially the people that get looked to, especially those that want to be in a though leadership position or whatever, spend more energy doing the mission and less energy complaining, and crap posting each other, and all this other stuff. I just don’t think that’s going to get us to where we need to go. I think it’s very off-putting to new people that want to come into the field to go, “Man, you could be a respected individual, and some other respected individual can just blast you publicly in blogs, and videos, and stuff for weeks without you even saying anything to them.”
I know this is a very specific example, but this happens a lot, and everybody goes through this through people in their own spheres. It’s that external pressure is unnecessary when we’re already going up against real challenges in real hard environments against real adversaries trying to do us harm. We don’t need to be fighting each other as well. And so, I wish, kind of a Bill and Ted, I wish we could just be excellent to each other. That’s probably the personal one.
Bryson Bort:
You waved your magic wand, now look into your crystal ball. Five year prediction, one good thing, one bad thing.
Rob Lee:
Yeah. One good thing is this focus of OT that we see, as it relates to those government discussions, and private sector discussions, and similar will help reveal what the problems are more than ever. I do believe our community will come together and make massive improvements in ways we’ve seen before. We are already doing so much better in ICS security over the last decade than in the previous decade, but I think it’s going to speed up very, very, quickly, once we start understanding these problems. And now, with the focus in board level discussions and similar, I think we’re going to see some real improvements that relate to health and safety.
I believe it is possible, maybe not in five years, but I believe it is possible not to solve security, but to make it where it’s impossible to kill people through a cyber attack, that should be the goal. How do we make that not at thing? Then, go work on the security beyond that about protecting intellectual property and things like that. If we can truly make it impossible or highly improbably for state adversaries and similar to hurt civilians and to kill people, sign me up. That’s where I think we’re going. That’s where I’d think it’d be awesome.
The one downside is, at this time that we’ve got this hyperconnectivity, at this time that we’ve got this journey coming, I think it’s going to get worse. We’ve got a lot of complexity built into our environments with little understanding of it. We are consistently copying and pasting IT security controls into ICS without understanding what they’re supposed to do or the value they’re supposed to bring. There’s plenty of them that are useful, but wrongly applied or applied for the wrong reasons that aren’t going to be effective. You have a lot of CSOs and teams that don’t know anything about industrial operations. Some of them are going, “Let’s learn. Let’s go take a case of donuts and talk to our operators.” Those teams are going to succeed. There’s going to be a lot that go, “I don’t have enough time in the day to do all the stuff I already need to do, so I’m just going to copy and paste what I’m already doing,” and they’re going to fail.
And so, you’re going to have a separation take place in the community, in my opinion, where there’s going to be some companies that really get it right and push the envelope in ways we’ve never seen. You’re going to have some that go further to the bottom than they already were, where they’re doing the wrong things at the same time or getting that hyperconnectivity. It’s in those places that I’m concerned that a lot of this early reconnaissance and learning-type activity by these threats we’re seeing are going to start manifesting in more attacks than we’re seeing today at time that we’re more connected, and interactive, and complexity. I think we’re going to get to a real bad place for quite a few companies.
That divergence in maturity is something that I’m not looking forward to, but I believe it’ll rectify itself as the community can start looking at, “Well, that’s what right looks like. Okay, well, let’s go share those lessons learned and let’s all do that as a community.” I think we’ll get there. Within a five year window, I am a bit concerned about the divergence.
Bryson Bort:
Grab bag, any last thing you want to talk about?
Rob Lee:
No, man, I think it’s all good stuff. Again, for anybody out there listening about industrial security and similar, then I would just say the water’s warm, come on in. It’s a perfect time to get involved. For those of you that are like, “Oh, well there’s not ICS security jobs in my company,” well, you probably work at an industrial company. Everything outside of financial sectors is industrial, data centers, HVAC systems, utilities are obvious, mining, rail, oil and gas, chemical. It’s not just the big industrials, you start getting into anything and everything has OT and can be an industrial system. Just try figuring out where that is in your company and see if you can get your leadership or whatever to support some assessments, where you go in yourself and start taking a look. Or go talk to the operators and engineers and say, “Hey, I’m here to learn. I don’t really have anything to contribute just yet, maybe I will. But I brought food, and I’m super curious about your operations.” Go build those bridges. It’s a great way to then start getting that experience and also furthering the mission at your company.
These are cool companies. If you’re in one, even if you’re not in an industrial security role, I promise you, there’s a route to doing more in industrial, and we massively welcome you to the community. It’s a great community. Come on over.
Bryson Bort:
Rob, thank you for your time and your insights today.
Rob Lee:
Yeah, thanks, man.
