‘More of an Art Than a Science:’ Behind the Government’s Effort To Measure Cybersecurity
“There’s no generally accepted, widely usable, scaleable, transparent way to measure cybersecurity,” said Paul Rosenzweig, a former Department of Homeland Security deputy assistant secretary for policy and senior fellow at the R Street Institute.
That’s due to many factors, he said, including the enormous complexity of computer software and the multiple ways that skillful hackers can attack it.
Without measurement, he noted, “cybersecurity remains much more of an art than a science,” bereft of the kind of objective parameters that are generally regarded as necessary to inform rational decision-making, especially in business.
…
“Computer code is amazingly complex,” he noted. A single device might run multiple software programs, each containing many millions of lines of code—much of it in open source libraries or programs the developers didn’t even write themselves. “Any single error could create a vulnerability,” Rosenzweig said. “The attack surface is impossibly large.”
And that makes it almost impossible to determine how secure a piece of computer equipment is. A mechanical device like an aircraft engine, can be tested by running it at unusual speed, or for very long periods of time, but that approach doesn’t work with code.
“They tried it with Huawei,” said Rosenzweig, referring to the center the U.K. government set up to test the source code of the Chinese tech giant, after security concerns emerged about the use of its equipment in the British telecoms backbone. But two years ago, the center’s oversight board concluded it could offer only “limited assurance” the code was secure.
“There’s just no good way to test something that is orders of magnitude more complex than the most sophisticated avionics engine,” he said.
And it gets harder still when you consider what it is being tested for. When you test a machine, you are testing it against the unvarying limits of physics—gravity, speed, mass. Testing the cybersecurity of a piece of software or hardware against an adaptive, learning adversary is a very different business.
“Steel doesn’t have an enemy,” Rosenzweig said. “Imagine trying to build a bridge if gravity kept changing the way it behaved to beat you.”
