Announcing the White Hat Cyber Forecasting Tournament
WASHINGTON (March 8, 2022)—Today, the R Street Institute, Red Branch Consulting and Metaculus launched the White Hat Cyber Forecasting Tournament, the first public crowd-forecasting initiative designed to shed light on the future of cybersecurity.
In applying the tools of crowd-forecasting to cybersecurity, R Street and Red Branch hope to generate accurate forecasts that will help practitioners and policymakers make better-informed decisions related to cybersecurity policy.
As Paul Rosenzweig, the project co-lead, put it: “We can’t do cybersecurity well until we have a better handle on what to expect. This initiative is a grand experiment to test whether the wisdom of the crowds can help us make better judgments about what lies ahead.”
Mary Brooks, another co-lead, explained: “The cybersecurity industry is fairly decentralized and relies on so many different types of insight and expertise—from software developers and information technology professionals to lawyers and government officials. This crowd-forecasting tournament creates an opportunity for these stakeholders and others to share what they know.”
According to Metaculus CEO Gaia Dempsey, “A weighted aggregation of public crowd forecasts synthesizes information from a potentially huge number of sources about the likelihood of specific outcomes. Probabilistic forecasts are weighted according to the best predictors with a proven track record of accuracy, and this empirical grounding makes them tremendously useful for certain types of problems.”
How It Works
The White Hat Cyber Forecasting Tournament will offer participants the opportunity to forecast on a range of key cybersecurity-related questions, including geopolitics; government and government policy; events and incidents; industry statistics; and ransomware. Forecasting for the tournament will be open until Dec. 31, 2022, and prizes will be allocated in June 2023 once all of the forecasts in the tournament have resolved.
Members of the public—including cybersecurity experts, policy aficionados and anyone else—are encouraged to join and issue predictions. There is no fee to participate, and prizes will be awarded to top-ranked participants. For more information on the White Hat crowd-forecasting process, see here.
The Technical Details
The crowd-forecasting process, managed by Metaculus, collects and aggregates predictions from a large and diverse group of experienced forecasters. The insights of the last several decades of forecasting research have shown that optimally aggregating predictions leads to a more accurate picture of future events than relying on a single subject matter expert or model. This technique represents a new type of epistemic technology, one that’s just beginning to get on the radar of traditional policymaking processes. A few key features of the system include:
-
The Community Prediction, which provides a recency-weighted median of all the issued predictions, whether they are binary probabilities or continuous distributions.
-
The Metaculus Prediction differentially re-weights and re-calibrates forecaster inputs based on their historical track record.
-
The Metaculus Tournament Scoring System provides quantitative performance feedback to forecasters on their accuracy and calibration, and produces a ranking of the top performers in any Metaculus tournament. The system takes into account the time forecasts are made (since forecasts made earlier are generally more difficult given less information), and their relative accuracy compared to the community median.
Research Goals
There are three main ways the cybersecurity forecasting tournament can help further research:
-
identify how and under what circumstances open-source, unclassified crowd-forecasting communities can generate accurate predictions.
In short: Will outcomes generated from the tournament be immediately useful?
-
identify how and under what circumstances open-source, unclassified crowd-forecasting communities fail to generate accurate predictions.
In short: Are there types of questions or outcomes in cybersecurity that are beyond our capacity to ask about?
-
identify what types of information the community has access to, what types of information we aspire to access, and what information is actionable and useful to decision makers.
In short: What is the subset of useful questions that existing data can answer – and where do we seek answers but have, at this point, insufficient information?
R Street and Red Branch Resources
-
“How Crowd-Forecasting Might Decrease the Cybersecurity Knowledge Deficit”
-
“Betting on Cyber: Offering an Analytical Framework for a Cybersecurity Crowd-Forecasting Platform”
About
R Street Institute is a nonprofit, nonpartisan, public policy research organization. Our mission is to engage in policy research and outreach to promote free markets and limited, effective government. We work extensively on both state and national policy, and focus on issues that other groups tend to neglect. Our specialty is tackling issues that are complex, but don’t necessarily grab major headlines. These are the areas where we think we can have a real impact.
Red Branch Consulting PLLC, is a homeland security consulting company. Its principal, Paul Rosenzweig, is also a senior advisor to The Chertoff Group. Rosenzweig formerly served as deputy assistant secretary for policy in the Department of Homeland Security. Red Branch provides comprehensive legal and strategic advice to companies, individuals and governments seeking homeland security, cybersecurity and privacy solutions for the challenges they face.
Metaculus is an online forecasting platform and aggregation engine working to improve human reasoning and coordination on topics of global importance. By bringing together an international community and keeping score for thousands of forecasters, Metaculus is able to deliver machine learning-optimized aggregate predictions that help partners set priorities and make decisions.
*For a detailed description of the forecasting process, including the requirements to be qualified for cash prizes, please see the Tournament Page and Metaculus FAQs.
