An economist is an expert who will know tomorrow why the things he predicted yesterday didn’t happen today. -Laurence J. Peter
It’s astoundingly easy to make predictions about the future of cybersecurity risk (and lots of people do!) but it is sadly much less common to revisit those predictions after the fact. The point here is not to point fingers and laugh at bad predictions. Forecasting the future is extremely hard and it should expected that even very well grounded predictions will regularly miss the mark. Instead, we think that being honest with yourself about how well a model’s predictions are performing is simply table stakes for correctly interpreting the forecasts in the first place.
With that in mind, we publish forecasting reports as part of our Risk Intelligence service on the likelihood of an organization experiencing an incident. A high level overview can be found here. One of our ground rules for creating these models in the first place was that we would publicly (and regularly) post at least a high level overview of how well the forecasts have performed in the past. This is the first in that series.
We produce forecasts for the likelihood of an incident one year in the future, so that is how we’ve structured our evaluation. Basically, we fit the same model to slices of the data that end further in the past, and then evaluate how well the forecasted trend aligns with reality. For example, only showing the model data through 2021 and then comparing the forecast to the actual, excluded data. The results are shown in the figure below.
The three evaluation slices are shown as the colored regions, and the overall “truth” is the gray curve. While the forecasts are all technically within the confidence interval for each slice, some years the model did much better than others.
The forecast from June 2021 forward is the slice where the model did the best. Then the model missed the steep acceleration from June 2022 (although it did predict an increase). The final slice from June 2023 forward sees the model catching up with the growth, but possibly starting to miss some flattening towards the end.
On balance, we think this is a fairly decent showing for a simple model. But as we noted, the point here isn’t really to demand or expect perfection. Every forecasting model, no matter how responsibly and carefully designed will be wrong sometimes, even by large amounts! We fully expect there to be future updates in this series where our model missed badly for one reason or another. That’s simply inevitable once you start trying to predict the future.
The important thing to ask yourself is this: are the people who you’re getting your predictions from showing you how well they’ve done in the past? Because we consider that simply baseline information that’s necessary for interpreting and using a forecast.