Measuring Ransomware, Part 1: Payment Rate

This blog series is sponsored by Secure Systems Innovation Corporation (SSIC), a cyber risk advisory services firm that empowers business decision making. To learn more about SSIC and its support for this research, please visit their website and/or related blog post on this topic. Posts in this series: Part 1: Ransomware payment Rate; Part 2: Ransom Demands; Part 3: Ransomware Prevalence

Usually, a blog series like this would begin with a description of the threat in focus, a timeline of events, some grave-sounding statistics, and perhaps a fear-inducing quote from a security expert. But let’s face it – everyone in your extended family dinner discussions has heard about ransomware at this point. We can skip all that.

Even though ransomware awareness runs everywhere, consensus around its prevalence, payment rates and amounts, and overall consequences does not. We’ve noticed some sharply divergent statistics bandied about over the last couple years, leaving us wondering what’s true truth vs fake news. Noticing the same issue, SSIC approached us about setting the record straight on ransomware risk factors, and we eagerly jumped at the opportunity.

What follows in this series is a meta-analysis of ransomware risk factors based on public research reports, blogs, and articles on the subject. I’m kicking things off by examining the payment rate, or percentage of ransom demands that are paid by victims. It is admittedly an odd place to start; beginning with the frequency or prevalence of ransomware seems more logical to me. But sometimes you have to follow a muse, and Tony Martin-Vegue set the stage with his blog post titled “The Birth of a Ransomware Urban Myth. In it, he investigates a study that claims 38% of ransomware victims pay the attackers and determines that it’s pretty much bunk (along with a lot of other “research” that is filled with agendas, yet devoid of sound methods).

While Tony makes a convincing case that the particular research in question is less-than-reliable, the question remains about whether the 38% statistic holds any water on its own. How far off the mark is that? Is it high or low? A lot or a little? How would we even know? What do other sources say about it? Which ones can we believe?

Of course, these questions aren’t limited to ransomware payments – they’re near universal in the cybersecurity field. We have an ever-growing set of research available (see the Cyentia Library for examples), but gleaning reliable information from them to drive decisions and practice is increasingly challenging. Especially when so many conflicting findings and conclusions abound. Improving this state of affairs is one of the primary reasons we started the Cyentia Institute and why we’re excited to kick off this series. Let’s do that without further adieu.

Data Sources on Payment Rates

To identify sources of statistics for this meta-analysis, we leveraged the recently-released Cyentia Institute Research Library. We also tracked down references to sources not included in the Library and topped it off by searching for additional material on the web. Table 1 provides a list of these sources along with the requisite statistics and sample details.

Table 1. Data sources used for meta-analysis of ransomware payment rates.

In addition to those identified in Table 1, we found several sources and statistics that we omitted from our analysis for various reasons. These are shown in Table 2 along with a rationale for why they were not included in our analysis. We include them partially for thoroughness and partially to show the many challenges inherent to this kind of analysis. It reinforces the lesson that even seemingly straightforward statistics and references can be hiding dubious reliability upon closer inspection.

Table 2. Data sources omitted from our meta-analysis.

Meta-analysis of source material

Scanning the statistics in Table 1 leads to the impression that TrustLook’s 38% isn’t all that out there. Or, if it is, at least it’s in pretty good company. Figure 1 strengthens that impression by placing all payment rate stats conveniently on the same chart. We’ve also added horizontal bars enclosing a 95% confidence interval around each point, the width of which is based upon the sample size (TrustLook’s being quite wide for reasons Tony’s post makes clear). The blue vertical bar represents the combined payment rate based upon the body of evidence found in Table 1 for surveys (40.4%). The shaded region around that line corresponds to our confidence around that estimate (+/- 2.4%). That the region around the combined rate is much narrower than that of any single source demonstrates the advantage of incorporating multiple perspectives into an analysis such as this–especially when individual sources have higher degrees of uncertainty.

Far to the left in Figure 1, the Symantec and Dell Secureworks marks might be missed at first glance. We’ve colored them differently to distinguish these empirical sources from those based on surveys for two reasons. First, they are slightly older; Symantec was conducted in 2012, and Dell in 2013. Second, they have a different set of potential challenges from surveys (e.g., convenience sampling from sensors that can’t see the whole picture). The comparison between the empirical data and survey responses is striking. Empirical research suggests a much lower 1.65% (+/- 0.8%) of ransomware victims give into the criminals’ demands. We didn’t include the WannaCry outbreak because its early interruption via the “kill switch” makes it an atypical case for studying payment rates. It’s worth noting, however, that it does side with the other empirical sources rather than the surveys.

Figure 1. Meta-analysis of source statistics on ransomware payment rate.

We initially planned to combine all sources into a single rate/interval, but the disparity between empirical and survey findings precludes that approach. I suppose we could drop a line somewhere in the 15-20% range, but that would be kinda like predicting the election of a 3rd party candidate because the mainstream parties lie on opposite ends of the political spectrum. It seems less likely that the truth lies in the middle and more likely that one (or both) of the data collection methods are flawed in some way and thus failed to find truth at all.

The surveys shown here may (probably) suffer from any number of biases that result in respondents over-reporting the payment rate. For instance, asking “did you pay?” vs “would you pay?” seems to elicit very different answers (which is why we relegated the latter to Table 2). The empirical studies may suffer from convenience sampling and/or may not have visibility into all components of the payment chain, resulting in undercounting payments (i.e., perhaps missing some Bitcoin wallets to monitor). It’s also possible that large-scale events like CryptoLocker and WannaCry have below average rates because of the sheer number and scope of victims involved. We could go on, but you get the point — there are many potential pitfalls in research like this that can call the findings into question and make it hard to compare across methodologies.

Figure 2. Comparison of regional statistics for ransomware payment rates.

These issues are even more apparent when comparing regional statistics across the different sources in Table 3. We find it hard to resolve 61% min-to-max disparity among the sources for payment rates in the US are curious what circumstances cause that region to rank highest in the BitDefender sample but lowest in MalwareBytes. However, we can accept that 75% of Canadians end up paying the ransom–presumably because they don’t want to be unkind to the criminals by refusing, eh?

So where does that leave us on our quest for a ransomware payment rate and why does it matter anyway? Well, it makes a big difference when attempting things like assessing size of the ransomware problem relative to other cybersecurity threats. Take Figure 3, for instance, in which Cisco estimates the gross annual income for ransomware campaigns using the 2.9% payment rate from Symantec (which is based on one day’s worth of observations for one event 5 years ago, remember). If we swap in what the combined surveys say (~40%), that $34M shoots up near half a billion dollars. We’re not promoting that estimate, mind you–just sayin’.

Figure 3. Estimate of ransomware campaign revenue from Cisco 2016 Annual Security Report

As for where that leaves us, it’s hard to say. We regrettably don’t see “The One Answer To Rule Them All” amid the sources and statistics examined here. If the surveys are believed, payment rate is around 40%, but if you have more faith in the empirical , then payment rate is much lower, around 1.5%. What is clear is that we as an industry need to improve the way we conduct and communicate research findings. Is it reasonable to expect that empirical studies and surveys arrive at the same findings? I’m not sure, to be honest, but I’d ideally like to see them supporting the same general conclusions. Either way, we need to be very careful about the methods we choose and mindful of how those choices affect the outcomes. Since you can’t take away “A Number” to quote for ransomware payment rates from this post, we hope you can take away a good reminder/example of why numbers shouldn’t be accepted at their face value and how to vet them against multiple sources.

Next up in this series we will examine financial losses incurred by ransomware victims, including the seemingly ever-increasing ransom amounts.