MCV/DEVELOP News, events, research and jobs from the games industry
A recent study into the prevalence of loot boxes made headlines last week, as The Guardian picked up on research from The University of York, which indicated that loot boxes are a major factor in modern PC gaming – with the report stating that loot boxes appear in 71% of the “most popular” on Steam.
While the study’s findings were certainly attention-grabbing, particularly in the more traditional corners of the press, it certainly needed a deeper look to garner a real insight into the work. So we got in touch with author David Zendle, lecturer at the University of York, for comment.
A bit of background first, or alternatively you can read the paper for yourself here.
The games deemed to be the “most popular” on Steam were found via SteamDB’s ‘Most Played Games’ category – the service tracks the number of concurrent users on the service by title.
With Steam being less than transparent, although arguably still more so than many such platforms, getting reliable data for such a study is bound to be somewhat problematic – something the industry only has itself to blame for, if it wants better academic work it needs to become more open with its data. Still, we’d argue that this particular method was likely to overrepresent free-to-play titles, which have spikier user patterns, and higher concurrent peaks than most paid titles.
Additionally, the sample set comes is global. Which is frustrating, as when UK-based academics, are reported on by a UK newspaper, which is likely seen by UK public and policymakers – there’s a danger that they will presume the data is equally applicable to the UK gaming market – which actually trends more to retail titles and console games than the global average.
Of course, a lot of our problems are less with the study and more with the reporting around it – For instance, The Guardian openly refers to loot boxes as a ‘form of gambling,’ and discussing worries about exposing children to ‘problem gambling’.
The study itself however, questions this, stating: “Spending on loot boxes has been repeatedly linked to problem gambling. However, it is uncertain whether this is because loot boxes cause problem gambling, or whether it is because individuals with pre-existing gambling problems spend more money on loot boxes.”
Still, the study certainly raised questions for the industry, so we reached out to its author David Zendle, lecturer at the University of York, for comment.
Why did you decide on this topic?
I think a lot of video game research is done by people who don’t play games (or care about games), and therefore misses the mark when it comes to relevance. The topics that are investigated just don’t chime with the things that gamers themselves are concerned with. We can point to the extensive literature on the violent effects of the depiction of violence as a good example of this. The side-effect of this is that many topics that are of great relevance to gamers just haven’t been explored: loot boxes are one, but so too are the effects and drivers of in-game toxicity; and the effects of a pay-to-win ecosystem on game design.
I got interested in loot boxes because people were starting to make grand claims about their effects, but the groundwork just hadn’t been done in terms of research. Usually these discussions centred around formal features of loot boxes (i.e. the extent to which they ‘looked like’ gambling). I figured establishing the robustness and size of some basic effects would be helpful in moving the conversation forwards.
This specific piece of research was inspired by competing claims: Either that loot boxes were tailing off and diminishing, or that they were experiencing some kind of sudden spike. I wanted to get to the bottom of what was actually going on. And it looks like (at least on desktop), neither of these things are happening.
In brief, what do you think the research suggests about changes monetisation going forward?
This isn’t a predictive model: You can make those – people in econometrics often do – but this isn’t one. So it’s not intended to tell us anything about the future. What it does inform us about is the recent past, and the present. We appear to be in a situation where the majority of desktop gamers are playing games with loot boxes in. And there is no evidence that this prevalence has been diminishing. This lies in contrast to perspectives that loot boxes are a relatively niche feature of gaming. In this data set, at least, they were not.
A study made in the UK, and reported in a UK newspaper, will inevitably engage a UK audience (including policy makers), but to be clear there’s no localisation in the sample set here?
No: This is a global sample, and applies equally well to all countries (or at least all countries that play games on Steam)
It looks like after some steep rises (in loot boxes and cosmetics) up to mid-2013 that things have largely levelled out, so why the strong interest in the subject in 2019?
I think this is a failing of both regulators, policymakers, and academics. Formal structures for horizon-scanning clearly aren’t working: We’re investigating the effects of innovation about 5 years after the horse has bolted. This isn’t just problematic when it comes to loot boxes – it suggests that the way the system is working may be systematically slow at investigating all emerging video game effects. And this is a problem because games innovate so rapidly: It’s likely we’ll be slow off the bat at investigating the next potentially problematic feature of games, too. Gamers have been complaining about this issue for several years. I think people need to think long and hard about why we are only researching this topic now.
Your method for selecting games, over 10,000 concurrent users, could prevalence titles with ‘spikier’ user cases, such as free to play titles, rather than fully-paid products. Did you/could you break down the results based on whether games had an upfront cost?
No, we didn’t! But you absolutely could if you wanted to. The big task here was working out whether games contained loot boxes or not, and we’ve already done that. We’re big believers in open science, so we’ve made all the data we used available freely and openly alongside the paper. That includes all that information. I’ve also included the analysis scripts we used. It would be fairly straightforward to extract just the paid for / free to play games and run separate analyses over these. It would be interesting to see what you got: I would expect that you’d see a greater prevalence of loot boxes in the F2P titles, but that’s just a guess.
It seems frustrating to me that there’s not better, anonymised, information available to academics from the industry to make such studies, it looks like they have something to hide, when surely the industry should be opening up to scrutiny?
Yes! This is hugely frustrating to me as well. A lot of the time, I feel that I’m spending my entire life trying to reconstitute the contents of a fairly minor Excel spreadsheet that a middle-manager at a AAA probably has on their desktop. It’s infuriating when you think about how easily important data could be collected and shared. A key theme of the recent parliamentary select committee inquiry in the UK (on ‘Immersive and Addictive Technologies’) was academics telling the government that they essentially couldn’t do their jobs properly because of a lack of industry data. Sadly, it seems as if industry isn’t keen to share this data themselves: More and more, people in the UK are talking about the creation of an independent video games regulator that would compel them to share this data. This seems like the only viable option moving forward if we really want to find out what the effects of games are.
It’s this issue of transparency that brings us to the real crux of the issue. While we may have disagreements about the study’s methodology – if the industry won’t provide more information to academics, then the studies around loot boxes are inevitably going to be flawed. As the practice faces mounting political pressure in a number of countries, it’s easy to argue that it’s in the industry’s best interest to be more transparent.
We somewhat doubt UK policymakers are reading MCV/DEVELOP on a regular basis – but they will read the Guardian. It helps everybody both inside and outside the industry if they could be better informed.
