Tennis for Two Thousand: the story behind Improbable’s mass event in Scavlab

Ever since the very earliest titles, such as 1958’s Tennis for Two, player count has been something of an obsession for the industry. Fortnite, for instance, has recently generated some very big numbers: 15m players were online for its last big event, as players tackled Galactus for the climax of its Marvel collaboration. Previous to that it boasted of 12.3m concurrents for its Travis Scott event.

Socially, though, the experience is more akin to a small club gig than a super-stadium extravaganza, as attendees are segmented into innumerable 100+ player servers.

Improbable though has recently shown off something much greater, a singular real-time environment featuring almost 10,000 live and simulated players. Now that’s a crowd worthy of the name, a crowd that could come together in a virtual space to defeat something truly epic, or collectively (and audibly) lose its shit when an act drops its biggest hit.

Enabling such ambitious virtual experiences was always the Improbable mission. But the company, after some years, is showing that the dream is becoming a reality. In part that’s because its core technology is maturing, but equally because it’s built out a broad range of studios and services to support the future of multiplayer titles more broadly.

TESTING AT SCALE

Improbable’s stadium-sized experiment featured near to 2,000 live players, all interacting in a single, relatively compact, space. If 2,000 people doesn’t feel like a big crowd to you (erhh… where have you been for the last 15 months?) then an extra 7,500-odd virtual players were then added as a stress test (with each such player connecting from its own client in the cloud to provide authenticity).

Entitled Scavlab, the demo is a spin-off of Midwinter Entertainment’s well-received early access title Scavengers. The Improbable-owned studio is utilising its parent’s tech at a more modest level to create a PvPvE battle royale-survival hybrid, with 60 players pitched against each other as well as thousands of AI controlled entities.

Scavlab shares a lot with its parent title, but introducing thousands of players is obviously a very different experience from a tightly controlled, competitive battle royale. So what is the company doing with those players? Well, it’s playing tennis with them for starters.

“The Scavlab event you saw was a hodgepodge of different things,” says Improbable CEO Herman Narula to us via Zoom. Although the last time we ‘saw’ him, he was a gigantic glowing avatar, a god-like form, towering over the Scavlab attendees.

“It was an experiment, we had Chris, the producer, and me, playing tennis with the players,” quite literally, by batting them en masse back and forth between the two of them. “That wasn’t why we built the environment, we were just like, is this fun? Maybe we try this?”

It certainly raises a smile, but more importantly it’s all part of the learning process.

“That’s why we call it Scavlab, because we’re making it into this experimental space, where we’re just gonna do lots of different things. We’re gonna run them regularly, bringing in players and then trying different things.

“In that event we had a battle, a racing experience, some fun social, hanging out in the beginning. We don’t know which of those things it’s going to be. But the more we experiment with it, the more likely we are to find it.

“I think getting a bit more playful, a bit more experimental with it, will result in us figuring out what [this additional scale] means… But right now, I’d say it’s a very vague space, and it needs more definition.”

That’s certainly true from a design perspective, mass events are unlikely to ever be simply the games we know scaled up, players can quickly lose a sense of agency in such a mob. That said, it’s easy to see the possibilities that such big crowds, in highly-responsive virtual worlds, could bring. To that end, we cautiously suggest that such technology is a key part of creating the much-discussed metaverse.

Narula likens the term to the dot.com boom. “People sort of had the right idea, that the web would be big and the internet mattered. But that was a sort of a vague notion. And the actual things people could appreciate as new and amazing on the web turned out to take a lot longer to figure out.

“So I think while people say ‘the metaverse’, we identify more with the phrase ‘virtual worlds’, because it’s a lot more specific – it’s a big interactive environment with lots of players that are doing things, not necessarily a game, but in many forms. We need the technology, we need more experimentation, and we need more examples before people can really define that space.”

250 MILLION OPERATIONS

And defining the complexity of such events is something Narula feels is a useful step: “We’re thinking about the kind of core metrics for such virtual worlds. Operations per second, for instance, to us that’s an important metric for how connected and how ‘live’ the environment is.

“How much stuff is happening at any moment? How much communication is taking place inside the environment? Typically, you’ll see games, complicated games, doing a few thousand, maybe tens of thousands of operations per second – in terms of networking and communication – that demo handles 250 million operations per second.”

It’s an incredible upscale in terms of data, too much to send to any single client. “We had to invent a solution that compresses the bandwidth for every individual player, down to less than a zoom call.”

Plus there are more typical issues with having massed groups. “Even the rendering, with all those people on screen. So we have to come up with new rendering approaches that allow us to distance render moving objects more effectively.”

And then people need to communicate. “Voice chat – so in the event, the performers were talking to everyone, but we couldn’t find any voice solution that could support a person talking to 9,000 people in real time. So we had to develop our own tools there as well. It just becomes a kind of shopping list of different challenges that are happening. But it was a very proud moment.”

SCAV ARMY

While Scavlab is a very different experience from Scavengers, it certainly wouldn’t exist without its parent; if only because there wouldn’t be a community of players to come and engage with the experience. And the core game looks to be gaining traction too.

“It’s still only in early access, but we had half a million people in the first week, which is a good starting point. I think it’s found an audience of players, but it’ll be a long way before it’s fully polished and where it needs to be.

“But having that audience of players and being able to run experiments with them, I think really helps. And means we can find a lot more.”

We wonder about the financial costs of running such a genuinely massively multiplayer experience, will they be prohibitive? “It’s a free to play game, let’s put it that way. The server costs are not really a factor,” responds Narula.

Although at present, all those additional virtual players are racking up a bill.

“The most expensive part of the experience is that we have to boot up relatively heavyweight fake players. So in order to test with 1000s of people, how do we get 1000s of people every day, so we built a system that boots up machines in the cloud, with full clients and logs them in as though they were players, and so actually running those 1000s of machines…” He doesn’t put a figure on it, but you get the idea.

THE LONG RUN

It’s good to see Improbable showing something truly impressive with a sea of real players. We first visited the company just over four years ago and at that point it looked as though its Spatial OS technology was ready to take the world by storm. Obviously, that still meant many years before we expected to see its full impact. But even then, it does seem to have been a longer road than we hoped. So does Narula feel the same?

“The process of solving problems like this… it’s generations of technology. Not a single line of code from when I originally was involved is even part of this,” he begins. “It took a long time to go through successive generations of Spatial OS and of the core technology, to get to a point where we can do the things that we can now do.

“And some of the problems we ended up solving on the way weren’t problems we thought we’d be able to solve. So a really good example is density. So what’s really remarkable about what we showed you, is that unlike in previous incarnations of our technology, we can’t only support scale, but we can actually support everyone in the same spot, all together.

“And the problem was one that if you’d asked me when we met four, five years ago, I would have said, ‘look, I don’t know how we would go about doing that’. Now to put it in a real game, Scavengers, to actually have it be a viable part. None of that would have happened back then. So I think about the fact that I’m glad we’re here.”

And Narula feels that there’s more improvement ahead than behind. “There’s still a long way to go. There’s so many more things we’d like to create, and so much further we’d like to take it. I don’t think it’ll ever be done. I think it’s an endless journey. And each successive generation of games makes it better.

“You see this with companies like Unity and Epic… where they’ve built incredible technology over a long time, over successive generations of games. And so I think you’ve got to be in it for the long term, if you want to build something that matters in this industry.”

SERVICING THE METAVERSE

And building is certainly something that Improbable has done. While the core mission remains, the company has quietly expanded to a surprisingly large 950 people, which now encompasses in-house development studios and a broad range of services for those making multiplayer titles.

“When we started the company, we were just focused on solving one problem, which was bringing scale [to MMOs]… but as we evolved, we realised that we wanted to bring higher scale, more interesting networking, easier to develop workflows, to every type of multiplayer game – why just restrict ourselves to one type?

“So in doing that, and just listening to and working with more and more customers, we realised that people have a lot of problems that have nothing to do with networking. Just running and hosting your game, that’s pretty hard. People want and need a lot of support with a lot of other multiplayer problems.”

And that’s why the company is now launching what it calls Improbable Multiplayer Services (IMS). “It’s our way of going ‘Okay, why don’t we just solve the key multiplayer problems that many people in the industry are facing’,” Narula explains.

“Where people want to do really impressive things with really high scale, well great, we’ve got the technology to support that. But when what they want is just a more efficient way of running their multiplayer game, a better operating platform and the ability to leverage bare metal in the cloud, all kinds of common industry problems, we have those solved as well.

“And what we’ve found is that we get into relationships with developers who come to us to solve one problem. And when we do that well, they want to work with us on more problems. And so there’s a really nice kind of flow of relationships and deals.”

And that greater integration into the broader industry can then be a trojan for its grander plans. “Before we could only really work with a very small number of people in the industry. Who were building very specific kinds of games, and we had to work with them in a very narrow way. It just didn’t work out very well, it’s better to be a partner that can do all of these things for someone.

“And with our new deck, we’re taking a very modular approach, rather than having to adopt everything. As you see with Scavengers, which in a sense is using one part of our technology for the game session, and then a different part for Scavlab. Theoretically, you can drop that part into another game, which maybe wasn’t even running on Spatial OS for the main game. That flexibility, that incremental adoption, we think is really where we need to be, in order to serve customers better.”

IMS isn’t totally new, it’s more an umbrella brand for a wide range of services that Improbable has been (somewhat quietly) providing for some time. Two acquisitions form key parts of IMS, explains Narula.

“One was Zeuz, which became part of our hosting and operations solution, which is kind of core to everything else that we provide, and it allows people to buy that stuff by itself, which is great, because again that means we can work with more and more developers.

“And on the co-dev side, we acquired the Multiplayer Guys. And we’ve built that out, we’ve been able to work with studios that have problems, even when those problems don’t necessarily involve our deck. And that means we can start relationships and solve problems with people. It’s growing really, really fast. Actually, it’s one of the fastest growing parts of our business.”

Co-dev is often a secretive business, but Improbable is able to speak on some projects the team, now numbering 150, have worked with. 2K, Zenimax Online Studios and Arkane Studios are mentioned, plus it contributed to Fall Guys: Ultimate Knockout. Primarily helping with multiplayer networking, but also other elements, such as technical art.

“It’s very specialised. So we work on helping people with multiplayer networking, back end and other kinds of problems. The very same problems that Improbable Multiplayer Services helps solve. So there’s a really nice synergy there,” says Narula. Although honestly we’re just thinking about a 10,000 player game of Fall Guys at that point.

Narula continues: “We think by positioning ourselves that way and supporting the industry as it shifts towards more ambitious games, that’s the best way to fulfil our original vision.

“It’s sort of the evil master plan in a way,” he smiles. “We want to be able to solve multiple problems that people have at any level of the stack, if they want a place to host, if they need support and engineers, if they want to be more ambitious. We want to be the company that helps them with all of those things.”

And helping it is. “On the hosting side, I think there are 10m monthly active users across different games that we host or support in some way,” he declares.

STUDIO LINE

And of course, providing services for other developers is only half of Improbable’s long-term, hands-on gameplan, with Midwinter Entertainment and Improbable’s Edmonton studio, both playing key roles.

“I think the lesson we learned over the last nearly eight years of doing this, is that to build really innovative technology, you can’t sometimes separate the work of building a great game from building great technology.

“You need to do things hand in hand, you need really amazing developers, with years of industry experience, that are helping you understand what really works and what doesn’t work, and are able to take some of the risk in supporting developing things. 

“Take Scavlab, for instance, if we showed up to a studio and said, ‘Hey, we want to cram 9,000 people into an event space inside your game’, they would have just called us crazy!

“To have an internal team is necessary for some of the innovation that we want to do. And then if you think about the actual work to develop something like Scavlab, it’s not all back-end networking technology. So we look at the studios as being about first and foremost building great games, but the only way we can build great technology for our partners is also by building great games. We just can’t not do those things.”

It’s certainly an argument that Epic would agree with.

“I think that’s a great way of putting it,” responds Narula. “And that’s exactly what we’re inspired by and what we want to harness, it’s having that combination of things, I think with Midwinter, it was really a meeting of minds, our teams just got along incredibly well. We were working really closely together. And we felt if we were doing it together, we could be even more ambitious. And I think that has borne out in what we’ve managed to produce with Scavengers.

“Even the technology grew in leaps and bounds. A lot of the criticisms that, rightly, many years ago could have been levelled at our very early technology: it’s harder to use, it doesn’t necessarily apply to different types of games. By working carefully with these great studios, and by developing things like what you’ve seen, we’ve been able to overcome these challenges. It really helps to have developers in house as we think about future ambitions, as we can plot those around game design ideas as well.”

VIRTUAL MOMENTUM

Off the back of its services growth and the successful launch of both Scavengers and the Scavlab demo, it does feel like Improbable has more public momentum again. So, will Narula now draw a line in the sand and define how the company will measure its success?

“I think the best judgement for us, and what we’re aspiring to all the time, is to have either our partners or one of our studios build a really amazing set of titles. We look at how many people those games are reaching, how much they’re demonstrating and showing the power and vision that we have for the future and what it can be, and what sorts of experiences we can provide people.

“I think, if people are able to go into Scavlab or experiences like it and have a wow moment, that’s what we’re aiming for. We really want to prove out and demonstrate and show the value of what virtual worlds can be.

“In addition, we think about all the studios we’re able to help. And a big metric for us is how many problems, what problems, are we solving for studios? How much are they consuming our technology? And how much can we help them? Those are key focuses for us.”

Narula won’t be drawn on a timescale for all this. “We’re very cautious, I think we’re in an industry where you’ve got to show not tell as much as possible. Our strategy is keep our heads down, deliver for customers, get out there and show more and more of these wins.”

Many years have passed since Improbable first set out its dream for scale. To achieve that it’s broadened its wings but the key mission still remains. And maybe just as importantly, the whole world over those years, has undoubtedly started to value Improbable ambitious aims more and more.

“We’re in an industry that’s shifting towards more multiplayer, more server hosted games, more complicated experiences. And we’re trying to be part of that industry transformation. And I think it will still take time before that transformation really matures. So we’re in it very much for the long term.”