Reviews - Updated on April 9, 2022

Destiny 2 recently experienced a rebirth: the core of the game became shareware, and the PC version moved from Battle.net to Steam. The excitement was serious: on the Valve site, the number of simultaneously playing players instantly jumped over 200 thousand – Destiny 2 servers could not withstand such an influx.

This story seems to be repeated with every major online game: it is almost impossible to enter it at the hour of release or immediately after the launch of a significant update. But why do developers step on this rake with enviable persistence? Why don’t they, as regular players often write, just add servers during the influx? What is so difficult about this?

Kotaku did a bit of research to get to the bottom of the issue.

Rails, rails, sleepers, sleepers

The site spoke to Rami Ismail, an indie developer and huge Destiny fan, on the topic of overload. According to him, the launch problems may indeed be due to a lack of servers: the network infrastructure does not have enough computers to handle all the connection signals that players generate when they enter. “Imagine a railway station that does not have enough tracks for all the trains,” Izmail draws a comparison.

However, the flaw may also lurk in the client installed on the computer or gamer’s console. For example, due to a bug, the game sends an excessive amount of data to the server. “It’s like the station has enough tracks, but there are too many unscheduled trains arriving,” the developer continues the analogy.

As Ishmael says, in order to roughly imagine the features of network traffic management, you can play Mini Metro.

Ishmael’s third theory is a weak link in the server code, which slows down the processing speed of data from players’ devices. “There are enough trains and tracks at the station, but each passenger who enters the platform needs to fill out a special document or something like that,” Izmail returns to the railway theme.

The fourth version is the code or server of a third-party company that participates in the process, but the game developers have no direct control over it: “The foreign station on the border passes too few trains, so people at our station have to wait a long time.”

Any of these obstacles can cause servers to crash, says Ishmael. “This is a common problem, so they start to distribute the excessive load to neighboring cores or servers. Here our station redirects trains or passengers to other stations that are already fully occupied. Now they are also overloaded, and the complexity has hooked all the stations. Some of them are closing, some of them cannot receive even more trains.”

Preparing for any of these situations is difficult, Ishmael assures. Yes, it is possible to conduct preliminary tests, but nothing will transfer the load from a real influx of players. “Usually, the whole system can be evaluated only when the trains start to run for real. Suppose a train from station A arrives at station D two percent slower than expected. At station D, a slow accumulation begins – until an error occurs.

Network tsunami - why servers constantly break under the influx of players

Destiny 2 shareware release simulator.

Butterfly Effect

All the developers Kotaku spoke to were skeptical about the suggestion that online games don’t start well because of a lack of servers. Now popular providers like Amazon, Google and Microsoft make it pretty easy to add more machines. There are always enough of them, the interlocutors of the site say.

Large companies have money to increase the number of servers if necessary. “When a huge game has a bad launch, it’s usually not because of a lack of networked machines,” says developer Josh Ling. “There are cars, but software stuffing causes problems: crashes, system failures, and so on.”

Drew Thaler is a developer who spent four years at Naughty Dog launching and maintaining multiplayer on Uncharted and The Last of Us. He now leads the Mesh company and provides network services. As Thaler recalls, Naughty Dog ran into difficulties in the multiplayer even when porting The Last of Us from PlayStation 3 to PlayStation 4. The versions of the game were almost the same, but some of the Sony technology that the team used was not previously used on such a large scale, therefore led to complications.

Network tsunami - why servers constantly break under the influx of players

Did you miss me?!

“For users, it looked like there was a long matchmaking going on,” Thaler shares. “The game code would form a match, and then some people could not directly connect to each other, which caused the match to be canceled.” Naughty Dog made a crutch with which to bypass the error, but the studio did not discover the source of the software infection until a few months later. It was covered in a small piece of code – “the SSL stack in the Java implementation” (we hope at least someone understands what it is about!).

It seemed that this piece could not affect the matchmaking so much, but that’s exactly what happened. “It’s like Google Maps on your phone is showing wrong directions because of bad wiring in your house,” Thaler compares.

After that, Naughty Dog undertook to launch the multiplayer Uncharted 4: A Thief’s End. By the standards of the studio, it was ambitious: more players in the match, teammates controlled by AI, virtual currency, and so on. All this goodness needed a new code. “This is where a lot of problems arise,” says Thaler. – In tests, the system works great, even in the simulation of several thousand people at the same time. But in the first hour, when a hundred thousand are trying to enter the game at once … “

“Network services are not just a server. This is a complex system. This server is responsible for matchmaking, this one is for queues, this one receives event records, this one does NAT tunneling, this one knows the location of players for improved connectivity, this one runs the leaderboards, this one handles authentication, and this one handles virtual currency. It needs to be especially resilient so as not to lose transactions and credits,” explains Thaler. Any link in the chain can tremble.

In general, you can’t just take and make more servers – for this you need to pump the whole structure cool.

Deadly Christmas Tree

The Last of Us for PlayStation 4 is more or less an easy project for developers, because compared to the PlayStation 3 version, there were relatively few changes in the code. But even in this project there were difficulties. Now imagine what will happen if developers constantly introduce something new or experiment a lot.

Even a tiny cosmetic feature can turn into a disaster. Once upon a time, game designer Raph Koster recalled a story from the life of Ultima Online. Once the team wanted to give all the players a festive tree in honor of the upcoming Christmas. The developers did not have ready-made assets for the Christmas tree, so Koster decided to cheat. He took an ordinary spruce and wrote a script for it so that small gems would appear on it. All gems had another callback script that made the gem appear and disappear. It looked like decorations were flashing on the tree.

For Christmas, a Christmas tree was given to everyone who logged into the game. So Ultima Online planted thousands of new trees. Each tree has twenty callbacks spaced approximately one second apart. The write queue was overloaded on all servers, after which the service completely fell off.

Ultima Online collapsed on Christmas Day, a weekend period when players want to play, and developers relax with their families and are not ready for such large-scale jobs. And all because of a modest gift!

Network tsunami - why servers constantly break under the influx of players

It seems that this is what the same Christmas tree looked like.

Another example of surprises in the network code happened in The Division 2. The game went through beta testing, but a few days after the release, users faced a nuisance: their turrets, drones, and other similar skills were destroyed immediately after activation.

The developers found that the servers were junk due to the accumulation of status effects associated with in-game skills. Apparently, this software hole was in The Division 2 from the very beginning, however, it gradually accumulated and burst only when many players passed through the shooter. The mistake was patched up quite quickly, but it was not possible to detect it in advance.

Many accumulated problems make themselves felt only under heavy load. Perhaps that’s why Destiny 2 crashed when the shareware version launched. Huge crowds of gamers tried to get into the game – they were probably much larger than what the developers had previously encountered.

Thaler says: “When AAA goes free, the user base grows exponentially, which may come as a surprise to those who don’t work with shareware games. We had a free-to-play weekend at Uncharted during the beta, and our servers were hit with 10 to 20 times the normal burden.”

Destiny 2 is moving to free-to-play video in color.

***

Kotaku’s interlocutors do not believe that network sores will ever be completely cured, but you can always minimize their number in the future. The more network elements in games, the more experience developers have, thanks to which they build infrastructure more reliably and fix bugs faster.

Third-party teams that developers use are also improving. For example, the British company Multiplay helped Electronic Arts and Respawn launch Apex Legends. Recall that the game is shareware and started simultaneously with the announcement, which means that the load on the servers was very high. Nevertheless, Apex Legends managed to cope with the wave of applicants.

According to Kotaku, developers are helped out even by beta tests before the release itself, which seem to many to be an ordinary publicity stunt.

However, there is no universal cure – developers still have to plow tirelessly. “You have to work under gigantic time pressure with deadlines before yesterday. Because everyone wants to play their favorite game, but they can’t do it until you solve the problem,” concludes Thaler.

Similar content:

Overland: Review

Reviews • 09/04/2022

We’ve seen quite a few turn-based tactical games more or less like chess. One of the most striking examples is Into the Breach, where ...

Everreach: Project Eden Review

Reviews • 08/04/2022

Since its announcement, Everreach: Project Eden has felt like a budget take on Mass Effect, a futuristic action role-playing game also set on distant ...

Reverie Knights Tactics Review & Gameplay

Reviews • 26/03/2022

Get ready, now there will be a lot of words “tactics” and its derivatives in a row, which is a nightmare for Andrey Makoveev ...

Felix the Reaper: Review

Reviews • 09/04/2022

Relevant for platform: NSW Main Content Step by step Thankless job For the sake of love, people are ready for a lot. So Felix ...