Wildstar

Hello, my name is David Ray and I am the Lead Network & Database Engineer here at Carbine Studios. A mini-bio for me is that I’ve been in the game industry for the last 19 years, having worked for some major studios such as Dynamix, Interplay, Activision, and Blizzard prior to starting at Carbine nearly 5 years ago. Prior to entering the game industry, I worked in aerospace (on the space shuttle project) and the cellular industry (working on bringing internet to cell phones).

A post was pointed out to me over on Wildstar-Central today with user “Mizpah” asking about how MMO servers work. I thought it may be nice to give you all a general overview. Obviously I’ll need to be vague about certain things, due to proprietary design, but this should serve to give you a basic idea of how things work.

Before we begin, some quick definitions for the less techy amongst us:

Daemon: In multitasking computer operating systems, a daemon (pron.: /ˈdeɪmən/ or /ˈdiːmən/) is a computer program that runs as a background process, rather than being under the direct control of an interactive user. (I stole this definition from Wikipedia.)

Realm: A collection of WildStar servers that collectively represent a full planet of the game. Each realm effectively represents its own planet, even though all of the realms are effectively identical. Some realms will have special rules that apply only to them, like PVP. In other MMO’s this is sometimes called “Server” or “Shard” or a wide variety of other names that all means the same thing.

SERVER ARCHITECTURE

Our server architecture is currently made up of 11 daemons running across several server machines. For some of our daemons, there’s only one per region (i.e. country), where others there are dozens, or maybe even hundreds, per realm.

When you connect to play our game you first connect to our “Auth Daemon” (auth is short for authentication), this is the process that accepts your username and password and decides if you’re allowed to play. Authentication information is always kept secure, even in our databases. Once the daemon has decided that you’re allowed to play, it fetches your security permissions. Based on your permissions it will present you with a list of realms you are allowed to play on. Once you select a realm, you will be handed off to other sets of daemons, and the Auth Daemon is fundamentally “done” with you.

After that you connect to a “User Daemon”. This daemon is responsible for keeping track of you as a playing entity, it tracks all of your character data, including but not limited to; character stats, inventory items, quest progress, and pretty much anything else from within the game. There is one of these daemons per realm. This daemon also handles the character creation process. You stay connected with the user server for the entirety of the time you are playing.

Once you have selected or created a character, the “User Daemon” will then also connect you to a “World Daemon”. This daemon may have already been running, or in the case of instances it can be launched just for you and/or your group. This daemon is responsible for the bulk of what people would consider gameplay. It handles all the creatures and their AI, the loot that is dropped, any combat and spells cast by players or creatures, pretty much everything else that is “game” happens here. Whenever something changes about the players, the World Daemon reports it back to the User Daemon to record it in our databases.

That is pretty much the fundamentals of how the game works. All in all we have 11 daemons currently. I’m not going to go into detail about each of them, but they all have specialized tasks, one of the daemons only job is to figure out what machine each of the other daemons runs on based on CPU and memory load. And that’s really important, because of the way we have broken up our daemons, it should allow us to add more hardware to handle additional load whenever we need it. If we want more players on a realm, we can add more server machines to run more World Daemons.

SOME Q&A

And now let me answer some of the specific questions that were asked in the thread from Wildstar Central:

What might the split be between worker infrastructure and storage look like ? How many server components and services are involved ? chat, login, federation, logic etc.
This is too big of a question to answer fully, but some details.

Our storage is MS SQL Server. We have a “few” realms per SQL server, this is one of our scalable components.

Chat can be handled on one to three of the daemons. If it’s a direct message from one player to another, then it can just be handled on the user daemon. If the chat is a say or a yell, it needs to go through the world daemon to figure out who is in the area to “hear” it. And then if the chat message is for a user who’s not…. Well I’ll leave that feature for a later announcement.

Where are the most likely bottlenecks - db access?
The database is where we’re expecting our bottlenecks to be. But a lot of our daemons were written with scalability in mind, so if we find out bottlenecks are indeed the database we can rebalance that load and add more database servers with relative ease.

What db techs might be relevant ?
As mentioned above, we use MS SQL server.

I would love to see a topology of a 'real world' mmo server infrastructure and an article that covers a deep dive't this is what's going on'.
Well, you got what you asked for, at least the high level view above. If you want more details about what the other 9 daemons are, we do have an open network programmer position on our jobs page.

I am also interested in how much innovation has impacted the mmo space sever and datacentre side in the last 5 years. If it follows the trends in the enterprise data centre, then the scalability and, provisioning and burst variations should have helped a lot...
Oh, lots of things have changed since I worked on my first MMO. The biggest in my opinion is the fact that all of our servers are running 64-bit now. It was a pain staying under 2 GB back in the day. Also, CPUs and hard drives are unbelievably faster than back then. I’m an old timer, when I first got into the game industry we were targeting 16Mhz-33Mhz machines with 1-4 MB of RAM, and distributing on floppy disks. So yeah, things have changed.

Thanks so much to David for taking the time to talk about servers. Do you have an idea for a cool blog post you'd like to see here in the future? Let us know on Facebook, Twitter, or one of our many fansite communities!