The Blockstackers secret sauce(tm)

So I run this minecraft host and given my peculiarities in the field of what software I run, I just went and built a whole infrastructure and control panel from scratch, just so it does what I want.

Over the months people have asked me about it, and have on more than one occasion asked if it's for sale - and unfortunately I have to turn them down considering it's pretty specific to how I like to run things.

But I figured people deserve a little look behind the scenes.


The Secret Sauce from up high

The basic premise is that every Minecraft server runs inside it's own OpenVZ virtual container. Of course, managing that is easy when you have one node, but when you have 2, 3, or 50, things get a bit more hectic. That's why the first element of the secret sauce is provisioning.

Each hardware node runs a custom daemon that listens for commands sent to it, these can range from "read a file on disk" to "put this jar file in that servers' location". There is one master node, and every other node is considered a slave; the master node is the one that receives commands, and distributes them to the slave nodes who then report back.

So for example, when someone orders a new Minecraft server, once payment has been received, a command is sent to the master node to please go and provision a new container, with the given amount of memory. The master node then looks in a database (we'll get to that later), and finds out which nodes exist, how many servers are on there, how much memory is still free, how much the average CPU load is, and whether there's any free IP addresses left on the node. It then selects the least-loaded node and asks it to create the container.

Container creation is based on a custom-rolled template that's based on Ubuntu, but pretty much has everything required installed already. After container creation is complete, the resulting container ID is linked to the customers' ordered package, and the fabled "robo-steve is done setting you up" email is sent out.

Inside a container

On the inside of a container runs a tiny bit of secret sauce, I guess it's a secret nugget of sorts. It is the startup script that starts the Minecraft server. It actually attaches itself to the console, and forwards all console output to a collector node. It also listens for the collector node to send it things, which it will then push onto the console.

That small part allows us to remote control a server. And gets us the server log. And the chat. And some statistics on memory usage and CPU usage.

Other than that, no software runs inside a container.

Inside a node

On the nodes on the other hand, a lot goes on. The hardware node itself runs a few pieces of the secret sauce; first obviously the node controller, then there's the Halcyon daemon (we'll get to that) and a monitor service.

The node  controller can access all files inside a container by virtue of the containers' filesystems being stored on the hardware node itself (obviously), and as such we can write new configurations, set new jar files, manipulate the whitelist, banlist, op list, and anything else - from the node controller.

The control panel that clients use basically uses the node controller to do things ranging from saving a configuration to removing someone from the whitelist.

Halcyon daemon?

Yes. We're currently beta-testing a modification to the standard vanilla server. It doesn't add any commands, or blocks, or gameplay behaviour, but what it does do is allow the minecraft server to emit events to a message queue system. The minecraft server also listens for certain things. What this lets us do is keep track of block placement, destruction. player positions, explosion damage, fire spread, lava spread, water spread, and pretty much everything including the kitchen sink.

We can also tell the server to place blocks at certain coordinates, send messages to online players, and a few other things. You can probably see how this fits together.

The daemon listens on the message queue, and processes the events. It's also able to intercept commands typed in-game, and if they aren't a built-in command, it will attempt to handle it on it's own.

So what you end up with is a vanilla server, that has 100% vanilla gameplay, but comes equipped with anti-grief features, and some "quality of life" commands. Enabling and disabling parts of these is done with a built-in permission system.

The reasoning behind this? The Halcyon server jar comes in at 2.5Mb versus the 2.0Mb for the vanilla jar, and 10Mb or more for bukkit. It also doesn't add any plugins to the server itself, but allows us to run plugins outside of the actual container.

It is, in fact, possible to have a single physical hardware node running the Halcyon daemon for upwards of 100 minecraft servers.

Database

Of course, we do generate a hell of a lot of data. In order to ensure that we can make the most of it as fast as possible, we run MongoDB, a NoSQL database, to store everything. To make sure it keeps running, it runs in a 3-node replica set. This means that we can lose 2 nodes, and still be able to operate.

It also allows us to process upwards of 100.000 block log records in under a second - and in the long run also allows us to generate cool statistics, such as the most placed block, and ore distribution.

What else?

We're also beta-testing an API for our clients, so they can integrate a lot of things into their own websites, such as online players, their location, and health, allows them to send server commands, or receive notifications when certain log entries are found.

On top of that, we're currently working on setting up a distributed OverViewer map rendering system to allow for every client to have their own world map - with some changes that tie into the Halcyon data we collect, to allow for things like live position markers, and chat!

The secret sauce... will remain secret.

Due to the way things are built, and the fact most of the software is written in Perl, it's not very easy to pack up and install somewhere else. It requires you to know the ins and outs of OpenVZ, server configuration, MongoDB, Perl, and a good helping of "huh, why is this not working" solving skills. It would take too long to make it a neatly installable package.

However, if you still think "wow, I want some of that", feel free to email me, and perhaps you can have the secret sauce after all. I will warn you right now that there will be a price tag attached to it.