Multi-Player Game Server Framework

In the last section, you learned how to build a basic client/server app using the NIO libraries. Now we're going to take the Chatterbox example to the next level and turn it into a full-featured multi-player game server framework.

Design Goals and Tactics

The design goals for the framework are as follows:

• Simplicity. The core classes should be as streamlined as possible.
• Versatility/extensibility. The game needs to support a variety of game styles, ideally within a single instance of the server.
• Scalability. The server needs to handle a large number of concurrent connections.
• Performance. The design must strive to provide the highest message throughput possible. Slow operations on events (such as those that require database or file access) must not block the flow of other events.

To achieve these goals, we're going to make several key refinements to the original example:

• Encapsulate the communication messages into GameEvents.
• Cleanly separate logic from mechanism.
• Separate the server process into various stages of a pipeline.
• Back each stage with a thread pool for concurrent operations.
• Separate each stage with a queue to buffer events.
• Provide a game logic plug-in architecture for supporting multiple games within the same server app.

Before we can tackle these lofty goals, we'll need to build some additional tools. , "Interactivity and User Interfaces," discussed threading in Java and introduced the ThreadPool class. In this section, we'll build a variation of that thread pool that uses a BlockingQueue of GameEvents. The combination of a ThreadPool with a BlockingQueue is a design pattern called a Wrap. In the paper "A Design Framework for Highly Concurrent Systems" (Computer Science Division, University of California, Berkeley, 2002), Matt Welsh and co-authors explain it like this:

The Wrap pattern involves "wrapping" a set of threads with a queue interface…. Each Thread processes a single task through some number of stages and may block one or more times. Applying Wrap places a single input queue in front of the set of threads, effectively creating a task handler out of them. This operation makes the processing within the task handler robust to load, as the number of threads inside of the task handler can now be fixed at a value that prevents thread overhead from degrading performance, and additional tasks that cannot be serviced by these threads will accumulate in the queue.

A BlockingQueue is simply a Queue data structure that has a blocking deQueue() method. In other words, when deQueue() is called, the running thread blocks until an element is available in the queue, possibly indefinitely. BlockingQueues are covered widely in tutorials on threading, so we just refer you to the source code for our implementation (com.hypefiend.javagamebook.common.EventQueue) for more details. Wraps are very beneficial for our server app because we can have a number of threads waiting to perform a step in the pipeline, buffered from other stages of the pipeline by the incoming event queue. An important detail here is that the number of threads in each Wrap or stage of the pipeline becomes a tunable parameter of the system, enabling just the right number of threads to be allocated for a particular task to achieve maximum event throughput.

The Design

Our server design is modeled after the flow of data in the system, as shown in Screenshot.

Screenshot Client/server communication flow.

The sequence starts with the client connecting to the server and the server accepting that connection. Then the client sends events to the server. The server reads, parses, and processes the events. This processing often results in outgoing events being generated and sent to one or more clients. The clients then receive the events and process them, and the cycle repeats. As you can see, most of the interaction is driven by the client, but the server might generate some events independent of client actions. For instance, this could be in response to timers on the server for signaling game end times or a system-wide administrative message. Some design choices must be made that will help shape the app. The first is to decide on an event format because events are the app's primary data structure. After that, you'll need to determine exactly what classes and interfaces are required to implement the previously discussed sequence.

GameEvents

Communication between client and server, as well as between each stage of the pipeline, is done via events. The choice of event formats is important to the individual game designs but is transparent to the framework because we provide an interface to the events and for passing events between segments. This way, each game can have its own distinct event format. There are many possible formats that can be used to transmit data in the system. The choice is highly dependent on the characteristics of the games to be hosted on the framework. At the most basic level, simple ASCII text messages can be used as in the ChatterBox app. However, an app with any real level of functionality requires a more complex structure. We discuss a couple of options here, but there are certainly other possibilities, and you should design an event format that is appropriate for your games. The design criteria for selecting an event format are listed here:

• Size/compactness. Because you are sending these events over the network, bandwidth can become an issue as you reach a high volume of messages. An arcade-style game requires a much larger quantity of messages than a card game.
• Human readability. Keeping the format humanly readable has its advantages in debugging but is at odds with the first criterion: size. A compromise approach is to use an abbreviated human-readable format to shorten the messages but maintain readability.
• Serialization/parsing ease and speed. Events must be converted from their object format into the over-the-wire format and back again. This process should both execute quickly and be easy to implement.
• Size of serialization/parsing code. Depending on the deployment scenario, the size of the client code can be a significant issue (for example, with a mobile device), so requiring a large code library to parse your events could be prohibitive.
• Flexibility. It needs to handle a variety of event types.

Let's take a look at a few of the common options available for event formats and see how they meet these criteria. Whichever format you choose, it is important to gather empirical data on the size and performance of a given method, to make sure it will meet your requirements for bandwidth, flexibility, and processor overhead-preferably before you start writing the main code for the project!

Serialized Java Objects

This method involves using Java's built-in serialization capabilities to convert a GameEvent object into a byte stream. The advantages are that it is easy to implement, size efficiency and serialization speed are reasonable, and very little additional code is required to perform the conversion because serialization has been built into Java's core libraries since JDK 1.1. This format is not humanly readable, however.

XML

XML is an increasingly popular format for all sorts of network communication and data storage needs. For GameEvents, it has the advantages of being easy to implement, very readable, and flexible for a variety of message types. The drawbacks can far outweigh these benefits, however. Serialized size is huge (unless using highly abbreviated element and attribute names), and parsing overhead could be too large when dealing with high event volumes. Also, the additional parsing code required on the client could be an issue. However, the 5k NanoXML parser (available at http://nanoxml.n3.net) provides a good option for size-constrained clients.

Custom Binary

This format basically involves writing out the fields of the GameEvent directly to a sequence of bytes, using as few bytes as possible for each primitive data type. Certain techniques can be used to squeeze primitives down beyond their native size requirements-for example, using Run-Length Encoding (RLE) to compress Strings, bit-packing multiple Boolean values into a single byte, and quantizing and compressing integers. This option can provide the best size, is fairly easy to write, and is quick to parse. Like serialized objects, it is completely unreadable. But if you need to squeeze the most out of your bandwidth, this is the only way to go.

Binary-Encoded XML

Programmers have realized that although the verbosity of XML is a good thing, it tends to hog bandwidth and storage. This is extremely important for wireless devices. The WAP Binary XML Format (www.w3.org/TR/wbxml/) is a World Wide Web Consortium recommendation that attempts to solve this problem. That or similar encodings might provide a good compromise-not readable while in transit, but normal XML when decoded. The format provides efficient size for transport, but encoding/decoding can add substantial overhead.

Terminology

Before we get into the implementation, let's define a few terms that we'll be using in the framework:

• GameName. A class of game, such as Chess, TicTacToe, or SpaceWar.
• Game. A particular instance, or play, of a GameName.
• GameID. A unique identifier for a game.
• Player. An identifiable user connected to the server that might or might not be actively engaged in a game.
• PlayerID. Unique identifier for a player.
• Session. The time between a player login and logout. This can span more than one connection/disconnection cycle.
• SessionID. Unique identifier for a session.
• EventType. One of a set of event types that determines the purpose of the event, such as LOGIN, CHAT_MSG, or MOVE.

Class Diagrams

Class diagrams for both the server side and the client side of the Game Server Framework are shown in Figures 6.6 and 6.7.

Screenshot Server framework class diagram.

Screenshot Client framework class diagram.

Common Classes and Interfaces

The framework contains a number of interfaces and classes that are shared between the client and the server. They are all contained in the com.hypefiend.javagamebook.common package:

• Attachment Class that is used as our SelectionKey attachment. Holds the read buffer and handles checks for complete message payloads.
• EventHandler Interface that defines only one method: handleGameEvent().
• EventQueue The Blocking Queue, mentioned earlier.
• GameConfig Class to hold configuration data for GameController classes.
• GameEvent Interface for GameEvents that allows the framework to treat each game's events generically.
• GameEventDefault A basic implementation of GameEvent, used in the sample game.
• Globals A class that defines common constants for the client and server.
• NIOUtils Contains a number of utility functions for dealing with channels and buffers.
• Player Interface for players that allows the framework to treat each game's players generically.
• PlayerDefault A basic implementation of Player, used in the sample game.

Some of these classes and interfaces require elaboration. Let's cover them in detail before we get on to the server-specific classes.

GameEvent Interface

For our framework, we define an interface for the events and provide a default implementation. It is expected, however, that games will either extend or replace the default implementation as needed. Listing 6.6 shows the complete GameEvent interface.

Listing 6.6 GameEvent Interface

public interface GameEvent {
 public int getType();
 public void setType(int type);
 public String getGameName();
 public void setGameName(String gameName);
 public String getMessage();
 public void setMessage(String message);
 public String getPlayerId();
 public void setPlayerId(String id);
 public String getSessionId();
 public void setSessionId(String id);
 public String[] getRecipients();
 public void setRecipients(String[] recipients);
 public void read(ByteBuffer buff);
 public int write(ByteBuffer buff);
}

In addition to the GameEvent interface itself, an interface that will allow for the passing of events between stages of the server's event-handling pipeline is required. This EventListener interface consists of a single method only:

public void handleEvent(GameEvent event);

All server and client classes that accept incoming events must implement this interface.

GameEventDefault

The default GameEvent implementation, GameEventDefault, uses the ByteBuffer put and get methods directly to read and write event data as raw byte sequences. For example, to write the eventType member, we use this line:

buff.putInt(eventType);

To write String members, such as a message, we first write the number of bytes representing the String length and then the bytes of the String itself:

buff.putShort((short)str.length());
buff.put(str.getBytes());

Listing 6.7 shows the entire read() and write() methods of GameEventDefault.

Listing 6.7 GameEventDefault read and write Methods

public int write(ByteBuffer buff) {
 int pos = buff.position();
 buff.putInt(eventType);
 NIOUtils.putStr(buff, playerId);
 NIOUtils.putStr(buff, sessionId);
 buff.putInt(gameId);
 NIOUtils.putStr(buff, gameName);
 buff.putInt(numRecipients);
 for (int i = 0;i < numRecipients;i++)
 NIOUtils.putStr(buff, recipients[i]);
 NIOUtils.putStr(buff, message);
 return buff.position() - pos;
}
public void read(ByteBuffer buff) {
 eventType = buff.getInt();
 playerId = NIOUtils.getStr(buff);
 sessionId = NIOUtils.getStr(buff);
 gameId = buff.getInt();
 gameName = NIOUtils.getStr(buff);
 numRecipients = buff.getInt();
 recipients = new String[numRecipients];
 for (int i = 0;i < numRecipients;i++)
 recipients[i] = NIOUtils.getStr(buff);
 message = NIOUtils.getStr(buff);
}

Note that the write method returns an integer specifying the number of bytes written. This is important for the framework's over-the-wire event protocol.

Over-the-Wire Event Protocol

To accommodate a variety of event formats, we need to precede the event data with a small header when sending the events over the network. This format of the header and payload is shown in Screenshot. First the clientId, the GameName hash code, and the payloadSize (in bytes) are sent, each of which is a 4-byte integer. Then the payload is sent, which is payloadSize bytes in length.

Screenshot Over-the-wire event format.

We discuss how to actually read and write this data, and instantiate the proper GameEvents on the receiving end when we introduce the SelectAndRead and EventWriter classes of the server implementation.

Server Implementation

The server side of the framework consists of a small number of classes, as follows:

• EventWriter
• GameController
• GameServer
• SelectAndRead

We cover each of these in detail because they form the foundation of the framework.

GameServer and GameController

The GameServer class is the centerpiece of the server app and contains the app's main() method. The duties of the GameServer are to accept incoming socket connections, keep track of connected clients, and manage the GameControllers. The code for setting up the ServerSocket and accepting connections is similar to the code in ChatterServer, as can be seen in Listing 6.8.

Listing 6.8 GameServer initServerSocket() and run() Methods

private void initServerSocket() {
 try {
 sSockChan = ServerSocketChannel.open();
 sSockChan.configureBlocking(false);
 InetAddress addr = InetAddress.getLocalHost();
 sSockChan.socket().bind(new InetSocketAddress(addr, Globals.PORT));
 selector = Selector.open();
 SelectionKey acceptKey = sSockChan.register(selector,
 SelectionKey.OP_ACCEPT);
 }
 catch (Exception e) {
 log.error("error initializing ServerSocket", e);
 }
}
public void run() {
 init();
 log.info("******** GameServer running ********");
 running = true;
 int numReady = 0;
 while (running) {
 try {
 selector.select();
 Set readyKeys = selector.selectedKeys();
 Iterator i = readyKeys.iterator();
 while (i.hasNext()) {
 SelectionKey key = (SelectionKey) i.next();
 i.remove();
 ServerSocketChannel ssChannel =
 (ServerSocketChannel) key.channel();
 SocketChannel clientChannel = ssChannel.accept();
 selectAndRead.addNewClient(clientChannel);
 log.info("got connection from: " +
 clientChannel.socket().getInetAddress());
 }
 }
 catch (IOException ioe) {
 log.warn("error during serverSocket select(): ", ioe); }
 catch (Exception e) {
 log.error("exception in run()", e);
 }
 }
}

The GameServer code is similar to that of ChatterServer. There are a few differences worth noting, however. The first is that we use a selector for the ServerSocketChannel. Just as you can use a selector to multiplex client channels for reading, you can use selectors to multiplex ServerSocketChannels. This is used when a server app needs to listen on either multiple addresses or ports. Our framework doesn't use this capability currently, but we talk more about this later when discussing remote administration consoles. The other difference in the run() method is that after a connection is accepted, it is passed off to a class called SelectAndRead for further processing.

SelectAndRead

Much like its name implies, SelectAndRead houses the selector that multiplexes client channels and reads GameEvents from those channels. Complete events are passed off to GameControllers based on the GameName hash code found in the event header. New client connections are handed off to SelectAndRead from the GameServer via SelectAndRead.addNewClient(). The SocketChannel for the client is added to a LinkedList, and selector.wakeup() is called to interrupt SelectAndRead's blocking select() call. This has the effect of breaking the SelectAndRead thread out of the select() method and then calling checkNewConnections(), which runs through the LinkedList of new client SocketChannels, registering them with the selector. Listing 6.9 has the code for both methods.

Listing 6.9 SelectAndRead, addNewClient(), and checkNewConnections()

public void addNewClient(SocketChannel clientChannel) {
 synchronized (newClients) {
 newClients.addLast(clientChannel);
 }
 // force selector to return
 // so our new client can get in the loop right away
 selector.wakeup();
}
private void checkNewConnections() {
 synchronized (newClients) {
 while (newClients.size() > 0) {
 try {
 SocketChannel clientChannel =
 (SocketChannel)newClients.removeFirst();
 clientChannel.configureBlocking( false);
 clientChannel.register( selector, SelectionKey.OP_READ,
 new Attachment());
 }
 catch (ClosedChannelException cce) {
 log.error("channel closed", cce);
 }
 catch (IOException ioe) {
 log.error("ioexception on clientChannel", ioe);
 }
 }
 }
}

SelectAndRead uses similar methods to those found in ChatterServer for reading data from the client channels, with a few notable exceptions. Instead of simply reading strings from the channel and checking for linefeeds to separate messages, it must now look for complete GameEvents. Because the event format is implementation specific, the framework reads only the raw bytes of the event and delegates the job of instantiating the GameEvent to the GameController implementation via createGameEvent(). Listing 6.10 shows the select() method along with the methods for reading and delegating the GameEvent.

Listing 6.10 Reading GameEvents

private void select() {
 try {
 selector.select();
 Set readyKeys = selector.selectedKeys();
 Iterator i = readyKeys.iterator();
 while (i.hasNext()) {
 SelectionKey key = (SelectionKey) i.next();
 i.remove();
 SocketChannel channel = (SocketChannel) key.channel();
 Attachment attachment = (Attachment) key.attachment();
 long nbytes = channel.read(attachment.readBuff);
 // check for end-of-stream condition
 if (nbytes == -1) {
 log.info("disconnect: " +
 channel.socket().getInetAddress() +
 ", end-of-stream");
 channel.close();
 }
 // check for a complete event
 try {
 if (attachment.readBuff.position() >=
 attachment.HEADER_SIZE) {
 attachment.readBuff.flip();
 // read as many events as are available in the buffer
 while (attachment.eventReady()) {
 GameEvent event = getEvent(attachment);
 delegateEvent(event, channel);
 attachment.reset();
 }
 // prepare for more channel reading
 attachment.readBuff.compact();
 }
 }
 catch (IllegalArgumentException e) {
 log.error("illegal arguement exception", e);
 }
 }
 }
 catch (IOException ioe) {
 log.warn("error during select(): ", ioe);
 }
 catch (Exception e) {
 log.error("exception during select()", e);
 }
}
private GameEvent getEvent(Attachment attachment) {
 GameEvent event = null;
 ByteBuffer bb = ByteBuffer.wrap(attachment.payload);
 // get the controller and tell it to instantiate an event for us
 GameController gc =
 gameServer.getGameControllerByHash(attachment.gameNameHash);
 if (gc == null)
 return null;
 event = gc.createGameEvent();
 // read the event from the payload
 event.read(bb);
 return event;
}
private void delegateEvent(GameEvent event, SocketChannel channel)
{
 if (event != null && event.getGameName() == null) {
 log.error("GameServer.handleEvent() : gameName is null");
 return ;
 }
 GameController gc;
 gc = gameServer.getGameController(event.getGameName());
 if (gc == null) {
 log.error("No GameController for gameName: "
 + event.getGameName());
 return ;
 }
 Player p = gameServer.getPlayerById(event.getPlayerId());
 if (p != null) {
 if (p.getChannel() != channel) {
 log.warn("player is on a new channel, must be reconnect.");
 p.setChannel(channel);
 }
 }
 else {
 // first time we see a playerId, create the Player object
 // and populate the channel, and also add to our lists
 p = gc.createPlayer();
 p.setPlayerId(event.getPlayerId());
 p.setChannel(channel);
 gameServer.addPlayer(p);
 gc.handleEvent(event);
 }

As SelectAndRead pulls data from the SocketChannel, it is stored temporarily in buffers in the Attachment class. The Attachment class is just a holding cell for this data and contains only methods for verifying that the header and payload data has been read. After every read, the event header is checked; if there are at least payloadSize bytes in the readBuffer, the event is complete. The GameName hash code is used to fetch the instance of the appropriate GameController from the GameServer. Next, the GameController's createGameEvent() method is called to instantiate a new event, and the event reads itself from the contents of Attachment.readBuffer. A bit of complication with the way initial user connections are handled and players are created bears explanation. The GameServer is the keeper of the master player lists. When SelectAndRead gets a complete event, it first checks the playerId and asks the GameServer if it has seen that ID before. If it hasn't, this is a new connection and it first calls on the GameController to instantiate a new player using the createPlayer() method. When the GameController gets a login event from a player, it must query the GameServer to get the player object that was previously instantiated. This seems a bit roundabout, but it enforces strict handling of player creation and tracking by the GameServer.

GameControllers

GameControllers encapsulate the server-side game logic and enable you to have a variety of different games running simultaneously in one GameServer. All GameControllers must extend the GameController base class, which, in turn, extends Wrap. Thus, each GameController has its own EventQueue and thread pool backing it. The GameServer, SelectAndRead, and GameEvent implementations form a team that provides the foundation for handling the client/server communication and hides the details of the network implementation (in this case, NIO channels) from the rest of the system. In other words, GameControllers don't have to care how the events are read or written-they care only that incoming GameEvents appear in the EventQueue and outgoing events get dropped into the EventWriter's EventQueue. You could rewrite the underlying network code to use UDP, HTTP, or other means of moving the events around without having to touch the GameControllers. GameControllers are loaded dynamically by the GameServer during initialization. This approach prevents the GameServer from being tied to a fixed set of GameControllers. In fact, it doesn't even know their names until they are first loaded. Listing 6.11 shows the code in GameServer that loads the GameController instances.

Listing 6.11 GameServer, getGameController(), and loadGameController() Methods

private void loadGameControllers() {
 log.info("loading GameControllers");
 // grab all class files in the same directory as GameController
 String baseClass =
 "com/hypefiend/javagamebook/server/controller/GameController.class";
 File f = new File( this.getClass(
 ).getClassLoader().getResource(baseClass).getPath());
 File[] files = f.getParentFile().listFiles( );
 if (files == null) {
 log.error("error getting GameController directory");
 return ;
 }
 for ( int i = 0; ( i < files.length); i++) {
 String file = files[i].getName( );
 if (file.indexOf( ".class") == -1)
 continue;
 if (file.equals("GameController.class"))
 continue;
 try {
 String controllerClassName = CONTROLLER_CLASS_PREFIX +
 file.substring(0, file.indexOf(".class"));
 log.info("loading class: " + controllerClassName);
 Class cl = Class.forName(controllerClassName);
 // make sure it extends GameController
 if (!GameController.class.isAssignableFrom(cl)) {
 log.warn("class file does not extend GameController: " +
 file);
 continue;
 }
 // get an instance and initialize
 GameController gc = (GameController) cl.newInstance();
 String gameName = gc.getGameName();
 gc.init(this, getGameConfig(gameName));
 // add to our controllers hash
 gameControllers.put("" + gameName.hashCode(), gc);
 log.info("loaded controller for gameName: " + gameName +
 ", hash: " + gameName.hashCode());
 }
 catch (Exception e) {
 log.error("instantiating GameController,file: " + file, e);
 }
 }
}

The hash code of the GameName is used as the key for storing the GameControllers in a Hashtable. This key is also used in the header in the over-the-wire protocol, as discussed earlier. The GameController base class defines a few abstract methods that its subclasses must implement:

• protected abstract void initController(GameConfig gc) This method is for GameControllers to perform any initialization that they require.
• public abstract String getGameName() Must return the GameName for the controller.
• public abstract Player createPlayer() Factory method that must return an object that implements the Player interface.
• public abstract GameEvent createGameEvent() Factory method that must return an object that implements the GameEvent interface.

The two factory methods, createPlayer() and createGameEvent(), allow the core GameServer to instantiate these objects on behalf of the controllers as needed, without requiring knowledge of the implementation a particular controller is using. GameControllers are responsible for pulling incoming events from their EventQueue (deposited there by the SelectAndRead class, as described previously), processing those events, and dropping outgoing events into the EventWriter's EventQueue. The processEvent() method (from the Wrap class, which GameController extends) is called every time a GameEvent arrives in the EventQueue. The method has the following signature:

protected abstract void processEvent(GameEvent event);

NOTE It is important to realize that by extending Wrap, all controllers are inherently multi-threaded and thus must use synchronization accordingly when processing events.

Having all controllers extend the GameController base class enables the framework to take care of some housekeeping, such as initialization and logging in a uniform manner across all controllers. The base class handles initialization of the Wrap and a Log4J category for the controller. The base implementation also manages the run cycle, pulling events from the incoming queue and passing them to the processEvent() method. By restricting controllers to the method defined previously, the framework provides a very controlled atmosphere in which to run the game logic. Note that each controller gets a reference to the GameServer as well. You might find that you need additional latitude depending on the complexity of your games, so by all means, feel free to extend the powers given to the controllers. The idea is just to have a central place where you can make changes or additions that will affect all controllers, and to isolate the GameController logic code from the network communication code.

EventWriter

EventWriter is simply a Wrap that handles a queue of outgoing events. Events are written using the utility method in NIOUtils, channelWrite(). When sending broadcast events, it grabs the list of recipients from the GameEvent and then sends the event to each player in turn.

The Client

The game client uses much of the code in the common classes detailed earlier and adds two classes of its own: GameClient and NIOEventReader. GameClient is an abstract base class for the client apps. Games can extend this class and need only implement the following methods:

• getGameName Returns the game's GameName.
• createGameEvent Factory method for creating game-specific GameEvents.
• createLoginEvent Factory method for creating a game-specific GameEvent used to log in to the server.
• createDisconnectEvent Factory method for creating a game-specific GameEvent for disconnections.
• processIncomingEvents Handles incoming GameEvents from the GameServer.

GameClient also handles initial connections to the server and writing of outgoing events. NIOEventReader is the client-side version of SelectAndRead. It is simplified because the client is reading from only a single SocketChannel, but otherwise it is nearly identical. NIOEventReader delegates event creation and processing to a GameClient instead of a GameController, as in SelectAndRead.