Update 2015/06/07

It’s been a long while since our last update post. A lot has transpired since then so I’ll try to do my best to outline the highlights.

Input Delivery

First and foremost, we re-architected how input is delivered throughout the engine. Previously, the input state was fetched from the devices and packaged into a single data object, which was then passed to whatever systems required input. While simple, this paradigm had many drawbacks, which I hope to discuss at length in a future post.

The new system is event driven. This allows us to employ a layer system, where “higher” layers in the stack can consume input events so that “lower” layers don’t try to handle them as well. A system like this is essential for games that have interactive UI elements overlaid atop the game scene.

This was by far the most significant change we have made in the last year or so, but it is well worth the benefits it affords us.

Compression

We added a Compress library. At present, this is a light wrapper around zlib, though it could include other compression algorithms in the future should the need arise.

Our primary use case for compression was our proprietary asset file format: the Sauce Engine Assembly (*.SEASM). Below is a table comparing the results between our uncompressed and compressed file versions:

Model SEASM v1 SEASM v2 Diff
Pawn 80.47 KB 31.66 KB 39%
Bunny 1.73 MB 1.14 MB 66%
Dragon 21.66 MB 11.94 MB 55%

Pawn-Bunny-Dragon

Streams

We added a new type of input stream to the Streams library: VolatileInputStream. This allowed us to significantly shorten our file load times. You can read about the details of the VolatileInputStream in the Streams post.

Asset Pipeline

We spent a good amount of time solidifying our FBX model importer. It can now load geometry and basic material data from Blender. Extracting data from the FBX SDK in a robust manner is no easy feat.

Also, a lot of progress was made on extracting animation data as keyframes from FBX, yet, sadly, this is still incomplete.

Interface

Thanks to the event driven input system, the Interface library now has keyboard support. We also added a primitive implementation of focus.

Furthermore, we reworked how the Anchor property affects the control placement and added the ability to “dock” controls. The result is that controls can now be arranged in the same way that .NET supports.

We also finally added a TextBox control. This was the last of the “standard control set” we had been hoping to implement. A TextBox has a lot of functionality under the hood to make them work as expected: text input, caret movement, selection, and even Clipboard support (Ctrl+X, Ctrl+C, Ctrl+V).

TextBox

JSON

The most recent addition to the engine is the JSON library. We now use JSON as our config file format (previously they were XML). You can read about the details in the JSON post.

Visual Studio

Last but not least, we migrated to from Visual Studio 2010 Express to Visual Studio 2013 Community Edition. This is fantastic! As soon as we found out that the new edition was released and offered the same feature set as the professional versions, we jumped on it.

This required a few changes to our VSGEN tool which generates the project and solution files with our configuration settings.

JSON

Early last month, I set out to add support for JSON into our game engine. To my surprise, it turned out to be a fun and rewarding adventure.

JSON Logo

JSON is a very nice format that is fairly easy to parse. Its feature set is small and well defined, including:

  • explicit values for null, true, and false
  • numbers (integers and floating-point)
  • strings
  • arrays
  • hash tables

This feature set is perfect for configuration files, stylesheets, etc. In the past, I have used XML for these sort of things, but JSON is much more direct and compact.

Initially, I reached for an external library to wrap, just as we have done for many of the other file formats we support, namely: PNG, XML, FBX, and OGG. Of course, when it comes to external libraries, your mileage will vary. For example, we use TinyXML 2, as the basis for our XML library; it was a real pleasure to use — a very straightforward, well designed interface. The FBX SDK, on the other hand, is pretty atrocious.

Unfortunately, I wasn’t very satisfied when it came to JSON. Many of the C++ JSON libraries out there make use of STL and/or Boost, dependencies we have striven to avoid. Eventually I settled on RapidJSON due to its high praise on the web; however, about half way through my wrapper implementation, I concluded that its interface is not as clean and “wrappable” as I had originally thought it to be.

After some reflection, I decided that the best way forward was to roll my own. I found that rolling your own is an excellent decision for a few reasons:

First, the JSON format is relatively small, unambiguous, and well documented. This allows you to focus on the architecture and interface of your wrapper. I found the experience both valuable and refreshing.

Second, you are able to employ the use of your native data structures. Naturally, this is a great way to test your functionality and interface. In the case of Sauce, I was able to leverage the following Core structures: String, Vector, Array, and HashMap.

Last, but not least, I found it to be a whole lot of fun! It’s been a while since I’ve done anything like implementing a format encoder and decoder. Hopefully when you’re finished, you feel the same.

After I finished our JSON library, I converted our config files from XML to JSON with very little effort. The result is that our config files are more compact than they were with XML, and now we have the utilities required for future development. Overall, I feel it was well worth the time and effort.

Streams

In Sauce, we have a small, tight Streams library to handle the input and output of data in a standardized manner. After all, a game engine isn’t very exciting without the ability to read in configuration and asset data.

We use a stream as our main abstraction for data that flows in and out of the engine. In the case of input, the engine doesn’t need to know the source of those bytes; they could be coming from a file, memory, or over the network. The same holds true for output data. This is an extremely important feature that we can exploit for a number of uses, including testing.

Also, it should be noted that a stream is not responsible for interpreting the data. It is only responsible for reading bytes from a source or writing bytes to a destination.

As you might expect, we have two top level interfaces: InputStream and OutputStream. We’ve seen code bases where these are merged into a single Stream class that can read and write; however, we prefer to keep the operations separate and simple. Each of these interfaces has a number of implementations as described below.

Input Streams

InputStreams

The primary function for an InputStream is to read bytes.

Also, we store the endianness of the stream. This is an important property of the stream for the code that interprets the data. If the stream and the host platform have different endians, the bytes need to be appropriately swapped after being read from the InputStream.

Our Streams library features three types of input streams:

  • File Input Stream
  • Memory Input Stream
  • Volatile Input Stream

File Input Stream

This is probably the first implementation of InputStream that comes to mind. The FileInputStream is an adaptor from our file system routines to open and read from a file to the InputStream interface.

As an optimization, we buffer the input from the file as read requests are made. However, this is an implementation detail that is not exposed in the class interface; we could just as well read directly from the file — the callsite shouldn’t know or care.

Memory Input Stream

The MemoryInputStream implements the InputStream interface for a block of memory. In our implementation, this block can be sourced from an array of bytes or a string.

This implementation in particular is extremely useful for mocking up data for tests. For example, instead of creating separate file for each JSON test, we can put the contents into a string and wrap that in a MemoryInputStream for processing.

Volatile Input Stream

Simply put, the VolatileInputStream is an InputStream implementation for an external block of memory.

For safety, the MemoryInputStream makes a copy of the source buffer. This is because in many cases, the lifetime of an InputStream may be unknown or exceed the lifetime of the source buffer.

Of course, in the cases when we do know the lifetime of the source buffer will not exceed the use of the InputStream, we can make direct use of the source buffer. This is the core principle behind the VolatileInputStream.

Output Streams

OutputStreams

The primary function for an OutputStream is to write bytes.

Also, just like in the InputStream, we store the endianness of the stream. This is an important property of the stream for the code that writes the data. If the stream and the host platform have different endians, the bytes need to be appropriately swapped before being written to the OutputStream.

Our Streams library features two types of output streams:

  • File Output Stream
  • Memory Output Stream

File Output Stream

Similar to the input version, a FileOutputStream is a wrapper around our file system routines to open and write to a file.

However, unlike the FileInputStream, we do not buffer the output.

Memory Output Stream

The MemoryOutputStream implements the OutputStream interface for a block of memory. The internal byte buffer grows as bytes are written.

For convenience, we added a method to fetch the buffer contents as a string.

Again, this is extremely useful for testing code like file writers.

Readers and Writers

Admittedly, the stream interfaces are very primitive. They are so primitive, in fact, that they can be a bit painful to use by themselves in practice. Consequently, we wrote a few helper classes to operate on a higher level than just bytes.

We’ve found this to have been an excellent choice. It is not unusual for a single stream to be passed around to more than one consumer or producer. Separating the data (stream) from the operator (reader/writer) provides us the flexibility needed and the opportunity to expose a more refined client interface.

Readers

For InputStreams, we implemented a BinaryStreamReader and a TextStreamReader.

The BinaryStreamReader can read bytes and interpret them into primitive data types, as well as a couple of our Core data types: strings and guids. We use this extensively for reading data from our proprietary file formats.

The TextStreamReader can read the stream character by character, or whole strings at a time. This makes it ideal for performing text processing tasks like decoding JSON.

Writers

For OutputStreams, we implemented a parallel pair of writers: BinaryStreamWriter and TextStreamWriter. In both, we perform the appropriate byte swapping internally when writing multi-byte data types.

The BinaryStreamWriter can take the same set of data types supported by the Reader and write their bytes to the given OutputStream.

The TextStreamWriter can write characters or strings to the given OutputStream.

Summary

The Sauce Streams library has been a vital component to our development. We use it to read in models, textures, and configuration files; and we use it to write out saved games and screenshots.

We hope that this high-level discussion will help our readers with designing their own set of stream classes.