In Sauce, we have a small, tight Streams library to handle the input and output of data in a standardized manner. After all, a game engine isn’t very exciting without the ability to read in configuration and asset data.
We use a stream as our main abstraction for data that flows in and out of the engine. In the case of input, the engine doesn’t need to know the source of those bytes; they could be coming from a file, memory, or over the network. The same holds true for output data. This is an extremely important feature that we can exploit for a number of uses, including testing.
Also, it should be noted that a stream is not responsible for interpreting the data. It is only responsible for reading bytes from a source or writing bytes to a destination.
As you might expect, we have two top level interfaces:
OutputStream. We’ve seen code bases where these are merged into a single
Stream class that can read and write; however, we prefer to keep the operations separate and simple. Each of these interfaces has a number of implementations as described below.
The primary function for an
InputStream is to read bytes.
Also, we store the endianness of the stream. This is an important property of the stream for the code that interprets the data. If the stream and the host platform have different endians, the bytes need to be appropriately swapped after being read from the
Our Streams library features three types of input streams:
- File Input Stream
- Memory Input Stream
- Volatile Input Stream
File Input Stream
This is probably the first implementation of
InputStream that comes to mind. The
FileInputStream is an adaptor from our file system routines to open and read from a file to the
As an optimization, we buffer the input from the file as read requests are made. However, this is an implementation detail that is not exposed in the class interface; we could just as well read directly from the file — the callsite shouldn’t know or care.
Memory Input Stream
MemoryInputStream implements the
InputStream interface for a block of memory. In our implementation, this block can be sourced from an array of bytes or a string.
This implementation in particular is extremely useful for mocking up data for tests. For example, instead of creating separate file for each JSON test, we can put the contents into a string and wrap that in a
MemoryInputStream for processing.
Volatile Input Stream
Simply put, the
VolatileInputStream is an
InputStream implementation for an external block of memory.
For safety, the
MemoryInputStream makes a copy of the source buffer. This is because in many cases, the lifetime of an
InputStream may be unknown or exceed the lifetime of the source buffer.
Of course, in the cases when we do know the lifetime of the source buffer will not exceed the use of the
InputStream, we can make direct use of the source buffer. This is the core principle behind the
The primary function for an
OutputStream is to write bytes.
Also, just like in the
InputStream, we store the endianness of the stream. This is an important property of the stream for the code that writes the data. If the stream and the host platform have different endians, the bytes need to be appropriately swapped before being written to the
Our Streams library features two types of output streams:
- File Output Stream
- Memory Output Stream
File Output Stream
Similar to the input version, a
FileOutputStream is a wrapper around our file system routines to open and write to a file.
However, unlike the
FileInputStream, we do not buffer the output.
Memory Output Stream
MemoryOutputStream implements the
OutputStream interface for a block of memory. The internal byte buffer grows as bytes are written.
For convenience, we added a method to fetch the buffer contents as a string.
Again, this is extremely useful for testing code like file writers.
Readers and Writers
Admittedly, the stream interfaces are very primitive. They are so primitive, in fact, that they can be a bit painful to use by themselves in practice. Consequently, we wrote a few helper classes to operate on a higher level than just bytes.
We’ve found this to have been an excellent choice. It is not unusual for a single stream to be passed around to more than one consumer or producer. Separating the data (stream) from the operator (reader/writer) provides us the flexibility needed and the opportunity to expose a more refined client interface.
InputStreams, we implemented a
BinaryStreamReader and a
BinaryStreamReader can read bytes and interpret them into primitive data types, as well as a couple of our Core data types: strings and guids. We use this extensively for reading data from our proprietary file formats.
TextStreamReader can read the stream character by character, or whole strings at a time. This makes it ideal for performing text processing tasks like decoding JSON.
OutputStreams, we implemented a parallel pair of writers:
TextStreamWriter. In both, we perform the appropriate byte swapping internally when writing multi-byte data types.
BinaryStreamWriter can take the same set of data types supported by the Reader and write their bytes to the given
TextStreamWriter can write characters or strings to the given
The Sauce Streams library has been a vital component to our development. We use it to read in models, textures, and configuration files; and we use it to write out saved games and screenshots.
We hope that this high-level discussion will help our readers with designing their own set of stream classes.