Serialization with JSON
Prerequisites
- UObject Base Class
- Variants - Generic Containers
- Serializable Objects
Article
While a database is great for storing collections of information, sometimes you need to store a bunch of disparate pieces of information. Configuration settings are a great example: preferred monitor, screen resolution, windowed vs. full screen, keyboard mappings, texture map resolutions... you could store these in a database and use our SQLite or MySQL loaders to get them, but it overcomplicates things. In addition, databases files are not easily editable by a user and often need a separate version control system.
These are all use cases for a flat file format - XML, JSON, INI, and other formats are great choices for configuration data. In this article, we'll look at JSON, since it doesn't have the overhead of XML and is more flexible than INI.
While you can use JSON for storing collections of objects, a better approach would be to consider doing so in many small files (eg. one file per game object, asset, etc.). This is what Unity does. Most flat file formats don't support in-place modifications to a subset of the data, so you'll need to re-create the file from scratch every time a piece of information changes. If you're working with large scenes, this will become a drain on the programming team as every save will take longer.
We'll be leveraging a third party library nlohmann/json to do most of the heavy lifting so we can focus on integrating it into our engine and architecture. As of the time of this article, it has good sponsorship, recent commits, and an active user base. It's also a header-only library, which is great for integrating into your engine, since they don't require matching the compilation settings in a separate project.
Let's take a look at the JSONLoader class:
#include <json.hpp>
using json = nlohmann::json;
class UJSONLoader : public UDataLoader {
public:
UJSONLoader() = default;
~UJSONLoader() = default;
/**
* Open / close this data source
*/
bool Open(std::string location);
/**
* Get all objects of a specific type
*/
std::vector<UDataRecord*> GetRecords(std::string type, std::map<std::string, UVariant> search = std::map<std::string, UVariant>());
/**
* Save records
*/
void SaveRecords(std::string type, std::vector<UDataRecord*> records);
private:
// Helper functions for conversions
UDataRecord* __JSONObjectToUDataRecord(json obj);
UVariant __JSONValueToUVariant(json value);
json __UDataRecordToJSONObject(UDataRecord* record);
json __UVariantToJSONValue(UVariant value);
protected:
// Loaded object
json m_json;
};This implements three of our base DataLoader functions: Open, which will open and read a JSON file into memory; GetRecords, which will return all records of a specific type; and SaveRecords, which will write all records of a specific type back to our JSON file. We did not implement the Close() function here - as you'll see, the Open and SaveRecords function open, stream data in or out, and then close it again.
Under that, we have some more interesting helper functions that convert JSON data types into our Variant data types and vice versa. To handle the special case of nested objects (unlikely, but we may find a use case), we will again load them as nested DataRecords which can be deserialized by the object that uses them.
Finally, we have our loaded json object in a variable called m_json.
Here is the Open function:
bool UJSONLoader::Open(std::string location) {
m_location = location;
std::ifstream ifs(location);
if (ifs.failbit == true) {
// File error
UASSERT(false, "JSON file error.");
return(false);
}
m_json = json::parse(ifs, nullptr, false);
if (m_json.is_discarded()) {
// Parse error
UASSERT(false, "JSON parse error.");
return(false);
}
return(true);
}Most of this is error handling - we store the location of the file (since we'll use it again in SaveRecords), open an input file stream to the file, and then attempt to parse the file. If we cannot open the file, or the JSON is badly formatted, we'll throw an error. The json::parse parameters here are to the file stream, an optional callback function, and whether we want to throw exceptions, which we set to false. If all went well, we return true.
Next we're going to diverge into our helper functions, since they are used in GetRecords. JSON files are stored from a root object node and inside of that, to match our own format, each property of our root object will be an array or collection of objects. Since this is configuration data (for our example), each collection will probably only have one object. To illustrate, here is an example of a file we'll load for some application settings:
{
"settings": [
{
"windowWidth": 800.0",
"windowHeight": 600.0,
"windowName": "Test Game"
}
]
}You can see we have a root node, which has a settings property - that contains an array, indicated by the square brackets, that has one object in it with our settings.
Structured the same way as our other DataLoader classes, we can expect using this information to look something like this:
UJSONLoader* jsonLoader = new UJSONLoader();
jsonLoader->Open("appsettings.json");
auto settings = jsonLoader->GetRecords("settings")[0];
window->Create(
settings->Get("windowName").AsString(),
settings->Get("windowWidth").AsInt(),
settings->Get("windowHeight").AsInt(),
false);To make this work, we need to be able to translate JSON objects into DataRecords. To do that, we need to translate JSON data types into Variant data types. We have two functions to help with this: __JSONObjectToUDataRecord will convert a passed in JSON object to a UDataRecord:
UDataRecord* UJSONLoader::__JSONObjectToUDataRecord(json obj) {
UASSERT(obj.is_object(), "Not an object.");
UDataRecord* record = new UDataRecord();
for (auto it = obj.begin(); it != obj.end(); it++) {
record->Set(it.key(), __JSONValueToUVariant(it.value()));
}
return(record);
}As you can see, this function simply creates a new DataRecord and iterates over its properties, translating those from JSON values to Variants by calling another function, __JSONValueToUVariant:
UVariant UJSONLoader::__JSONValueToUVariant(json value) {
if (value.is_number_float()) {
return(UVariant(value.get<float>()));
}
if (value.is_number_integer()) {
return(UVariant(value.get<int>()));
}
if (value.is_number_unsigned()) {
return(UVariant(value.get<unsigned int>()));
}
if (value.is_boolean()) {
return(UVariant(value.get<bool>()));
}
if (value.is_string()) {
return(UVariant(value.get<std::string>()));
}
if (value.is_object()) {
if (value.contains("x") && value.contains("y")) {
if (value.size() == 2) {
// Process vector2
vector2 vec;
vec.x = value.at("x").get<float>();
vec.y = value.at("y").get<float>();
return(UVariant(vec));
}
if (value.contains("z")) {
if (value.size() == 3) {
// Process as vector3
vector3 vec;
vec.x = value.at("x").get<float>();
vec.y = value.at("y").get<float>();
vec.z = value.at("z").get<float>();
return(UVariant(vec));
}
if (value.contains("w") && value.size() == 4) {
// Process as quaternion
quaternion quat;
quat.x = value.at("x").get<float>();
quat.y = value.at("y").get<float>();
quat.z = value.at("z").get<float>();
quat.w = value.at("w").get<float>();
return(UVariant(quat));
}
}
}
if (value.contains("a") && value.contains("r") && value.contains("g") && value.contains("b") && value.size() == 4) {
// Process as vector4
vector4 vec;
vec.w = value.at("a").get<float>();
vec.x = value.at("r").get<float>();
vec.y = value.at("g").get<float>();
vec.z = value.at("b").get<float>();
return(UVariant(vec));
}
// All other objects
return(UVariant(__JSONObjectToUDataRecord(value)));
}
UASSERT(false, "Unrecognized type.");
return(0);
}JavaScript - and therefore JSON - like other weakly typed languages, has a generic Value. That Value can be casted to different types, seen here using the value.get<type>() function. A value of "1" can be casted to an integer value of 1, a string value of "1", a boolean value of true, or a floating point value of approximately 1.00000. This function iterates, using if statements, over each possible data type, and if applicable, maps it to the same Variant data type. For numeric, boolean, and string data types, this is fairly straightforward.
For objects, we have a couple of special cases: our Variant supports 2-, 3-, and 4-component vectors, as well as quaternions. Inside of the is_object block, you can see that we check to see if the object has "x", and "y" properties. If so, and there are only two properties, we assume it is a vector2. If it has a "z" property and only 3 properties, we assume it is a vector3. Finally, if it has a "w" component and only 4 properties, we consider it as a quaternion. But what about a vector4? Unfortunately, a vector4 uses the same x, y, z, and w components as our quaternion. However, the GLM library mimics a concept in OpenGL that allows multiple names for the same properties - in this case, a vector's r, g, b, and a properties actually point to the same variables as x, y, z, and w. We can copy this for our vector4... if our object has the properties a, r, g, and b, and those are the only 4 properties, we'll assume it's a vector4. We'll also implement the same logic in the save function.
Finally, any other objects are loaded as nested DataRecords.
Since we can now load the properties of objects as Variants and objects as DataRecords, we can revisit our GetRecords method:
std::vector<UDataRecord*> UJSONLoader::GetRecords(std::string type, std::map<std::string, UVariant> search) {
std::vector<UDataRecord*> retval;
UASSERT(m_json.contains(type), "Key not found.");
// Populate retval with objects
auto obj = m_json.at(type);
for (auto it = obj.begin(); it !=obj.end(); it++) {
retval.push_back(__JSONObjectToUDataRecord(it.value()));
}
// Filter
if (search.size()) {
for (int i = retval.size() - 1; i >= 0; i--) {
for (auto sit = search.begin(); sit != search.end(); sit++) {
if (retval[i]->Get(sit->first) != sit->second) {
// Does not match filters, remove
retval.erase(retval.begin() + i);
}
}
}
}
return(retval);
}The first part is fairly straightforward - convert all JSON objects inside of the root object property into DataRecords. The filtering gets a little loopy, but boils down to: for each object in the returned set, loop over the search criteria and see if it meets it. If it does not, remove it from the returned set.
That's all for retrieving records. The code snippet and sample file above now work as expected and we can fetch configuration data.
Next let's look at saving records, which will be a matter of converting data in the opposite direction. This time let's start top-down with the SaveRecords function:
void UJSONLoader::SaveRecords(std::string type, std::vector<UDataRecord*> records) {
// Convert to JSON from UDataRecords
json jsarr = json::array();
auto it = records.begin();
for (; it != records.end(); it++) {
jsarr.push_back(__UDataRecordToJSONObject(*it));
}
// If this is a new file or collection (first save), create a root node
if (m_json.is_object() == false) {
m_json = json::object();
}
m_json[type] = jsarr;
std::ofstream out(m_location);
out << m_json.dump(4, ' ');
}First, to keep the same structure as our other loaders, we start with an array that we will put our object records in to. Then, we iterate through our DataRecords and convert them to JSON objects - we'll return to this momentarily. Next, we need to save the data to a file. If the root object in m_json isn't yet initialized, we create a new root object. Then, we update the property of our object to be our new array of data, and we dump that out to an output file stream. The json::dump function takes two parameters: the number of characters to indent by for each level of indentation, and the charater to use. You could set this to -1 to get a tightly packed JSON file, or to 1 and the TAB character (ASCII decimal 9) - we use 4 space characters.
Let's see how we convert DataRecords to JSON objects:
json UJSONLoader::__UDataRecordToJSONObject(UDataRecord* record) {
json jsobj = json::object();
// Build our JSON object field by field
auto keys = record->GetKeys();
for (auto it = keys.begin(); it != keys.end(); it++) {
UVariant v = record->Get(*it);
jsobj[*it] = __UVariantToJSONValue(v);
}
return(jsobj);
}The opposite of our JSON to DataRecords helper function, here we create a new JSON object, iterate over the record keys, and set the properties of the object from Variant values to JSON values. Let's look at the __UVariantToJSONValue function:
json UJSONLoader::__UVariantToJSONValue(UVariant value) {
if (value.IsBool()) {
return(json(value.AsBool()));
}
if (value.IsFloat()) {
return(json(value.AsFloat()));
}
if (value.IsInt() || value.IsInt64()) {
return(json(value.AsInt()));
}
if (value.IsUInt() || value.IsUInt64()) {
return(json(value.AsUInt()));
}
if (value.IsString()) {
return(json(value.AsString()));
}
if (value.IsVector2()) {
vector2 val = value.AsVector2();
return(json({ {"x", val.x}, {"y", val.y}}));
}
if (value.IsVector3()) {
vector3 val = value.AsVector3();
return(json({ {"x", val.x}, {"y", val.y}, {"z", val.z}}));
}
if (value.IsVector4()) {
vector4 val = value.AsVector4();
// Use argb to differentiate from quaternion on load
return(json({ {"a", val.w}, {"r", val.x}, {"g", val.y}, {"b", val.z}}));
}
if (value.IsQuaternion()) {
quaternion val = value.AsQuaternion();
return(json({ {"x", val.x}, {"y", val.y}, {"z", val.z}, {"q", val.w} }));
}
if (value.IsObject()) {
// Is this object a UDataRecord? eg. already serialized
UDataRecord* dr = value.AsObject<UDataRecord*>();
if (dr == 0) {
// If not, serialize it
dr = new UDataRecord();
USerializable* obj = value.AsObject<USerializable*>();
obj->Serialize(dr);
}
return(__UDataRecordToJSONObject(dr));
}
UASSERT(false, "Unrecognized type.");
return(json::object());
}Again, this function is pretty much the opposite of our loading function that converts JSON to Variants. You can see here that for 4-component vectors, we store the properties as a, r, g, and b. In this case, if we have nested objects, it will attempt to serialize those into a new data record for storage as well and then store it as a nested JSON object.
Conclusion
We can now load, modify, and re-save JSON data. Revisiting our example from earlier, we could now update user preferences with something like this:
// Load configuration data
UJSONLoader* jsonLoader = new UJSONLoader();
jsonLoader->Open("appsettings.json");
// Load settings object
auto settings = jsonLoader->GetRecords("settings");
settings[0]->Set("windowWidth", 1200);
jsonLoader->SaveRecords("settings", settings);This will update our JSON file with the new window width and save it back to disk.
We now have 3 ways to serialize data: into a local SQLite database, a local JSON file, or a local or remote MySQL database. These will serve us well for our use cases, whether that's easy to read and modify configuration data, or collections of objects in a SQL database. The last format we'll look at is binary serialization of DataRecords, which will be necessary to transmit them across a network.