Loading Files and Resources
Prerequisites
- The UObject Base Class
Article
All game engines need to read from and write to files - models, textures, configuration, save data, and more. Ideally, you want to load as much of your data up-front as possible, or use multi-threading to stream data in as needed (especially for open worlds).
Many of these files will be resources: textures, audio files, scripts, 3D models, animation, and so on. Resources are a special case, since we only want to load those files once, even if they are used in multiple instances (eg. many objects sharing the same script). In addition, resource files usually have a post-processing step - you'll load them into a buffer or a system or convert them to a format that can be used real-time.
We'll break file and resource management down into three classes:
- File - largely a wrapper around the POSIX file functions.
- Resource - a special type of file that should only be loaded once and has a callback for post-processing
- ResourceSystem - a system (in our ECS model) that will be responsible for managing Resource objects
Let's start out with the File class:
class UFile : public UObject {
public:
UFile();
~UFile();
/**
* Open file
*/
void Open(std::string filename, std::string mode);
/**
* Close file
*/
void Close();
/**
* Read from file
*/
int Read(void* bytes, int size);
/**
* Write to file
*/
int Write(unsigned char* bytes, int size);
/**
* Get/set current position
*/
void Position(int position);
int Position();
/**
* Test whether a file exists or not
*/
static bool Exists(std::string filename);
/**
* Load all file data
*/
void Load();
/**
* Get properties
*/
unsigned int Size() { return m_size; }
std::string Filename() { return m_filename; }
std::string Path() { return m_path; }
std::string Extension() { return m_extension; }
protected:
// Internal pointer
FILE* m_fp;
// File data
unsigned char* m_data;
// File size
unsigned int m_size;
// File info
std::string m_filename;
std::string m_path;
std::string m_extension;
};At the top you'll see 4 primary functions you would expect from every file implementation: Open, Close, Read, and Write. These largely mirror the POSIX versions of the functions fopen, fclose, fread and fwrite respectively:
void UFile::Open(std::string filename, std::string mode) {
m_fp = fopen(filename.c_str(), mode.c_str());
UASSERT(m_fp != 0, "Unable to open file");
// Get file size
fseek(m_fp, 0, SEEK_END);
m_size = (unsigned int)ftell(m_fp);
fseek(m_fp, 0, SEEK_SET);
// Parse filename into parts
size_t offset = filename.find_last_of("\\/");
m_filename = filename.substr(offset + 1);
m_path = filename.substr(0, offset);
offset = filename.find_last_of(".");
m_extension = filename.substr(offset + 1);
}
void UFile::Close() {
fclose(m_fp);
}
int UFile::Read(void* bytes, int size) {
UASSERT(m_fp != 0, "File not open.");
if (m_data) {
// Read up to max bytes
int r = std::max((int)m_size - m_offset, size);
// Get the data
memcpy(bytes, m_data + m_offset, r);
// Increment position
Position(m_offset + r);
// Return size of data read
return(r);
}
int r = fread(bytes, 1, size, m_fp);
m_offset += r;
return(r);
}
int UFile::Write(unsigned char* bytes, int size) {
UASSERT(m_fp != 0, "File not open.");
return(fwrite(bytes, 1, size, m_fp));
}Our Open function opens the file pointer, but also reads the file size and parses the filename into parts: the filename, path, and file extension. This presumes that a full file path is provided to the function, which we'll see in our ResourceSystem shortly. The Close and Write functions are fairly straightforward. The Read function will attempt to read from memory first (if the file has been loaded), otherwise it will read the bytes from the file pointer.
Next we have the Position functions - one reports the current position in the file or data and the other sets it - these are also largely wrappers around fseek and ftell:
void UFile::Position(int position) {
UASSERT(m_fp != 0, "File not open.");
fseek(m_fp, position, SEEK_SET);
m_offset = position;
}
int UFile::Position() {
UASSERT(m_fp != 0, "File not open.");
return(ftell(m_fp));
}To make all of this possible, we have a Load() function that simply reads all of the data in the file into memory:
void UFile::Load() {
UASSERT(m_fp != 0, "File not open.");
if (m_data) {
return;
}
m_data = (unsigned char*)malloc(m_size);
fread(m_data, 1, m_size, m_fp);
fseek(m_fp, 0, SEEK_SET);
m_offset = 0;
}In this case, if we're already loaded data, we simply return. You could make this more robust in the future and add some checking to see if the file has been updated and needs to be reloaded, but for the base case this will suffice. Finally, we reset the pointer back to the beginning of the file after reading everything so that it matches our offset.
Last but not least, we have a static Exists() function that will tell us if a file already exists given a file path:
bool UFile::Exists(std::string filename) {
FILE* fp = fopen(filename.c_str(), "r");
if (fp) {
fclose(fp);
return(true);
}
return(false);
}There are many ways to check for a file's existence - you can use stat(), fopen(), access(), or use something OS-specific. In our case, the difference between them will be negligible - we're not going to be checking thousands of files, and hopefully we're only doing this during load time.
Resources
Now that we have a class to open, read, and write files, we can extend that into a Resource class that let's us retrieve data and process the data once it's loaded.
We'll also create this as a base class for other classes: we'll have Texture2D and Mesh classes in our render system, an AudioClip class in our audio system, and a Script class in our scripting system (for example). All of those will be Resource objects that will get loaded and controlled by our ResourceSystem.
Let's take a look at our Resource class:
#include <Core/UFile.h>
class UResourceSystem;
class UResource : public UFile {
public:
UResource() = default;
virtual ~UResource() = default;
/**
* Get all bytes
*/
unsigned char* Bytes();
/**
* Get contents as string
*/
std::string ToString();
protected:
friend class UResourceSystem;
/**
* Callback for loaded resource data
*/
virtual void LoadFromFile(std::string filename, void* bytes, int size) { }
};The Resource class extends the File class to provide 3 additional functions: Bytes() and ToString() that return the file contents as binary or string data, and a LoadFromFile() callback that passes all of that data in to an override function that will (optionally) do the processing.
Let's look at the Bytes() and ToString() functions that load in and return all of the data:
unsigned char* UResource::Bytes() {
// Check filesize
size_t filesize = this->Size();
// Load file as binary
unsigned char* bytes = (unsigned char*)malloc(filesize);
// Reset to zero and read all bytes
this->Position(0);
this->Read(bytes, filesize);
return(bytes);
}
std::string UResource::ToString() {
// Check filesize
size_t filesize = this->Size();
// Load file as string
std::string bytes;
bytes.resize(filesize + 1);
// Reset to zero and read all bytes
this->Position(0);
this->Read((void*)bytes.data(), filesize);
bytes[filesize] = '\0';
return(bytes);
}Largely these functions do the same two things: get the file size, allocate a buffer, and read the data into it.
That's it! Other classes will override the LoadFromFile callback function, so we won't show that here. Let's take a look at the ResourceSystem class to finish off this section.
ResourceSystem
To load resources from anywhere in our engine, game, or scripts, we'll use a central system to ensure good memory and file management - the ResourceSystem.
The ResourceSystem will ensure only one copy of each file is loaded, even if we call the load function multiple times. It will also help us define search paths - since most linked or embedded resources won't have the right file path, we'll often only have a filename to go find. For example, when loading a model of a castle, you might have a linked castle_texture.png file to use as a diffuse or albedo map, and a castle_normals.png to use as a normal map. We'll need to locate those files in our game's structure, and the ResourceSystem will help us search for those files.
Since the ResourceSystem is responsible for and owns the Resource objects, it will also be responsible for creating them. That makes using our ResourceSystem as easy as calling a single function:
UMesh* castleMesh = (UMesh*)resourceSystem->Load("castle.fbx", "rb", "Mesh");We'll dive more deeply into this code snippet in a moment, but notice how we're specifying a resource type as the last parameter (in this case a "Mesh"). The type will ensure that the ResourceSystem creates this file as a UMesh. It could also be a Texture2D, an AudioClip, a Script, or other resource types. Creating the right class type will also ensure the right LoadFromFile override method is called.
That means that our ResourceSystem will need a way to map a string to a type of class to create new objects. Because all of our objects will inherit from UResource, we can accomplish this with a nifty little bit of C++ templating.
Let's take a look at our ResourceSystem class:
class UResourceSystem : public USystem {
public:
UResourceSystem() = default;
~UResourceSystem() = default;
/**
* Load a resource
*/
UResource* Load(std::string filename, std::string mode, std::string type);
/**
* Register resource type
*/
template<class T>
void RegisterResourceType(std::string name) {
m_types[name] = CreateResource<T>;
}
/**
* Add a resource search path
*/
void AddSearchPath(std::string path);
/**
* Find a resource's full path
*/
std::string FindResourcePath(std::string filename);
// Internal function to call a C++ constructor
typedef UResource* (*ResourceCreateFunc)();
protected:
// Create a new C++ object based on a registered script interface type
template<typename T> static UResource* CreateResource() { return new T; }
protected:
// Loaded resources
std::map<std::string, UResource*> m_resources;
// Registered resoure types
std::map<std::string, ResourceCreateFunc> m_types;
// Search paths
std::vector<std::string> m_paths;
};For now let's focus on a few key functions - RegisterResourceType, CreateCreateFunc, and CreateResource - as well as the m_types map.
ResourceCreateFunc is a function pointer to a function that returns a UObject and takes no parameters. Fortunately for us, we have also defined just such a function: CreateResource. CreateResource will create a templated pseudo-factory for UResource objects - that is, because it is templated, CreateResource<UMesh> is actually a different function than CreateResource<AudioClip>. That means we can store those as function pointers! In this case, we store them in the m_types variable, mapping them from an std::string type to a templated function pointer. RegisterResourceType (also a templated function) is how this mapping gets accomplished:
/**
* Register resource type
*/
template<class T>
void RegisterResourceType(std::string name) {
m_types[name] = CreateResource<T>;
}This function can be called from anywhere in the engine, and can register a class type (in essence a constructor) to a string type name using our internal CreateResource function. For example, for our UMesh class (called later from our RenderSystem):
resourceSystem->RegisterResourceType<UMesh>("Mesh");This store a function pointer to a templated CreateResource call for UMesh objects into a ResourceCreateFunc function pointer and sticks it into m_types for later use with a key of "Mesh". I'm still amazed that this works to this day, and we're going to use this trick later in our metaprogramming area.
One other note: I do not check that the type is not already registered - this is intentional. When we look at supporting multiple platforms, we may have base classes and platform specific classes that may need to be instantiated or registered depending on what is in use. For example, when we look at our rendering system, we'll have a UTexture2D base class, which will store a generic in-memory copy of our texture. However, we'll also have a UGLTexture2D class, which will be used when OpenGL is the selected graphics platform. Our generic RenderSystem may call resourceSystem->RegisterResourceType<UTexture2D>("Texture2D"); first, and our GLRenderSystem may call resourceSystem->RegisterResourceType<UGLTexture2D>("Texture2D"); later and specify a different class type (with a different override of LoadFromFile) that we should use for textures.
These functions and mappings give us a very powerful approach to loading resources by string names and types that mirror the inheritance possible in C++.
Now let's look at the more boring functions of the ResourceSystem, starting with AddSearchPath and FindResourcePath:
void UResourceSystem::AddSearchPath(std::string path) {
m_paths.push_back(path);
}
std::string UResourceSystem::FindResourcePath(std::string filename) {
// Test if this is a full path already
if (UFile::Exists(filename)) {
return(filename);
}
// Go through our search paths to find the file
auto it = m_paths.begin();
for (; it != m_paths.end(); it++) {
if (UFile::Exists((*it) + "/" + filename) == true) {
return((*it) + "/" + filename);
}
}
return(std::string());
}Add search path does exactly as advertised: adds the given path to a list of search paths we can use later. Recall that in our File class, we expected a full path to the file to be passed in (so that it could be parsed). Similarly, we know that we'll often only get a filename (eg. a texture in a 3D model) and it's expected we'll be able to go find it. FindResourcePath first checks if a full path has already been passed in (eg. maybe from a script), and if not it iterates through all of the search paths that have been added so far and uses the UFile::Exists function to check if the path is valid.
Finally, let's put all of this together by looking at the Load function:
UResource* UResourceSystem::Load(std::string filename, std::string mode, std::string type) {
// Make sure we have a full resource path
filename = FindResourcePath(filename);
// Check if this resource is already loaded
auto it = m_resources.find(filename);
if (it != m_resources.end()) return(it->second);
// Find the type
auto tit = m_types.find(type);
UASSERT(tit != m_types.end(), "Unregistered resource type.");
// Create a new object
UResource* obj = tit->second();
obj->Open(filename, mode);
m_resources[filename] = obj;
// Call the callback function to process data
int size = obj->Size();
unsigned char* buffer = (unsigned char*)malloc(size);
obj->Read(buffer, size);
obj->LoadFromFile(filename, buffer, size);
free(buffer);
return(obj);
}The Load function first tries to find a full file path for the resource. With a full path, we can see if the resource is already loaded, and if so, exit early and return the existing resource, solving the issue of our resources being loaded more than once.
If the resource hasn't been loaded yet, we find the constructor / factory by type name in our m_types map, throwing an error if we don't know anything about this type. If we have a valid function pointer, it is used to create a new UResource object, which is opened with its full file path, and added to our existing resource list.
Finally, we initialize a buffer for the full file size, read in all of our data, and pass it off to the (potentially) overridden LoadFromFile callback function to process the data.
Conclusion
We can now open, read from, and write to files. We have created a special type of file, called a Resource, and can load all of its contents as binary (for binary file formats) or as a string (for text formats). Finally, we have a ResourceSystem class, that can be called from anywhere in our engine, game, or scripts, that ensures we only load a resource once and can create and map resources from a filename and a type to a C++ class type and override function to process the data.
Future extensions of this can unload resources, either manually or based on a last access time. I've played around with different approaches to that in the past, but it requires an additional layer of diligence that Resource objects record timestamps or an equivalent marker in all of their functions. I will leave that as an exercise for you, should you come across the need.