This month’s main attraction is the new Modules API that will serve as the basis for how Snowflake is developed and packaged in the future. There has been some UI progress as well; but if there is one thing I’ve learned this month, is that frontend developement is very difficult. Nevertheless, there has been some good progress in some of the core elements like the home view. Finally, there have been some improvements in filename parsing to make it more accurate than the previous regex-based method.

If you know how to inject an overlay CEF on top of an arbitrary OpenGL/DirectX surface like the Steam Overlay, please let me know!

UI Progress

While the UI is not yet usable, I did get a lot of parts of the Material Design theme done in isolation these few weeks.

This will be the “Home”, or the main view of the Material Design theme; the big red header section is meant to display some information about the currently selected platform. This view might have been one of the most difficult ones to write; a huge challenge was figuring out how to performantly display over a hundred games. Fortunately, react-virtualized helped a bunch, but there is still some jankiness when there are over 300 games; I couldn’t figure out how to optimize this yet so I’m hoping React Fiber will smooth this out once it’s ready.

This is how one would select a platform to view games for; instead of having a sidebar like a lot of desktop-oriented frontends, I decided to go with a separate menu instead. Perhaps this might be too mobile-like but for now this is how it’ll stay for this theme at least.

This is very much a work in progress; the game view is where users can set launch options for their emulator, view and edit details about the game, and change their controller configuration. The UI toolkit I’m using, Material-UI v1, doesn’t support all the components I need to make this work properly, so I will need to work around that by using their older version. Please ignore the fact that there is a Super Mario Bros. image on top of a Super Mario World background..

Currently, React lets me develop each part in isolation which makes development a lot easier when I don’t have to worry about side effects. Eventually I will have to hook everything up and decide how one view links to another, this is called routing. I’m still thinking very deeply on how to do this properly, and how to store the currently selected game and platform.

Note that most of the UI work is currently done in the remoting branch, and has yet to be merged to master.

Module Loader and Plugin API Rewrite (PR #249) — More reliable plugin system.

Since the port to .NET Core, plugin loading has been extremely fragile; it was basically held together with duct tape and gum to port it as soon as possible so that work could be resumed quickly under the restrictions of the new SDK. What used to work under the Windows-only .NET Framework no longer did reliably when cross-platform was taken into consideration, and while Snowflake was from the start built with cross-platform support as an eventuality, the plugin loading wasn’t something I was especially concerned with.

Not only have I completely rewritten the plugin loading backend, there have been some changes made to the way plugins have to be packaged to make things easier to manage and install. Before, plugins and all their dependencies were simply thrown into the plugins folder. Proper load order was basically hoping for the best, which isn’t quite reliable especially if you have pluginsn depending on plugins. Now, plugins are packaged as modules, which are loaded individually and have separate folders to put their dependencies in. Each plugin is loaded in isolation to other plugins, so dependency load order is always correct.

Another change made to the plugin system are how they are initialized. Before, plugins were registered with the plugin manager through a container that had access to the entire instance of Snowflake. There was no way to tell what plugin had access to what services, and no way to tell if a plugin can even be loaded properly when a service doesn’t exist. The new method requires all plugin containers to list which services they require, so that the plugin loader when to load a plugin, when all of its required services are available. If the services are never available, the plugin will simply never be loaded.

With the new changes in plugin loading, a lot of service implementations have been moved outside of the core framework. For example, the plugin manager implementation is now implemented as its own plugin (technically a service module). When launching Snowflake, only a few essential APIs (eg. a service that gives the application data path, a service that allows more services to be registered, etc.) are provided initially, the rest are loaded externally as their own separate service modules. This is all towards the goal of making Snowflake as moddable as possible.

For most people looking to extend Snowflake, the so-called Plugin API should be sufficient, which relies on the plugin manager to provide it certain things like loading metadata from a resources folder, a configuration database to store options, etc. The lower level Service API allows modules to register services that other plugins can use and hook into; this is how a bunch of APIs that used to be explicitly instantiated are now implemented, like the plugin manager. By keeping implementation details like this out of the core framework API, this reduces the size and complexity of a plugin greatly. This is very similar to a programming concept called “dependency injection”, which is used to reduce complexity by making what a certain object requires very explicit.

Tokenizing Filename Parser (PR #250) — ROM filename parsing improvements

While Snowflake also uses advanced techniques like looking into the ROM header to get information to scrape from, oftentimes the ROM’s filename also provides useful information. Snowflake used to glean the game’s title, year, and country using regular expressions, but this wasn’t very accurate.

The new method splits up the filename into “tokens”, or a word with a specific meaning. Since ROM filenames often have flags in brackets or parenthesis, these flags are extracted individually, and classified by meaning as if the ROM was simultaenously named with the GoodTools convention, the No-Intro convention, and the TOSEC convention. Snowflake then picks the convention with the best result and uses that to determine the title, the year, country, languages, etc. The old method would fail on a lot of examples, but this new method proves accurate for quite a few test cases.

Looking forward to the next report…

For the month of July, most of the work will be in the user interface, and hopefully getting it up to a usable state so that games could at least be added to the frontend, if not yet launchable. I also hope to work on the packaging side of things to make Snowflake easier to install and update. I’m also looking into launching a Patreon very soon, please let me know what you think, no matter if you plan to donate or not. Some income would help a lot with hosting fees and infrastructure; currently I’m using a lot of free services for nightly builds and such, but I have some big plans that I can’t possibly implement without any money.

I’m also thinking about a complete redesign of the website and logo; a lot of the visual identity stuff is getting dated seeing as how they were designed two years ago, and I like to think my design skills have improved since then. This will probably not be done in July though, as getting the UI functional takes immediate priority.

When I first started writing Snowflake, one of the things I was concerned with was how to refer internally to different game consoles. This wasn’t an issue exclusive to Snowflake; databases like TheGamesDB and OpenVGDB all have different identifiers for different consoles.

Stone originally started out as a more normalised way to identify individual consoles. I wasn’t happy with numeric, “black-box” identifiers (the NES is platform #21 according to the GiantBomb API!), or human readable, but hard to parse strings like how TheGamesDB likes to use, Nintendo Entertainment System (NES). OpenVGDB does have “OEID”s like openemu.system.psx, but they aren’t use anywhere else in the database, preferring to use the numeric system ID internally instead (38 for the Playstation).

If I were to use a new set of identifiers, they would have to be

1. Human readable
2. Short and concise
3. Obvious

With #3 being the most important of all. With those criteria in mind, the very first iteration of “Stone”, was simply a list of identifiers for all consoles that were once, or are currently on the market. Instead of 21, the NES would be known as NINTENDO_NES, the Sega Genesis as SEGA_GEN, so on and so forth. It was obvious, unique, and with the lack of spaces and other frivolous punctuation, minimised the amount of possible bugs just for parsing identifiers.

That was fine and all, and identifiers are one of the most important parts of Stone as a whole. But those of you that have been following Snowflake’s development know of a peculiar feature, heuristic scraping. Snowflake is able to, by examining the file signatures of different ROMs and ISO files, reasonably determine what game console (I’m going to refer to a ’game console’ as a ’platform’, to use Stone terminology from here onwards) that file is a game of.

Again, originally this was a very simple affair. Simply check the file, and tell the scraper the identifier of the platform. But as the scraping pipeline got increasingly complex, it began to make sense to sort out each file not just by platform, but also by file type.

Some files, like .nes files can be hashed quickly and easily. Other files like Wii .wbfs files probably would take a while to hash, but conveniently have a unique ID in their header. Snowflake’s record based database also makes this a requirement; since there could be multiple files associated with one game, there has to be a way to tell apart image files like cover art, and the actual game binaries, and files like cuesheets and GDIs. Snowflake also needed a way to specify if a certain file was in a zip archive, as many emulators support running ROMs inside ZIP files.

Thankfully, there already exists a mechanism to identify file types called a “Media Type”, or a “Content Type”, also known as a “mimetype”. For every file type I could find for every console that could even relate to running a game, a unique content type was given to the file type. For example, your humble .nes file can be referred to as the content type application/vnd.stone-romfile.nintendo.nes-ines. A bit lengthy, but descriptive, and no one really minds long content types.

These content types are probably one of Stone’s biggest advantages when it comes to emulation and emulator frontends. Having a unique way to refer to a certain file type makes certain things very easy, especially if the frontend or emulator is a multi-system emulator. .bin by itself is hardly descriptive of anything, but application/vnd.stone-romfile.sony.psx-discimage makes it obvious that it is a Playstation 1 disc image. I don’t think anyone has attempted to give standardised content types to ROM and ISO files (especially since there are so many varieties of files that are called ‘ISO’ files), and Stone assigns them in a standard, consistent, and descriptive manner as well.

On the issue of emulation, Stone also describes known hashes and common file names for the BIOS files required for many platforms. This allows possible frontends to check for the presence and integrity of such BIOS files to be able to properly notify the user if an emulator does not work. Much of this data was assembled from sources across the internet, and Stone brings it all together into a neat package.

Stone also has support for arbitrary metadata for platforms. Many frontends put metadata in some closed database, if they even bother at all. Stone can act as a source of metadata for platforms, including information such as release date, company, abbreviations, and of course, a human readable Friendly Name. Along with everything previously mentioned, this metadata is specified in a platform definition that forms the core of Stone. A definition is simply a parsable description of a platform including all its identifier, file types, BIOS files, and metadata. I encourage everyone to continue to add to this metadata for as many platforms as you can by checking out the definitions, and making edits on GitHub!

Besides content types and BIOS files, Stone also describes the layout of various platforms controllers, in controller layout definitions. While the platforms definitions were created in hopes of being useful to not only Snowflake, but other frontends and databases, the controller layout definitions may admittedly be less useful. They were created originally for Snowflake’s on-the-fly input configuration generation feature, but since this data had not been compiled online anywhere else in any useful form, I’ve decided to include this data in hopes that it could be useful to someone else for some project.

The controller layout definition specifies the layout of inputs on a controller for a certain platform. Probably the harder-to-understand section of Stone, every possible input on a conventional controller is given a name and semantic meaning, called an element. Elements that exist on a given controller are listed out, for example, the NES controller has a ButtonA, a ButtonB, ButtonStart, and ButtonSelect, as well as the 4 D-pad directions DirectionalN, DirectionalE, DirectionalS, and DirectionalW. The element is then given a type, which specifies that the ButtonA, for example, is really a Button. A label is also given to match what the button would be called originally, for example, ButtonA is labeled “Cross”, for human readability.

The theory behind these layers of abstraction is that any possible button on the hardware controller, meaning the input device the user is holding, can map to any other element with the type of Button. For example, the user could map the A button on their XBOX 360 controller to a B button on their emulated NES controller. While this is mostly an implementation detail on Snowflake’s end, it has the benefit of describing exactly the layout of many controllers in a computer-readable way. While the information itself is widespread, I believe Stone is the first time it has been made available in an easy to parse manner.

I’m sure that Stone fills an extremely small niche, and many of you reading this probably kept in mind a certain black and white webcomic about so-called “standards”. But while Stone has certainly been useful for me, I’m publishing this under the MIT license in hopes it will be useful for others as well. Contributions are always welcome, be it adding a missing platform (Stone covers a lot, but hardly all), or more metadata. Read through the docs and explore the definitions at the Stone website. I’m aiming for Stone to be the OpenVGDB, not for games, for consoles and controllers, a source of data for all to use in their own projects.