Game Development - Tokenizer

Recently, I've started studying a bit about a very specific and somewhat nightmarish method of programming: Tokenizers

What is a tokenizer?


A Tokenizer is a piece of code that serches through a text document or data stream for different pre-set "tokens" and modifies the program as it reads. If it does not recieve a specific token, that part of the object is then set to a value standing for unused(In my case, 0).

So, Why bother?


Here's the interesting thing about tokenizers: They make your game infinitly moddable. Take dwarf fortress for example; everything in the game, from the material the earth is made up out of, to the structure and types of creatures, to even what foods and goods can be produced, are built from a token based system. This means that the end user can add and tinker with the game as they please, adding much more replayability to what may stagnate quickly.

I also want to build a tokenizer system for ease of content creation. I havent decided yet to waht extent the end user will be able to influence the world through token modification(There will be some things, such as overland geography, that will be locked out.)

Yeah, that's all well and good. Aren't you making a browser game though?


This is what really excites me about a tokenizer system. With the correct group of data tags, I can just import all the game data in from external, easy to mod sources. The framework is the only piece of the puzzle that will be time consuming, and once it's up and proven then the basic frame can be used in any number of projects.

It basically allows me to go from a full hard coded source to a engine, which is really exciting for reusability.

So... how would it work?


The basic idea is that you have a script(Probably in a flash file) that is retrieving a constant data stream from a server location. All of that information is just placed in a properly formated XML file, for example. The script goes through each character of the stream, looking for start and stop characters(eg. [ and ]).

Once the script has found a start character, it stores all the values between it and the stop character in a temporary string, which is then compared to all of the tolkens in the program. Once it finds the tolken it's looking for, it empties all of the information between the tags into a data appropriate container. It does this with all the information on the file in the preloading sequence.

Occasionally, the program will also update these tolkens and refresh the values stored in the engine, lets say once a hour.

(I have some more to add, but I'll do it later. Please pester me if I forget.)