:Firth pre-alpha 1– a Forth-like language for DSL creation.

So what is :Firth? It’s a programming language. It’s a compiler-interpreter implemented (currently) in Lua. It’s performant, portable, embeddable, stack-based, extensible, and self-modifying. It’s free and open source. It’s more of a compiler construction toolkit for defining DSLs, than a programming language as we normally think about them.

What :Firth definitely is not is FORTH. I describe it as Forth-like, because it shares many idioms and mechanisms with Forth. ANS Forth is definitely the #1 influence on the design of the language and implementation thereof. But, I’ve diverged from that starting place, sometimes dramatically. These differences emerge when I just want to get something working as fast as possible, when I decide to do things differently for technical reasons or personal preference, or when I just can’t be chuffed to look up how Forth does something (or DPANS is too opaque for a relative outsider).

Why :Firth?

There are plenty of Forths and Forth-likes out there. There are plenty of embeddable scripting languages designed for DSL creation. Why create :Firth? Well, the reasons are many, and the chain of events that lead to its creation is somewhat circuitous.

In brief, a few years ago I had an idea for a scripting language that I thought would be awesome for game development. I sat on it for a while, adding and revising design notes from time to time. Then, a couple of years ago, I wanted my language for a game project I was working on. In the end, decided it would take too much time to roll my own compiler, VM, and runtime from scratch for this project, and decided Lua would be the closest match for what I wanted. Since then, I’ve become rather enamored of Lua.

Jump forward a year and a half, and I was thinking about doing some heavy language meta-programming to implement a BNF-like grammar specification language; a grammar defined in this language would itself actually be the source code for an executable parser. This kind of thing would make it relatively easy to implement my original scripting idea, or most any other language I or anyone else might want.

Lua was the obvious initial choice for underlying platform, with its meta features, small size, and fast performance. It quickly became obvious that it didn’t have quite the amount of syntactic flexibility that I wanted. I briefly considered Ruby, which is a little more powerful in that regard, but Ruby is relatively large, slow, and not optimized for embedding.

This is when :Firth was conceived. Building it as a layer on top of Lua would give me all the benefits of the Lua type system, the underlying virtual machine, and even garbage collection for free. It would also give me access to an optimizing JIT, when running it in LuaJIT. By generating subroutine-threaded Lua code with some inline operations, I could have a fast, convenient, dynamic language.

But mostly, I thought it would be fun!

So How Do I Use This Thing?

To start with, head on over to github and check out the repo. Feel free to clone and/or fork it. The easiest way to start playing with it is to cd into the repo root directory, and execute the command:

$ luajit firth.lua

This will drop you on the REPL prompt, where you can execute commands interactively. Alternatively, you can enter code directly on the command line; :Firth will evaluate each of its arguments, and print the contents of the stack when it’s done. This lets you use firth.lua as a sort of RPN calculator, or test tiny scripts directly from your shell:

$ luajit firth.lua '40 52 * 60 * 60 *'
[7488000] $ luajit firth.lua 22 7 /
[3.1428571428571] $

There is currently no install script or convenience wrapper for executing it directly from the command line, though those will be simple enough to create when the time comes. But, maybe what you’re more interested in is how you go about actually programming something in :Firth. It is definitely a little opaque if you’ve never used Forth or perhaps a high-end HP calculator before. When you fire it up, you get a version / copyright banner, the simple prompt ok, and nothing else. I admit this could be rather mystifying, if you don’t have any context, and there’s no online help system just yet.

The core compiler and the Lua modules it uses are pretty cleanly coded, and have lots of comments (which can be used to generate Doxygen documentation). But, this doesn’t really give a good feel for how to use the language it implements.

Probably a better place to look is in the file firth/core.firth, which implements the standard core library, or vocabulary. It demonstrates how the mechanisms and facilities that you will commonly use are built up from more primitive operations. Even comments are implemented this way. You should definitely familiarize yourself with this file.

But, perhaps actually seeing the code in action, doing something verging on useful will be more enlightening.

Code Examples

The examples/ directory will contain a growing number of code samples. Currently, it has one, iniloader.firth, plus a sample input file, test.ini. Let’s look at the input file.

{
    FOO = BAR
    BAZ = 123
    BLERG = 127.0.0.1:8080
}

This should be pretty familiar to anyone who’s ever used JSON, Python, or Lua, though it’s not compatible with any of those. Say we want to read this file into memory, such that variables FOO, BAZ, and BLERG are set according to the declarations in the file. We could do this by scanning for the opening brace, switching to parsing out “key = value” pairs, throw away the ‘=’ character, and create and assign a variable for each parsed-out declaration. Or we could do it the :Firth way! Below is the entire parser for our test INI file.

false val: INIDONE
: = ( var -- ) ( TS: val ) word swap ! ;
: inidef ( -- ) ( TS: name = val )
    word dup char } equal? if
        true ` INIDONE ! drop
    else
        variable word call
    end ;
: { ( -- ) true while inidef INIDONE not end ;
: inifile: ( -- ) ( TS: path )
    false ` INIDONE !
    word loadfile ;

Let’s go over it in smaller pieces.

false val: INIDONE

First we create a special kind of variable called a “value”; it’s kinda like a constant that we can reassign if really necessary. INIDONE will act as a flag to signal when we’re done processing input, later. It is initialized as false.

: = ( var -- ) ( TS: val ) word swap ! ;

Here we are defining the ‘=’ operator. This is called a colon definition. It creates a function (called a word) named ‘=’. The comments (between pairs of parentheses) after the name indicate that it expects a variable ref on the top of the stack, and doesn’t leave anything behind; and, it pulls a string value from the token stream. word is the parsing function that collects the next token. Then we swap the top two items on the stack, because ! (store value into variable) expects arguments in the order var val !. The end result is that the variable named to the left of the = is set to the value to the right.

: inidef ( -- ) ( TS: name = val ) ...

Next is inidef. From the comments, we see it won’t take or leave anything on the stack. It will remove three tokens from the input stream, if everything goes well.

word dup char } equal? if

Here, we parse out the first word from the token stream, which corresponds to name in the comment. We dup a copy on top of the stack and push the character } as a string, before testing for their equality; this consumes the top two items and leaves a boolean value in their place. At this point the stack will have the original copy of the name, with a boolean on top of that. The if consumes the boolean to decide which branch to take.

true ` INIDONE ! drop

If the boolean was true—the parsed name is equal to “}”—this branch is taken. It sets INIDONE to true (i.e. val var !) and drops the dangling string value from the top of the stack. This signals to break the processing loop, later. Note the use of the ` (back-tick) operator, which pushes a reference to the following word’s definition. This is necessary because we are using a value rather than a standard variable.

variable word call

The else clause, taken when the name string isn’t equal to “}”, creates a variable with that name. It then parses the next word from the input stream, looks it up in the dictionary, and calls that function. This finishes up the definition.

: { ( -- ) true while inidef INIDONE not end ;

Here, we actually define { as the name of an executable word. We declare a while loop which we want to execute at least once. The body of the loop calls inidef to try to parse out a single definition, and then checks that INIDONE is not true, as the loop condition. When the then branch of inidef sets INIDONE to true, this is where that is caught, and the loop aborted.

And, finally…

: inifile: ( -- ) ( TS: path )
    false ` INIDONE !
    word loadfile ;

We define inifile: which has no external stack effects of its own, and parses a file path from the input stream. First, we initialize INIDONE to false, to make sure the loop will run until it finds “}”. Then, we parse an input token, which will be used as a path argument by loadfile, which then reads and parses the specified file. Note that a limitation of this implementation is that it won’t work for files with whitespace characters anywhere in the path. We could fix this by changing inifile: to expect a path string on top of the stack, or by making it parse the name delimited by e.g. double quotes instead of whitespace. Or, we could simply make a “no whitespace in file path” rule, in the spec for this feature.

It’s also worth pointing out that what loadfile actually does is load, parse, interpret, and compile the contents of the file, as :Firth code. We just redefined the language and the compiler so our INI-style format is valid :Firth. This means we can embed arbitrary code before and/or after the brace-delimited data definition, including comments. It would be pretty simple to refactor the code to load the declared key/value pairs into a hash object instead of global variables, and leave that on top of the stack when it finished. Such a change would make it a little more like JSON, Python, or Lua data definitions, as one might expect by looking at the file format; this is probably a desirable next step. It would also be pretty simple to refactor and extend the code to allow for hierarchal data…

{
    OpenGLVer = ES2.0
    FSAA = true
    Window ={
        Width = 400
        Height = 800
    }
    ShaderPath = res/shaders
}

…or conditionally assign a value only if the given name isn’t already bound to a value…

Firth code:

inifile: data/userpref.ini // user-defined preferences
inifile: res/defaults.ini  // default fallback values

res/defaults.ini:

...
UseCustomFonts ?= true
CustomFontName ?= DejaVu-Sans-Mono
...

But, we won’t get into all that right now. This is only the beginning, and there’s no way to cover everything that’s possible.

Project State and Future Direction

As the title of this article suggests, this is pre-release code; a lot of stuff is missing, and many of the features that do exist will definitely change before we hit 1.0. I might call the current state just a little beyond proof-of-concept, or a prototype. It is actually useful, stable, and fast for many potential uses.

If you’re a seasoned Lua hacker, some of the code may look a little strange, especially code it generates. I created an experimental optimization branch after corresponding with Mike Pall, author of LuaJIT. I don’t know that he’d especially endorse any of the code I wrote as a result, but I definitely did see a speedup of a few orders of magnitude in some cases, so I folded it all into master.

Speaking of generating code, it’s all very ad hoc. I’ve been corresponding with Francesco Abbate, author of the LuaJIT Language Toolkit. I am strongly considering switching to generating an AST to feed into his code generator. This will make the generation process more formal and structured, which can only be a good thing. But, it will also open up future opportunities for optimization, when I have a proper representation of the code that I can walk and analyze.

Eventually, I’d like to have the code factored in such a way that the bare minimum amount of code is written in Lua. The rest of the system, written in :Firth, will bootstrap itself using these primitives. (There’s already a fair bit of this going on; check out how “:”, “;”, and “//” are defined, near the top of firth/core.firth).

Ionoclast Laboratories™

Devlogs and Ramblings from our Mad Scientists