catseye / falderal

MIRROR of https://codeberg.org/catseye/Falderal : Markdown-embedded literate testing for programming languages

Falderal

Version 0.14 "Jean Baptiste Point DuSable Lake Shore Drive"

This is the reference distribution of Falderal, a file format for literate test suites. What sets Falderal apart from most other test frameworks is that it recognizes that your tests are for a functionality, not a particular implementation of that functionality.

Falderal is particularly suited for:

• documenting programming languages with prose and examples
• testing multiple implementations of a programming language
• performing Test-Driven Language Design (TDLD)
• being embedded in Markdown documents

Falderal in three words: "Doctests for DSLs".

If you're more interested in running the tools than learning about the format itself, skip down to Implementation.

Motivation

Say you have a programming language. You probably have a document describing the language. That document probably includes examples. Those examples are probably intended to produce some kind of output. That output is probably expected to be a certain, predicted thing, and if it's not that thing, the implementation is probably considered incorrect.

So why not write those examples in a format that can be run and tested?

You could write a bunch of standalone test sources, and store the output you expect from them in a bunch of other files, and write a shell script that runs each program and diffs the output with the expected output. But this is a lot of clutter — finding a particular example might not be so easy. Each test source exists in a void, not necessarily logically grouped with other, similar tests. And any text you write describing a test needs to be in the comment syntax of your programming language (if your programming language supports comments) and is also detached from all the other test descriptions.

You could write unit tests in the unit test framework of your choice, but if your programming language has more than one implementation one day (and you should really consider that possibility) then you might not be able to re-use it so easily for other implementations in other languages.

In a language like Python, you could write doctests, but that also ties your tests to one implementation of your language. There can be awkward quoting issues with how you embed your test sources inside those embedded strings that comprise your doctests, as well.

Or... you could write a Markdown document with beautiful yet precise prose describing your wonderful language, alternating with example code (in the form of embedded Falderal tests) clarifying each of the points you are making; then you could use a Falderal-comprehending tool to run each of these tests against any implementation of your language which exists or will exist in the future.

And you could even write this document before you even start implementing your language; then when it is all clear "on paper", you have a target at which you can aim while writing your language. As you implement more and more of it, more and more tests in your test suite will pass. This is simply the idea behind Test-Driven Development (TDD) applied to language design, which we will call Test-Driven Language Design (TDLD).

Features of the Format

Falderal is just a file format; it does not specify exactly what tools must do with the tests they extract. However, it is generally assumed that most tools will want to, at the very least,

• Run tests from one or more documents.
• Report the results, with some given level of detail.

There is, of course, a reference implementation which does both of these things. It is called falderal and it is written in Python 3.x. More about it can be read in the falderal document.

Each Falderal test is for some abstract functionality, and each functionality may have multiple concrete implementations. Thus the same tests can be run multiple times, once for each implementation of the functionality they test.

Directives in the Falderal document may assign functionalities to tests, and may define implementations for given functionalities. Implementations may be defined outside of any document, as well. Falderal defines one general kind of implementation, implementation by Bourne shell command, but is not inherently restricted from supporting other kinds of implementations.

Inherent Limitations

Being tests, rather than proofs, Falderal tests cannot tell you that your implementation of a language is correct. If one or more tests fail, that's an indication that your implementation is not correct; but even if all tests pass, you have no guarantee that the implementation doesn't do something contrary to the spec in one of the infinite number of cases that you have not enumerated.

There is also no way to test that certain programs represent infinite loops, for reasons that should be obvious.

Contents of this Distribution

This distribution contains:

• doc — contains documents about Falderal. For the specification of the file format, see doc/Falderal Literate Test Format.md. (Note that this specification should not be expected to remain stable through the 0.x version series.) There are other documents in there too.
• bin/falderal — the reference implementation of Falderal. See "Implementation", below, for details.
• script — miscellaneous small tools intended to be used in tests.
• src — source code for the falderal tool.
• tests — a set of tests for Falderal itself. (Note that these are not written in Falderal, as that would just be too confusing.)
• HISTORY.md — changelog for releases of Falderal.
• TODO.md — areas where Falderal and its implementations could be improved.

Implementation

This distribution contains falderal, which is the reference implementation of Falderal, written in Python and sometimes referred to as "py-falderal".

To use it, you can clone this repository and run it as bin/falderal from the directory of your clone, or you can put the bin directory on your executable search path, and run it as falderal anywhere.

Or you can install it using pip:

pip install -e Falderal==0.14

(Depending on your needs, you may wish to establish a virtual environment first. Describing how to do so is outside the scope of this document.)

The implementation is (somewhat) documented in doc/falderal.md.

The implementation has its own test suite (not written in Falderal, as that might be too confusing). It can be run by running ./test.sh.

Development

The git repository for the Falderal distribution can be found on Codeberg at https://codeberg.org/catseye/Falderal.

Official release distfiles are available via the Falderal project page at Cat's Eye Technologies.

Projects using Falderal

• ALPACA
• Castile
• Equipage
• Exanoke
• Flobnar
• Hev
• hatoucan
• Iphigeneia
• Madison
• Nhohnhehr
• Pail
• Pixley
• PL-{GOTO}.NET
• Quylthulg
• Robin
• Samovar
• SixtyPical
• Tamsin
• Velo
• Xoomonk
• Yolk
• ZOWIE

Xoomonk, Madison, Velo, and Exanoke are good examples of how a literate test suite can be useful in both describing a programming language through examples and testing that an implementation of the language does not violate the language specification. They are, in fact, exercises in Test-Driven Language Design (TDLD), where the tests were written as part of designing the language, before any attempt at implementation; the others are more like traditional test suites, written after-the-fact.

falderal's Issues

Unexpected backslash processing during variable expansion

I've been trying to get Falderal working on Windows, but I found my target interpreter wasn't able to open the test source, because the path in the %(test-body-file) parameter was corrupted (the backslashes in the path were being "expanded", e.g. the \t in c:\tmp would be interpreted as a tab).

Now I realise that Windows may not be a platform you want to support, but this looks like it's a general problem for all of the variable expansion, and could affect other platforms. For example, if you were using the %(test-body-text) variable to pass the source code as text, then a backslash there could also be mistakenly processed.

The problem is the use of re.sub for the variable replacement, since that does backslash processing on the replacement string. You can easily fix this just by replacing those calls with the string.replace method instead (there is no need for regex in any of these cases).

In case it helps, this is a patch of the fix I've been using: variable-expansion-patch.txt

Support for input in freestyle format

I'm a big fan of the freestyle format, but unfortunately it doesn't support input yet. I don't know if you already had something in mind for this, but I was thinking one possible syntax could be <=, <==, and <=== which nicely mirrors the output prefixes. So an example test case (in Befunge) could look something like this:

~~~...@

<= ABC
=> 67 66 65 

It turns out this wasn't as difficult to add as I had imagined it would be, but here's my patch FWIW.
freestyle-input-patch.txt

I made a brief attempt to add some test cases as well, but I got half way through adding them to the test-freestyle-format file before I realised it would need a different functionality, and I figured I should probably just leave that to you to sort out. :)

Falderal can't easily test the Befunge reference implementation

I was trying to use Falderal to create some Befunge tests and run them against the Befunge reference implementation, bef. However all the tests failed with an error message something like:

Error: couldn't open 'tmpug9hwq.bf' for input.

It turns out the problem was that Falderal was saving the output in a temporary file with no extension (e.g. something like tmpug9hwq), but when bef is given a filename with no extension it automatically appends .bf and thus fails to find the file.

So I was wondering whether it might be a good idea for Falderal to have an option when setting up the implementation to specify an extension to use for source files, e.g.

Functionality X is implemented by shell command Y with extension Z

But maybe that's an overcomplication for something that isn't really Falderal's problem - it might easier, and more reasonable, to have bef fix the issue. For example you could make bef check if the provided source file exists, and only try append a .bf extension if it can't be found with the original name.

Anyway, if you think it's bef's problem, you can just close this and I can re-raise this issue there (I'd be happy to put together a patch if that'll help).

Or if you think it's too much of an edge case for either app to deal with, I can accept that too - it's not that big a deal to work around, since I'm already using a kind of proxy launcher to redirect the calls to the different implementations I want to test.

Make quoting rules less silly

In a pragma, when you say

...is implemented by shell command "foo"

in "foo", the rules for escaping quotes are awkward. You don't have to backslash-escape double quotes. (Unless of course they are within other double quote inside "foo", but the double-quotes on the outside of "foo" do not count towards the need-to-escape count.)

Single quotes in input aren't escaped

This is not something I use in Windows, but I noticed it while browsing the source code, and it looks to me like it's probably a bug. In the section of the code where the %(test-input-text) parameter is substituted, the content comes from the input variable, but the single quote escaping is performed on the body variable.

I'm referring to this code:

# escape all single quotes in input
body = re.sub(r"'", r"\'", body)
# replace all occurrences in command
command = re.sub(r'\%\(test-input-text\)', input, command)

I'm guessing that second line should be:

input = re.sub(r"'", r"\'", input)