chromy / ink-proof

Testing for Ink compilers and runtimes

ink-proof

Conformance testing for Ink compilers and runtimes

Ink is an open-source narrative scripting language created by Inkle. ink-proof is a tool for conformance testing Ink compilers and runtimes.

Users author interactive stories as .ink files. Inkle provide a compiler (inklecate) which converts these .ink files to a json based format which is then interpreted by a C# runtime. There are other compiler and runtime implementations at various levels of completeness. For example inkjs, godot-ink, inkcpp.

ink-proof consists of a number of .ink and .json test cases. Each test case contains a .ink or .json source file, an input file, and an expected output or "transcript" file. ink-proof runs each test case against every runtime and compiler and compares the actual output to the expected output. Results are generated as a webpage for easy viewing.

The latest public run of ink-proof is available at https://chromy.github.io/ink-proof however you can also run the tool offline as follows:

git clone https://github.com/chromy/ink-proof.git
cd ink-proof
python3 install_deps.py
python3 proof.py --serve
# Now navigate to http://localhost:8000

Test cases

ink-proof has two kinds of test cases: ".ink" test cases and ".json" test cases.

Ink test cases live under the ink directory. They consist of a story.ink file (for example ink/I002/story.ink), an input.txt file, an expected output file named transcript.txt, and metadata in a file named metadata.json.

Json test cases live under the bytecode directory. They consist of a .json file (for example bytecode/B005/story.json), an input.txt file, an expected output file named transcript.txt, and metadata in a file named metadata.json.

Compiler and runtime drivers

ink-proof can test both Ink compilers and Ink runtimes. To do this it uses small shim programs which wrap each compiler or runtime to provide a consistant interface. These shim programs live in the driver directory.

Compiler drivers are named with the suffix _compiler_driver, for example inklecate_v0.9.0_compiler_driver. During testing they are invoked as follows:

your_fancy_compiler_driver -o output.json input.ink

Runtime drivers are named with the suffix _runtime_driver, for example inkjs_v1.9.0_runtime_driver. During testing they are invoked (kind of) as follows:

cat input.txt | your_fancy_runtime_driver story.json >actual_output.txt

In other words they get passed the compiled json story as the only command line argument then they get fed each choice as input and are expected to produce the output on stdout.

This is how inklecate works currently so for other runtimes/compilers this means writing a wrapper to make them work like inklecate.

ink-proof's Issues

NotImplementedError exception on Windows

When running the latest code on master (efdfb70), running proof.py on Windows results in the following error.

Task exception was never retrieved
future: <Task finished coro=<Job.run() done, defined at proof.py:423> exception=NotImplementedError()>
Traceback (most recent call last):
  File "proof.py", line 431, in run
    await self.do_work()
  File "proof.py", line 445, in do_work
    process = await asyncio.create_subprocess_exec(self.command[0], *self.command[1:], stdout=fout, stderr=ferr, stdin=fin)
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\asyncio\subprocess.py", line 217, in create_subprocess_exec
    stderr=stderr, **kwds)
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\asyncio\base_events.py", line 1544, in subprocess_exec
    bufsize, **kwargs)
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\asyncio\base_events.py", line 462, in _make_subprocess_transport
    raise NotImplementedError
NotImplementedError
Task exception was never retrieved

I did a little bit of Googling. Looks like some features of subprocess are not supported on Windows?

https://stackoverflow.com/questions/44633458/why-am-i-getting-notimplementederror-with-async-and-await-on-windows
tiangolo/fastapi#964

It appears there might be some way to use a different back-end implementation or something that will work on Windows, but I don't know enough about python to really say.

ink implementation with incompatible bytecode

Hi, thank you for this project,
I'm working on an ink implementation in lua. So far it's very minimal. I find the set of tests / examples in this project very useful.

I have managed to write a driver for my compiler and runtime.

The problem is I use my own datastructure instead of the JSON file so my compiler only works with my runtime and vice versa.

Would you be interested in a PR anyway?
It will fail in the matrix with the other implementations but for me it's useful to test my implementation

Test for `LIST_ALL` on a list with multiple origins

While Blade Ink Rs doesn't currently test itself using Ink Proof, I found an inconsistency between it and other runtimes. bladecoder/blade-ink-rs#4

LIST a = A
LIST b = B
{LIST_ALL(A + B)}

{"inkVersion":21,"root":[["ev",{"VAR?":"A"},{"VAR?":"B"},"+","LIST_ALL","out","/ev","\n",["done",{"#f":5,"#n":"g-0"}],null],"done",{"global decl":["ev",{"list":{},"origins":["a"]},{"VAR=":"a"},{"list":{},"origins":["b"]},{"VAR=":"b"},"/ev","end",null],"#f":1}],"listDefs":{"a":{"A":1},"b":{"B":1}}}

This tests whether LIST_ALL() on a list containing items from two different origins, returns the sum of all of those origins, or only one of them. I expect A, B as the output, rather than just A or just B.

Linking to Runtime Specification

Hey there!

First off, thank you so much for this project! I wanted to let you know that we're working on an actual specification for the Ink 1.0.0 runtime, and linked to ink-proof as the community-standard way of testing! tcannonfodder/ink#8

Let us know if there's anything we can do to help out!