koto-lang / koto

A simple, expressive, embeddable programming language, made with Rust

---

---

Koto is a simple and expressive programming language, usable as an extension language for Rust applications, or as a standalone scripting language.

Info

• About Koto
• Koto Language Guide
• CLI Docs
• Online Playground
• Example Rust application with Koto bindings

Development

The top-level justfile contains some useful commands for working with the repo, for example just checks which runs all available checks and tests.

After installing just, you can run just setup to install additional dependencies for working with the justfile commands.

MSRV

Koto is under active development, and tested against the latest stable release of Rust.

koto's Issues

Arrow key movement broken in the REPL on Windows

When using the REPL on Windows, left and right arrow key movement is broken, with movements moving to unexpected positions, eventually leading to a panic.

Allow indexing on strings

It would be nice if indexing on strings was supported:

s = "@Hello"
assert_eq s[1..], "Hello"

IIRC I didn't add this earlier because I wasn't sure what the indexing behaviour should be (bytes, codepoints, etc), but now it seems clear to me that indexing over grapheme clusters would be the natural thing to do.

s = "héllö"
assert_eq s[4], "ö"

Stack overflow when displaying recursive objects

The following script causes a stack overflow during the print call, due to recursively displaying the contents of l.

l = [1, 2, 3]
l.push l
print "l: $l"

It's acceptable for containers to contain references to the same shared data, so the recursion during print needs to be blocked.

BTreeMap<String, Value> for add_map

I'm trying to use koto and struggling to get a koto instance with values / contexts from a BTreeMap<String, serde_yaml::Value>

I don't suppose anyone has done this and could provide an example?

Thanks

[question] How to add modules implemented in Koto to the prelude?

A previous question was about dynamic loading of modules written in Rust. However, I need something similar for Koto modules. I want my application to download Koto modules based on a dependency specification and make them available to the Koto script executed by the application, preferably without having to import them in the script itself (something like a dynamic prelude). Is that possible somehow?

Single-threaded runtime

Koto currently supports multi-threaded scripting, with containers implemented in terms of Arc<RwLock<…>>.

I added this support for threading mostly as an experiment, and given that Koto is primarily targeted for use as an embedded language for host applications, and single-threaded use cases are clearly the priority, I now find it unsatisfying for the runtime to be dealing with the overhead of having a thread-safe runtime.

I think that switching to a single-threaded runtime now will be a good step forward, and then support for different flavours of parallelism can be enabled via support libraries, with features added into Koto for specific use cases.

Add language support to Helix text editor

I love the language, but my main editor is helix, and id rather not open/install vscode just for this language. I don't know how hard it is to add support for it if that's even possible, but I figure I might as well put it here.

Iterators can't be copied

Discussed in #89

^{Originally posted by ales-tsurko October 9, 2021}
I've found that copy and deep_copy of list doesn't work if item is iterator:

foo = [(0..10).iter()]
...
» foo.deep_copy().get(0).next()
0
» foo.deep_copy().get(0).next()
1
» foo.deep_copy().get(0).next()
2

Also, is there a way to copy an iterator directly?

Iterators currently share state between instances. The Iterable data that's being iterated over can still be shared, but the index should be made unique to the degree where the following example would work as expected:

x = (1..100).iter()
y = x

x.next() # 1
y.next() # 2 - I would expect 1 here

x.next() # 3
z = x

x.next() # 4
z.next() # 5 - I would expect 4 here

Defining a variable programmatically

Can I define a variable using a function? Like this, for example:

def_var "foo", 42

assert_eq foo, 42

I tried to find a way to do this today surfing the source code. And here are my thoughts/questions on this so far:

1. As modules are maps, can I access the current context/module and modify it? Is there some kind of self/this for current context? If I could add keys to the current context, the key would be the variable.
2. I could make it using a fake module, or, saying, using the import in a wrong way:

   fake = {}
   fake.insert "foo", 42
   
   import fake.foo
   
   assert_eq foo, 42

   But I didn't find a way to pass the key as a string in the import statement. Some kind of import all (from fake import * or import fake.*) would work, by I didn't find how to do this either.
3. On the Rust side I can use prelude. A little confused about it. It's kind of a global context? Or it's the current context? Anyway, I was sure that when I'd add a value to this, this would be the variable definition. But it turned out I needed to import it, so I returned to 2.

Is there any other ways to do this? Or, may be, solutions for the issues I faced in the above points?

Easier embedding of Rust / into Rust

Hi there. I'm experimenting with embedding koto in another crate, and I feel the protocol around ExternalVaues (as showcased in the example; to my knowledge there are no other docs) is quite cumbersome.

I have built a procedural macro that can insert Rust functions into a given prelude, however, I'd like to also extend it to derive support for my own types (perhaps in conjunction with the proc macro also setting up a kind of vtable). What do you think about simplifying this interface to an ExternalValue + a Box<dyn fn(KeyValue) -> ExternalFunction> that will do the vtable dispatch? This might both be faster and easier to code against.

Bitwise operations are missing

Raised in #36

That would also be great not only have bitwise operations on numbers, but also being able to overload bitwise operators (when #36 be ready).

Is it possible to extend the core modules from Rust?

It should be possible to extend existing modules with more functions. Let's say I want to make use of the built-in io module but my interpreter needs to add more functionality which fits into the same category and should thus use the same namespace. Then I'd like to extend the existing KMap with the new functions in the prelude.

Add control over # of decimal places in string formatting

There's currently no way to set the number of decimal places that should be displayed when formatting numbers.

"{}".format 1/3
# 0.3333333333333333

Borrowing Rust's syntax seems like a likely solution as it fits with existing value selectors and can be naturally expanded with other formatting modifiers.

"{x:.2} -> {y:.3}".format {x: 1/3, y: 1/6}
# 0.33 -> 0.167

Better language guide search in the REPL

• Only top-level guide topics should be listed when typing help,
  • e.g. 'Maps', 'Lists', etc., instead of every sub-topic.
• Show sub-topics when showing a top-level topics as 'see also' references.
  • e.g. help loops should list the available looping topics
• Include H3 section headings as search keywords
  • e.g. help truthiness should show the section for Null

KotoFile::id() should return a ValueString

KotoFile::id() currently requires implementations to return a String, forcing recreation of the ID on each call.

Add `iterator.flatten`

It would be useful to have an iterator adaptor similar to Rust's Iterator::flatten.

The idea is to have an iterator adaptor that inspects incoming values, and if the value is iterable then iterate over its contents, otherwise output the value.

e.g.

x = (1, 2, 3), 4, [5, 6, 7, [8, 9]]
x.flatten().to_tuple()
# (1, 2, 3, 4, 5, 6, 7, [8, 9])

The flattening only affects one level of iteration, so in the example above the [8, 9] list is left alone.

Function objects, `@||` meta key

It would be useful to be able to define function objects in Koto.

This could be achieved by adding a @|| meta key that can be defined in Maps, which would then allow the map to be treated as a function.

e.g.

my_function = 
  @||: || 42

my_function_with_args =      
  @||: |x, y| x + y

my_instance_function =      
  data: [1, 2, 3]
  @||: |self| self.data

print my_function()
# 42

print my_function_with_args 2, 3
# 5

print my_instance_function()
# [1, 2, 3]

Add load/run functions to allow for running Koto code inside a script

It would be useful to have an eval style feature in Koto.

There are lots of use cases to consider, but I think a meaningful starting point could be achieved with the following:

koto.load

|String| -> Chunk
|io.File| -> Chunk

Compiles a String or io.File and returns a Chunk.

The chunk would be a Map with an ok boolean that can be checked for successful compilation, and @display could be implemented to print out error information.

The chunk could be executed repeatedly with koto.run, or the chunk could be a function object if/when that feature gets introduced (see #163). The result of the script would be returned from the call.

Example

match koto.load "1 + 1"
  chunk if chunk.ok then 
    print chunk.run()
  error then 
    throw error
# -> 2

koto.run

|String| -> Value
|io.File| -> Value
|Chunk| -> Value

Compiles and runs a String or io.File, or runs a Chunk, and returns the resulting Value.

This would be useful as a shortcut for when a one-off script needs to be executed. Compilation errors would throw.

Example

koto.run "1 + 1"
# -> 2

Settings / Prelude / Exports

The load and run functions could be extended to expose settings that control compilation or runtime behaviour, but I think these features should be added later based on concrete use cases.

Module cannot be called 'test'

It seems that a module called test is not loaded at all. There's no error message when trying to import it, which confused me for a good amount of time. If this is intended then please add an error message in the runtime about test being an invalid module name.

Allow coloured CLI output to be disabled with NO_COLOR

See https://no-color.org

`+=` and other assignment operations broken in the REPL

The result of operations like += isn't assigned correctly for numbers in the REPL.

» x = 1
➝ 1

» x += 1
➝ 2

» x
➝ 1

koto.args should be a Tuple instead of a List

koto.args shouldn't be modifiable. Until there's some notion of const, the args value can be replaced, but at least it shouldn't be a mutable List, so a Tuple would be a better choice.

Add throw expressions

try/catch/finally are available to recover from errors, but the ability to manually throw an error is missing.

I think that throw x could be comfortably added to the language, where x is either a string or a map that implements @display (attempting to throw another value type would be a runtime error).

e.g.

try
  throw "A terrible error occurred"
catch error
  error.print()

try
  throw 
    data: foo
    @display: |self| "A terrible error occurred: {}".format self.data
catch error
  log_error_data error.data
  throw error

Implementation thoughts:

• A new 'throw` keyword and bytecode op can be added.
• A new error variant can be added to RuntimeError that includes the thrown value.
  • A backtrace should probably be built up in the new variant, see RuntimeError::extend_trace
• I'm unsure about how to handle uncaught errors, should they be resolved to a string before exiting the VM? Probably not, but then it would be good to have a convenient helper for easily evaluating the error message.

Issue created following discussion in #54.

Getting a function's argument name

Is there a way to get a function's argument name?

In my case I need it to capture arguments from callback, to expose them later as parameters. There would be API, for example:

# there is a function to define a node, the arguments are:
# - name of the node
# - output types (audio/control rate per output)
# - input types
# - callback, which will generate a syntax tree from which Csound code will be generated
node_def "synth 1", "aa", "kkk", |freq, amp, mul| 
  l = oscili.a freq, add, mul * 0.5
  r = oscili.a freq + 10, add, mul * 0.5
  out l, r

# node definitions are added into a global dictionary, then I can init one like this
# (we specify types as nodes are polymorphic (= there can be the same type of nodes, 
# but with different types of inputs/outputs))
n = node "synth 1", "aa", "kkk", 440, 0, 0.5

# then I'd like to be able to set parametrs by name, for this I need to get the names
# of the arguments from the callback
n.set "freq", 220

I also need the arguments names to generate readable variables in the produced Csound code, but I can go with some random names for that.

Broken Link

'docs examples' tests is broken

https://github.com/koto-lang/koto/blob/main/docs/language/_index.md

Add 'key function' overloads for iterator.[min, max, min_max]

The iterator.[min, max, min_max] functions would benefit from having optional 'key' functions, where the result of calling the function with the iterator's output is used in comparisons. A key overload is already available for list.sort.

x =
  foo: 42
  bar: 99
  baz: -1

# What I would like to write:
# max_key, max_value = x.max |(_, value)| value

# But instead I have to write something like:
max_value = x.each(|(key, value)| value).max()
max_key, _ = x.get_index x.position |(key, value)| value == max_value

io.print "Entry with maximum value in x is $max_key: $max_value"
# Entry with maximum value in x is bar: 99

Reverse iterators

It would be good to have an iterator.reversed() function that reverses the direction of any iterator that can support being reversed.

Allow `import` reassignments using `as`

It's currently possible to rename imported items by reassigning them with a new name:

rng_bool = from random import bool

This isn't bad, but it becomes unwieldy when reassigning multiple items:

rng_bool, rng_number = from random import bool, number

Introducing as would allow for inline reassignment, which is more straightforward, and will be less error prone for more complex import expressions.

from random import bool as rng_bool, number as rng_number

Function pipe operator

Motivation

Long function call chains can get a bit hard to read at first glance:

input = parse_input io.read_to_string io.extend_path koto.script_dir, "input", "03"

This can be broken up with parentheses or indentation to aid the reader:

input = parse_input(io.read_to_string(io.extend_path koto.script_dir, "input", "03"))

# or 

input = parse_input 
  io.read_to_string 
    io.extend_path koto.script_dir, "input", "03"

...but an arguably more natural way to express the flow of results being passed to subsequent functions is to use a pipe operator:

input = io.extend_path koto.script_dir, "input", "03" 
  >> io.read_to_string 
  >> parse_input

I find this very easy to read, along the lines of 'do this, and then do this, and then do that'

Details

Behaviour

The >> operator will take the value to the left, and then pass it as an additional argument to the function call to the right.

e.g.

f x >> g y >> h
# is equivalent to:
h(g(y, f(x)))

Implementation

As a guess for how this could be implemented:

• The lexer can look ahead after encountering > to see if it's a pipe or greater than token.
• The parser can support this by introducing a new BinaryOp with low precedence and left-to-right associativity.
• The compiler then passes the LHS as an additional argument to the RHS, ensuring that the RHS is always a interpreted as function call.

Choice of operator

Alternative operators to consider for piping:

• |>
  • This is used by Elixir and Elm, but I think the reader should only see | when making functions.
• ->
  • The arrow-like operator is nice and clear, and perhaps a bit less 'shouty' than >>, but if/when type annotations are introduced for functions then -> will be used for annotating the return type (e.g. f = |x: Int| -> Bool, and I think it would be good to avoid ambiguity here.

Better ergonomics for implementing Koto functions / types in Rust

Please add some macros for adding new functions to KMaps and for defining and registering custom types. IMHO the interface is too complicated and verbose at the moment.

Windows support for the CLI

It would be good to enable a CI job for Windows that runs at least the REPL tests.

`@as_bool` meta key

It could be useful at some point to allow customization of the behaviour when a value is encountered in conditional expressions.

make_foo = |x|
  data: x
  @as_bool: |self| self.data > 0

my_foo = make_foo 42
if my_foo 
  do_something()
else 
  do_something_else()

Optimize converting ValuePairs to Tuples when calling iterator functors

Iterator output can either be a single value, or a pair of values.

ValuePairs are an optimization for iterators that provide a pair of values as output (like map.iter() and iterator.enumerate), with for loops taking the pair of values and placing them directly in registers, rather than creating a new ValueTuple for each pair.

Iterator adaptors like iterator.each collect any ValuePairs into a tuple before calling the adaptor's functor.

e.g.

for key, value in {foo: 42, bar: -1}
  # Here, `key` and `value` are placed directly into registers by the runtime
  debug key, value

{foo: 42, bar: -1}
  .each |(key, value)| # Note that `key` and `value` are unpacked from a tuple here
    debug key, value
  .consume()

The reason to collect the pair of values into a tuple is to simplify the use of .each(); it's easy to communicate to the learner that a single value comes in, and a single value is produced as output. If the input value is a tuple, then it's easy to unpack.

The downside is that a tuple is created for each iteration when it would otherwise be unnecessary.

It would be good to find a way to avoid having to create ValueTuple for each call to an iterator functor, perhaps by allowing iterator adaptors to pass in the ValuePair as a TemporaryTuple?

`num2`/`num4` improvements

• Replace num2 / num4 keywords with maker functions in the num2/num4 modules, e.g. num2.make_num2, and include them in the top-level of the prelude.
• Add iterator.to_num2() and iterator.to_num4()
  • This will allow the maker functions to be simplified; they can work only with numbers, and then the iterator functions can be used for converting from lists or tuples.
• Add useful aliases for components for working with coordinates (x(), y(), etc.), or colours (r()/g()/b(), etc).

Make curly braces optional for inline maps

When using a series of comma-separated key: value pairs to define a map, it would be nice to make the curly braces optional.

e.g. this:

x = {foo: 42, bar: "abc"}

could be written as:

x = foo: 42, bar: "abc"

without any ambiguity.

Curly braces are still useful for defining empty maps and using the 'valueless entry' feature:

empty_map = {}
foo, bar = 42, "abc"
x = {foo, bar}

Does koto runs on a stack based vm?

Yeah well so i liked koto which syntax seems to be like python so i have just wanted to know is koto running on a stack based vm?

Add support for hex, octal, and binary number notation

Number parsing in Koto doesn't support hex, octal, or binary number notation.

a = 0xabadcafe

...

Unexpected token
 --- temp.koto - 3:6
   |
 3 | a = 0xabadcafe
   |      ^^^^^^^^^

"pop_frame: Empty call stack" when folding a tuple of maps

See the following example for the issue and a workaround:

import io, koto, string.print

export main = ||
  map_is_valid = |m| m.get("bar") != ()
  tuple_of_maps = ({foo: 2}, {}, {foo: 4, bar: 2})

  # uncomment the following print for the following error:
  #
  #     pop_frame: Empty call stack
  #      --> fold_test.koto - 1:1
  #        |
  #      1 | import io, koto, string.print
  #        | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  #
  # print "is_valid 2: {}" (map_is_valid tuple_of_maps[2])

  print "is_valid 2: {}" (tuple_of_maps[2].get("bar") != ())

  n_valid = tuple_of_maps.fold(0, |acc, m| acc + (if map_is_valid(m) then 1 else 0))

  print "n_valid: {}" n_valid

Add REPL support for the Delete key

Backspace is supported in the REPL, and it would be nice if Delete was supported too.

Allow direct access to the module's export map

It would be good to be able to directly access a module's 'export' map (referred to as 'global' in the code) so that items could be added programatically.

I'm thinking that adding koto.exports() as a getter that returns the current module's export map could be a good way to go.

e.g.

import koto
x = "value_x"
koto.exports().insert x, 99
assert_eq value_x, 99

See #45 for discussion.

Remove packed numbers (Num2 / Num4)

As much as I like the idea of having the Num2 and Num4 types in Koto, there are good reasons for removing them.

• They're not mutable, so any operations on them have to be assigned, which is fiddly in practice.
• They add complexity to the language, making Koto harder to learn.
• Their presence adds complexity for library authors (more than one standard number type to accept as function arguments).
• Having standard packed number types means their standard library modules need to cover too many use cases.
  • e.g. should num4 be a color type or a 3d or 4d vector? Or a rect?
• Libraries/apps will generally add wrappers for their own internal geometry or color ops.

In the end, I think their value is limited compared to the cost of including them, so they should be removed as soon as possible to avoid breaking too many existing programs.

Is it possible to extend a primitive?

On the language level. Say, adding a method to number or string?

Operator overloading

Hi. I see a branch for operator overloading, but it looks like you switched to other tasks. Do you have any plans to finish it and whether there's any estimation, when it will be ready if the answer is yes?

And a little off-topic, as I haven't found how to contact you personally. Koto is awesome! I started a project to make some permissively licensed embeddable alternative to SuperCollider (you can read motivation here, and here is a little about design, but it has changed since then). I considered to use JS with some syntactic sugar and Csound as a sound engine. But now, when I found koto, I considered making it either just a library for koto or something based on koto. I think it's very suitable for live coding and programming newcomers. What's your plans on koto?

Optimize tail calls

Currently no effort is made in Koto to optimize tail calls, making it easy to run into stack overflows when implementing recursive functions.

e.g. The following script takes a huge amount of memory to execute:

f = |x|
  if x == 0
    return
  f x - 1
f 123456789

The idea would be that if a terminating expression (i.e. throw or return (implicit returns for the last expression in a function included)) in a function is a call, then instead of creating a new frame, the current function's execution frame can be reused.

Remove Value's implementation of fmt::Display

Since introducing the ability to customize the way values are displayed via @display, it doesn't make sense to implement fmt::Display for Value, and instead any uses should call UnaryOp::Display on the value via the VM.

Noob question. How do i print something without a newline?

Hi,
is it possible to print text without a new line?
Thanks.

Add geometry and color libs

With the removal of packed numbers (#201) it would be good to introduce a couple of libraries that cover the primary use cases for Num2 (2d vectors) and Num4 (3d and 4d vectors, and color ops).

The libraries don't need to be extensive at first, just enough simple types and operations to get the ball rolling.

Add `iterator.cycle`

Since #99 iterators have been copyable, so it should be possible now to implement an iterator.cycle iterator adaptor that copies an iterator and repeats its output endlessly.

(1, 2, 3).cycle().take(10).to_list()
# [1, 2, 3, 1, 2, 3, 1, 2, 3, 1]

Can I link a module dynamically?

I found how to compile the modules with Koto itself. But can I write a module in rust, compile it separately and then import it in already compiled Koto?

Run wasm tests on CI

The wasm CI job checks that the Koto library compiles on CI but it doesn't run any tests, it would be good to run at least one test with the compiled wasm, but maybe we can do a bit better, e.g. run all tests in the koto/tests folder?