The caramel from leostera

Nested let-in expressions compile to wrong erlang

Describe the bug
When a let expression value contains a let expression, the generated erlang is "flattened" and running the code produces an incorrect result (or may not compile at all)

To Reproduce

Create a main.ml file containing

let print_int thing = Io.format "~0tp~n" [ thing ]

let main _ =
  let one = 1 in
  let three =
    let two = one + 1 in
      two + 1 in
    print_int three

I would format it slightly differently, to have the last line nested one level less, or with parens around the nested let-in, but that's how the formatter prints it.

Run command

$ caramel compile main.ml && escript main.erl

See the incorrect result:

Compiling main.erl      OK
2

The generated erlang is:

% Source code generated with Caramel.
-module(main).

-export([main/1]).
-export([print_int/1]).

-spec print_int(_) -> ok.
print_int(Thing) -> io:format(<<"~0tp~n">>, [Thing | []]).

-spec main(_) -> ok.
main(_) ->
  One = 1,
  Three = Two = erlang:'+'(One, 1),
erlang:'+'(Two, 1),
  print_int(Three).

Expected behavior
It should print 3, just like this Ocaml file would:

(* let print_int thing = Io.format "~0tp~n" [ thing ] *)

let main _ =
  let one = 1 in
  let three =
    let two = one + 1 in
      two + 1 in
    print_int three ;;

main ()

$ ocaml main.ml
3

The generated Erlang should probably be:

-module(main).

-export([main/1]).
-export([print_int/1]).

-spec print_int(_) -> ok.
print_int(Thing) -> io:format(<<"~0tp~n">>, [Thing | []]).

-spec main(_) -> ok.
main(_) ->
  One = 1,
  Three = begin
    Two = erlang:'+'(One, 1),
    erlang:'+'(Two, 1)
  end,
  print_int(Three).

Environment (please complete the following information):

OS: [e.g. macOS, Linux, Windows] Ubuntu Linux 18.04
Caramel version [e.g. 0.1] 0.1.0+git-f7b1222
Erlang version (e.g. 23.2.4) 23.2.6

Large repository due to metadata

Hey,

I am willing to have a look at the project and hopefully contribute to it in the future (improving my ocaml skills now :P) but one thing caught my attention.
the .git folder his huge, would you consider a cleanup?

du -sh caramel
184M	caramel
du -sh caramel/.git
181M	caramel/.git

Fun references of operators have the wrong arity and module name

From #57 by @michallepicki:

This leads to another related issue, currently:

Lists.foldl ( + ) 1 [1; 2; 3]

generates

lists:foldl(fun '+'/0, 1, [1 | [2 | [3 | []]]])

where it lacks the erlang: prefix and the arity is wrong. Not sure if this is something that should be supported right now or not.

Ensure OCaml variable names are translated to valid Erlang

The following OCaml code idiomatically indicates that the new variable started as the old variable and is somehow changed, by priming it, or adding a ' to its name.

let f () =
  let x = 1 in
  let x' = x + 1 in
  let x'' = x' + 1 in
  x''

This should compile to Erlang as:

f() ->
  X = 1,
  X1 = erlang:'+'(X, 1),
  X2 = erlang:'+'(X1, 1),
  X2.

but it instead compiles to this:

f() ->
  X = 1,
  X' = erlang:'+'(X, 1),
  X'' = erlang:'+'(X', 1),
  X''.

To fix this, we need to work on the Compiler.Ocaml_to_erlang.Names module and make sure that anywhere we are creating a variable name we are making sure it is valid Erlang.

This has some issues, since automatic renaming needs to be done in a scope-aware manner for it to be entirely safe, but we have some advantages:

all variable names go through one specific translation path
variable names are syntactically different than any other names

Which means we could potentially get away with a very naive replacing of the number of ' to a suffix that is valid Erlang. This could be just the number, or something less likely to clash, like X_prime and X_prime_prime.

Update GitHub Actions workflows to use Env files

We're getting a bunch of warnings about set-env being deprecated. e.g: https://github.com/AbstractMachinesLab/caramel/actions/runs/339535013

The set-env command is deprecated and will be disabled soon. We should upgrade to using Environment Files.

For whoever picks this up, here's a guide: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/

Build documentation on CI instead of requiring contributors to run mdbook locally

I would like to contribute to the documentation, but I am not very happy when doing that requires me to install additional tooling. Then, there are potential conflicts in HTML. Additionally, I think because mdbook is available in various versions and platforms, requiring contributors to install it may result hard to debug bugs or regressions.

Rendering the documentation on CI, would allow contributors to only modify .md files to update the documentation. Rendering could be done in a workflow that only runs from master, checks out the changes, runs mdbook and pushes rendered HTML to a separate branch. (There is a small security concern here, to not allow people to push to that branch from CI runs e.g. on pull requests). I saw similar solutions using something-different-than-jekyll to use GitHub Pages for a blog. (I am assuming here that the manual is served from Github Pages, but if it's not, then deployment would probably be similar to the current one, just based on a different branch.)

parsing erlang terms

I run Erlang.Parse.from_file on
https://github.com/erlang/otp/blob/master/lib/wx/api_gen/wxapi.conf

and get the error

failed: In wxapi.conf.copy, at offset 820: syntax error.
  Erlang__Erl_parser.MenhirBasics.Error

probably because the file defines terms to be read by file:consult/1
and is not appropriate to the front door of your parser.
but with a different entry point it could parse terms?

Forbid let rec inside of a let binding

The following OCaml

let let_rec () =
  let rec f x = f (x + 1) in
  f 1

should emit a compilation error that let rec bindings inside of a function definition are not allowed. It should suggest the programmer to just define an auxiliary function outside, and it can reuse the name of the let rec binding for it. For example:

We found a `let rec` binding for `f` inside of `let_rec/0` and these are not supported.
Did you mean to define a `f_aux` outside instead?

Instead it compiles to this Erlang:

let_rec() ->
  F = fun (X) -> f(erlang:'+'(X, 1)) end,
  F(0).

which is misleading at first, but we can tell is wrong because it is trying to recurse into the function f/1 which is in fact not defined anywhere.

This can be fixed in the mapping of Texp_let values by checking if the recursive flag is set to true and extending the OCaml_to_erlang.Error module with the right message.

Other notes

I'd like to enable this pattern in the future, as it could be done translating the expression to:

let_rec() ->
  RecF = fun(G) -> fun (X) -> (G(G))(X+1) end end,
  F = RecF(RecF),
  F(0).

which could be standardized as part of the caramel.erl runtime libraries:

let_rec() ->
  F = caramel:letrec1(fun (F, X) -> F(X + 1) end)
  F(0).

But I fear this would not be as useful right now. We'll see later!

mix plugin

Right now to be able to use caramelc in an Erlang or Elixir project, we sort of have to manually invoke it in the right place and then also compile the caramel stdlib erlang sources and make sure its available. An entire dance.

It'd be useful to have a mix plugin to see what the interop from the Elixir side feels like so that we can just create a new mix project, and if you have caramel available in your path then it will do the build.

For some inspiration have a look at https://github.com/seanhinde/rebar3_caramel and #22

The end result would be that you can drop in a couple of .ml files in your sources and have it available in the repl.

Support OCaml style floating point syntax without trailing numbers after dot

Currently this Caramel code compiles:

let main _ =
  42.

to this Erlang:

% Source code generated with Caramel.
-module(main).

-export([main/1]).

-spec main(_) -> float().
main(_) -> 42..

Erlang compilation fails with:

main.erl:6: syntax error before: '..'

This should either:
a) compile to 42.0 in Erlang
b) or print a helpful error when compiling Caramel

Caramel 0.1, Erlang 23.2.2

Support parenthesized expressions in Erlang parser

The following Erlang code:

A + (B + C)

should parse to the equivalent of:

% assume open Erlang.Ast_helper
Expr.apply
  Well_known.Erlang.plus
  [ Expr.ident (Name.var "A")
  ; Expr.apply
      Well_known.Erlang.plus
      [ Expr.ident (Name.var "B")
      ; Expr.ident (Name.var "C") ] ]

But is currently unsupported by the grammar in ./src/erlang/erl_parser.mly.

Support Reason syntax too

That's it. Just bring in the Reason parse transform onto the pipeline.

v0.2 - OCaml libraries to manipulate Erlang

Summary

v0.1 of this library focused on getting as much of the language as possible handled by the parser, AST, and printer. This next version should complete that work, and put some emphasis on the usability of the parser with better errors by tracking location information.

Checklist

Usability and Errors

Tokens carry positioning information

Language Support

Pattern Language

Record patterns

Expression Language

Support for Guards

Discussing support for guards with @MargaretKrutikova don't think we can come up with a transform that would preserve the semantics of the guard in the presence of side-effects.

E.g, if they are lifted up, bound to a variable name, and then used in a regular boolean equality check, you'd evaluate all the guards even if you match the first one.

This ocaml:

let f x = 
  match x with
  | y when (g x) -> `ok
  | y when (h x) -> `noop
  | _ -> `default

would compile to some Erlang, like:

f(X) ->
  G_X = g(X),
  H_X = h(X),
  case X of
    Y when G_X -> ok;
    Y when H_X -> noop;
    _ -> default
  end.

But while the above ocaml didn't run (h x) if (g x) was true, the erlang version would run them! Imagine if g or h had side-effects so that h depended on the side-effects of g and it crashes.

Not saying that it is a good idea to use side-effectful functions as guards in OCaml, but it is certainly possible!

So the maybe for now we just want to say no to guards unless they can map directly to Erlang built-in functions. We'd have to add them to the list of allowed translations in ./src/compiler/ocaml_to_erlang/names.ml and possibly mark them as built-in or userland and do a little work in ./src/compiler/ocaml_to_erlang/fun.ml to check if the function used as a guard has a qualified name that is a built-in. If it is not, then we can just error out.

problem with windows security

wondering if its a false positive

Standard Library Ideas

In this issue I'm collecting thoughts of the structure of a standard library. This may seem a lot larger than you'd have in OCaml/Reason to begin with, but Erlang's is actually pretty big, and includes tons of IETF RFC languages used in networking and dist. sys, so we're not actually being that crazy here.

This also doesn't mean we need to build all of these, as in most cases there exists a single canonical library in the Erlang/Elixir world that we can use for it already, so we just need to provide good bindings to it.

Libraries

Ops: tools for operations
- Tracer
- Log
- Perf
Core: common datatypes
- Format
- Strings
- Dates
- RegExp
- Result
- Option
- Bool
- Fun
- Atom
Async
- Future
- Timers
Collections
- HashMap / HashSet
- Map / Set
- Queue/DeQue
- List / Vector
Num: working with numbers
- Int / Float
- BigInt / BigDecimal
FileSystem: working with the file system directly
- File
- Path
OS: working with the OS directly
- Env
Net: common networking datatypes / specs
- HTTP
- URI/URL
- MIME
CoDecs: common encoders/decoders
- MIME
- Base
- CSV, XML
- TOML / YAML
- JSON
Crypto / Rand
- UUID
- SHA
- ...
Typed OTP
- Supervisor

Feel free to drop below your ideas for the standard library and if you can drop a link to where it came from 🙌🏼

Licence

Hi this project looks really interesting but it's missing a licence file

Support parsing of Record expressions and patterns

Checklist:

Record patterns
Record expressions
- New record
- Record update

Cascading match cases result in empty patterns

This OCaml:

match 0 with
| 0 | 1 -> true
| _ -> false

Should compile to this Erlang:

case 0 of
  0 -> true;
  1 -> true;
  _ -> false
end

but compiles to this Erlang:

case 0 of
  _ -> true;
 _ -> false
end

This is a bug somewhere in compiler/ocaml_to_erlang/fun.ml:L246-L258 -- and possible in the mk_pattern function of the same module.

Local function is shadowing imported function with the same name in generated erlang

To Reproduce

Create a file main.ml containing

let print thing = Io.format "~0tp~n" [ thing ]

let weird_join_but_ok list joiner =
  let join el acc = acc @ [el; 0] in
    Lists.foldl join [] (Lists.join joiner list)

 let main _ =
  print (weird_join_but_ok [1; 3] 2) ;
  ()

Run command caramel compile main.ml && escript main.erl
Erlang gets generated with

-spec weird_join_but_ok(list(integer()), integer()) -> list(integer()).
weird_join_but_ok(List, Joiner) ->
  Join = fun
  (El, Acc) -> erlang:'++'(Acc, [El | [0 | []]])
end,
  lists:foldl(Join, [], Join(Joiner, List)).

When running, it results in:

[1,0,3,0,2,0,0,0]

Expected behavior
Erlang gets generated with

  lists:foldl(Join, [], lists:join(Joiner, List)).

and running it results in

[1,0,2,0,3,0]

Environment (please complete the following information):

OS Ubuntu Linux 18.04
Caramel version 0.1
Erlang version 23.2.2

sponsor directly through github

I am unable and now unwilling to sponsor through patreon, furthermore direct github sponsorships do not take a cut.

I'd like to give you 20$/month to support this project.

Partial function application compiles to wrong erlang

Describe the bug
A simple function that taking no arguments in Caramel, that only runs Io.format, compiles to invalid erlang that says it is taking two arguments, which erlang/OTP cannot compile.

To Reproduce

$ caramelc --version
0.0.14+git-27bc02d
$ cat sweetness.ml 
let text () = "Hello, world!"

let hello () = Io.format (text ())
$ caramelc compile sweetness.ml
Compiling sweetness.erl OK
$ cat sweetness.erl 
% Source code generated with Caramel.
-module(sweetness).

-export([hello/2]).
-export([text/0]).

-spec text() -> binary().
text() -> <<"Hello, world!">>.

-spec hello(ok, list(any())) -> ok.
hello() -> io:format(text()).


$ erl
Erlang/OTP 23 [erts-11.1.6] [source] [64-bit] [smp:6:6] [ds:6:6:10] [async-threads:1] [hipe]

Eshell V11.1.6  (abort with ^G)
1> c(sweetness).
sweetness.erl:4: function hello/2 undefined
sweetness.erl:10: spec for undefined function hello/2
sweetness.erl:11: Warning: function hello/0 is unused
error

Expected behaviour
.erl gets created with

-export([hello/0]).

-spec hello() -> fun((list(any())) -> ok).

Getting the README example running

Readme example
I'm trying to get the server on the README page running. Two things so far:

Process.make - I'm thinking this should be Process.spawn now?
I can't figure out how to get erlang versions of the stdlib runtime modules

To Reproduce
Steps to reproduce the behavior:

Create a new rebar3 app
Add the caramel README code to src/holder_annotated.ml
caramel compile holder_annotated.ml
rebar3 shell
holder_annotated:start().
The process erlang module is (obviously) not found

Expected behavior
The server should run

What I've tried
I tried the obvious of generating .erl files from the stdlib .ml by running caramel compile *.ml in stdlib/{beam|ocaml} but get circular dependency errors:

~/local/bin/stdlib/beam$ caramelc.exe compile *.ml
File "_none_", line 1:
Error: cycle in dependencies. End of list is not sorted.
beam.ml: Beam.ml Stdlib_binary.ml Stdlib_calendar.ml Stdlib_erlang.ml Stdlib_ets.ml Stdlib_gen_server.ml Stdlib_gen_statem.ml Stdlib_io.ml Stdlib_lists.ml Stdlib_maps.ml Stdlib_process.ml Stdlib_supervisor.ml Stdlib_sys.ml Stdlib_timer.ml

This is with the tagged release 0.0.6 that includes the stdlib ml files in the bistro.

Any pointers welcome. I'll have a go at a rebar3 plugin if I can figure out a working manual workflow.

Cool project !

Annotating types for function arguments compiles to wrong erlang

Describe the bug
When function arguments have their types specified in a function definition, the Erlang code produced is not valid. Currently only the function return type can be annotated without breaking.

To Reproduce

Create a file main.ml containing:

let print thing = Io.format "~0tp~n" [ thing ]

let add (x : int) (y : int) : int = x + y

let main _ = print (add 1 2)

Run command

$ caramel compile main.ml && escript main.erl

See error

Compiling main.erl      OK
main.erl:11: function x/0 undefined
main.erl:11: function y/0 undefined
escript: There were compilation errors.

main.erl contains:

% Source code generated with Caramel.
-module(main).

-export([add/2]).
-export([main/1]).
-export([print/1]).

-spec print(_) -> ok.
print(Thing) -> io:format(<<"~0tp~n">>, [Thing | []]).

-spec add(integer(), integer()) -> integer().
add(_, _) -> erlang:'+'(fun x/0, fun y/0).

-spec main(_) -> ok.
main(_) -> print(add(1, 2)).

For some reason add arguments are ignored (_) and usages of the arguments are not valid (fun x/0 instead of just X).

Expected behavior

Erlang gets generated the same way as if the user didn't provide type annotations for the arguments:

-spec add(integer(), integer()) -> integer().
add(X, Y) -> erlang:'+'(X, Y).

Environment (please complete the following information):

OS: [e.g. macOS, Linux, Windows] Ubuntu Linux 18.04
Caramel version [e.g. 0.1] 0.1.0+git-f7b1222
Erlang version (e.g. 23.2.4) 23.2.6

Unbound module Stdlib when trying to check a basic erlang module

On the latest release page I saw an interesting point:

new command for type-checking erlang: caramelc check

And I wanted to try this out. So I downloaded the release, created a basic erlang module and saw this:

$ caramelc check mymodule.erl 
Unbound module Stdlib$ cat mymodule.erl 
-module(mymodule).

-export([hey/0]).

hey() ->
    ok.

What am I doing wrong?

v0.1 - Standard Library on the BEAM

Summary

While #6 is about getting support for a set of OCaml language features, this issue is about putting together a standard library that can be used to interface with the Erlang standard library in a more-or-less idiomatic way.

This likely means 2 things: FFIs and a small runtime library. Some modules might be a combination of both.

Checklist

FFIs

In any case, some of these modules will have to break direct compatibility with their Erlang equivalents due to their APIs being uncomfortably unidiomatic OCaml or untypeable in OCaml.

Runtime

These are libraries that actually need the runtime overhead to satisfy some of the type-level constraints.

Process -- as per the experiments i've done on typing message passing, it's likely that we'll need a way to constrain the process types and expose a receive function as a stand in for receive expressions. This module would implement that runtime overhead.
- spawn/2
- contramap/2
- send/2

Forbid name shadowing through let binding

The following OCaml code idiomatically indicates that the new variable started as the old variable and is somehow changed, but it has the same name.

let f () =
  let x = 1 in
  let x = x + 1 in
  x

This should either error, saying that name shadowing is not supported. Itinstead compiles to this:

f() ->
  X = 1,
  X = erlang:'+'(X, 1),
  X.

Which will error at runtime.

To fix this, we can modify Compiler.Ocaml_to_erlang.Fun.mk_bindings to error out the moment it finds a let binding in its ~var_names argument.

Other Notes

In the future I'd like this to be supported, and do the appropriate renaming during translation to ensure no name-clashes happen on the Erlang side.

Rust not mentioned in documentation + `make deps build` fails

Describe the bug

The documentation does not inform users that cargo and rust dependencies are needed to build ocaml-tree-sitter when running make deps build.

Even when rust version 1.49.0 is installed, the following error occurs:

nicolas@localhost:~/nicobao/caramel$ make deps build
opam install dune menhir ocaml-compiler-libs cmdliner ppx_sexp_conv sexplib ocamlformat bisect_ppx
[NOTE] Package bisect_ppx is already installed (current version is 2.5.0).
[NOTE] Package ocamlformat is already installed (current version is 0.15.0).
[NOTE] Package sexplib is already installed (current version is v0.14.0).
[NOTE] Package ppx_sexp_conv is already installed (current version is v0.14.1).
[NOTE] Package cmdliner is already installed (current version is 1.0.4).
[NOTE] Package ocaml-compiler-libs is already installed (current version is v0.12.3).
[NOTE] Package menhir is already installed (current version is 20201216).
[NOTE] Package dune is already installed (current version is 2.7.1).
dune build @all -j8
         gcc ocaml-tree-sitter/tests/sexp_parser.o (exit 1)
(cd _build/default/ocaml-tree-sitter/tests && /usr/lib64/ccache/gcc -O2 -fno-strict-aliasing -fwrapv -fPIC -D_FILE_OFFSET_BITS=64 -D_REENTRANT -O2 -fno-strict-aliasing -fwrapv -fPIC -g -I /home/nicolas/.opam/default/lib/ocaml -I ../src -o sexp_parser.o -c sexp_parser.c)
sexp_parser.c:1:10: fatal error: tree_sitter/parser.h: No such file or directory
    1 | #include <tree_sitter/parser.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
       bootc src/stdlib/beam/beam.{cma,cmi},src/stdlib/beam/binary.{cma,cmi},src/stdlib/beam/calendar.{cma,cmi},src/stdlib/beam/caramel_runtime.{cma,cmi},src/stdlib/beam/erlang.{cma,cmi},src/stdlib/beam/ets.{cma,cmi},src/stdlib/beam/io.{cma,cmi},src/stdlib/beam/lists.{cma,cmi},src/stdlib/beam/maps.{cma,cmi},src/stdlib/beam/process.{cma,cmi,erl},src/stdlib/beam/timer.{cma,cmi}
Compiling maps.erl	OK
Compiling ets.erl	OK
Compiling erlang.erl	OK
Compiling calendar.erl	OK
Compiling binary.erl	OK
Compiling io.erl	OK
Compiling process.erl	OK
       cargo ocaml-tree-sitter/src/dllocaml_tree_sitter.so,ocaml-tree-sitter/src/libocaml_tree_sitter.a (exit 101)
(cd _build/default/ocaml-tree-sitter/src && /home/nicolas/.cargo/bin/cargo build --target-dir ../../target)
    Updating crates.io index
    Updating git repository `https://github.com/AbstractMachinesLab/tree-sitter-sexp`
   Compiling cc v1.0.66
   Compiling memchr v2.3.4
   Compiling proc-macro2 v1.0.24
   Compiling unicode-xid v0.2.1
   Compiling syn v1.0.58
   Compiling lazy_static v1.4.0
   Compiling regex-syntax v0.6.22
   Compiling anyhow v1.0.37
   Compiling ocaml-sys v0.19.0
   Compiling cty v0.2.1
   Compiling thread_local v1.1.0
   Compiling aho-corasick v0.7.15
   Compiling quote v1.0.8
   Compiling tree-sitter v0.17.1
   Compiling tree-sitter v0.17.1 (/home/nicolas/nicobao/tree-sitter/lib)
   Compiling tree-sitter-sexp v0.1.0 (https://github.com/AbstractMachinesLab/tree-sitter-sexp?branch=main#b46b9f78)
   Compiling regex v1.4.3
   Compiling synstructure v0.12.4
   Compiling ocaml-derive v0.19.0
   Compiling ocaml v0.19.0
   Compiling ocaml-tree-sitter v0.1.0 (/home/nicolas/nicobao/caramel/_build/default/ocaml-tree-sitter)
error[E0603]: module `ffi` is private
   --> src/lib.rs:147:53
    |
147 |     let raw_lang = raw_ptr.0 as *const tree_sitter::ffi::TSLanguage;
    |                                                     ^^^ private module
    |
note: the module `ffi` is defined here
   --> /home/nicolas/nicobao/tree-sitter/lib/binding_rust/lib.rs:1:1
    |
1   | mod ffi;
    | ^^^^^^^^

error[E0603]: tuple struct constructor `Language` is private
   --> src/lib.rs:148:29
    |
148 |     let lang = tree_sitter::Language(raw_lang);
    |                             ^^^^^^^^ private tuple struct constructor
    | 
   ::: /home/nicolas/nicobao/tree-sitter/lib/binding_rust/lib.rs:34:21
    |
34  | pub struct Language(*const ffi::TSLanguage);
    |                     ---------------------- a constructor is private if any of the fields is private
    |
note: the tuple struct constructor `Language` is defined here
   --> /home/nicolas/nicobao/tree-sitter/lib/binding_rust/lib.rs:34:1
    |
34  | pub struct Language(*const ffi::TSLanguage);
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to 2 previous errors

To Reproduce

Install the rust toolchain : https://www.rust-lang.org/tools/install
Run make deps build in the root directory of the repository

Expected behavior

The build should succeed.

Desktop (please complete the following information):

OS: Fedora 33 [x86_64]
rust version 1.49.0
opam version 2.0.7
make version 4.3
dune version 2.7.1

Additional information

Content of ocaml-tree-sitter/Cargo.toml was modified:

[package]
name = "ocaml-tree-sitter"
version = "0.1.0"
authors = ["Leandro Ostera <[email protected]>"]
edition = "2018"

[lib]
crate-type = ["staticlib", "cdylib"]

[dependencies]
anyhow = "*"
ocaml = "*"
tree-sitter =  { path = "/home/nicolas/nicobao/tree-sitter/lib" }
tree-sitter-sexp = { git = "https://github.com/AbstractMachinesLab/tree-sitter-sexp", branch = "main" }

The tree-sitter dependency corresponds to my local fork of tree-sitter.
If i use "*" instead of { path = "/home/nicolas/nicobao/tree-sitter/lib" }, I get the same error.

rebar3 plugin

As promised in #19 I've had a go at a rebar3 compiler plugin

So far this is standalone and very basic, relying on an existing caramel installation and very new rebar3 features

I'm not sure about rebar3 support for git submodules, but I'm happy for you to host this as another repo under AbstractMachinesLab if and when you think it's ready.

v0.1 - OCaml on the BEAM

Summary

I have not found an easy way to lay out what a release for Caramel should look like, so I'm sketching out this issue with a a big checklist of stuff that I think could be somewhat usable for pure-OCaml modules.

To do this we need:

the core type language, excluding support for the fancier features of the type system (like GADTs, or universal quantification in records), so variants, records, tuples, and type parameters.
immutable, functional ocaml, excluding all the imperative and object-oriented features, and some features like recursive let bindings which have no direct translation to Erlang
enough pattern matching so we can work with all the values allowed by the core type languages
support for modules, but not as first class values, and without functors or inclusions

This could already be enough to write some OCaml on the BEAM successfully. We'll see once we get there.

Checklist

To make it easier to keep track of what's what and where we currently are, here's a list of the features that are currently working and missing. Where necessary I'll try to make separate issues and link them here.

Idioms

Empty tuple () is a common value in OCaml, this is translated to the atom ok in Erlang. This is not entirely idiomatic, since some functions are generated that take for last argument the atom ok, but it is much nicer for the returned values.
OCaml let rec's inside of a module-level function definition should be reported as errors. It is common in OCaml to define a recursive helper function to traverse a value using a default accumulator that is hidden from the user, but we can ask the programmer to move this outside for now. See #13
OCaml allows ' (prime) to be used as part of variable names, so it's not uncommon to see x and x' and x''. The equivalent Erlang would be X, X1, and X2, primarily because the single quote is not a valid character to have in a name. See #15
OCaml allows shadowing of names, which is not supported in Erlang. so that let x = 1 in let x = x + 1 in x yields 2. The equivalent Erlang would crash. See #16

Module System

The goal here is to support the basic OCaml module system, with its nesting and aliasing to provide a flexible and safe way of structuring large amounts of code. Module interfaces should be allowed to dictate what is exported and what isn't.

Modules definitions -- we treat files as modules, and we can create nested modules. Nested modules will be flattened out into separate Erlang modules namespacing the module names. Modules without exports should generate no Erlang sources.
Module interfaces -- interface files will be respected and anything that is not exported there will not be exported in the generated Erlang source. This applies to both types and functions.
Module Declarations -- only functions will be allowed at the module level. Defining values should be an compilation error.

Type Language

These are the features of the type-language that I'm aiming to support and compile to Dialyzer specs.

Expressions

The following expressions will be supported on the OCaml side

Patterns

These are the patterns that we can use on the left side of match branches and function headers.

Foreign Interfaces

Support for external
- without a name override
- with name overrides

Advent of Code 2020

This is a list of bugs found and changes made during the Advent of Code 2020.

Day 1

Lists.foldl/3 and Lists.foldr/3 had the wrong FFI signature
Lists.foreach run_one days generated a function reference that crashed at runtime when using escript — turns out this is an issue with the way escript runs the code rather than with the generated code!

Day 2

Nothing new yet! This one ran just fine.

Day 3

...

Handling of Errors

After some consideration, I think I'd like error handling to be closer to Rust:

Everything is a result if it can fail
Or you consciously panic!

Panics are actual terminations, and would fit well with the supervision model of OTP. A Panic is a guaranteed process restart by the supervisor.

This would also mean forbidding all remnants of try / catch / exceptions from the language, and treating panic as a reserved word.

For interop with Elixir, stuff with a bang (!) is known to panic, stuff without it is expected not to. Human convention here.

For interop with Erlang, we don't have a clear indicator, but we should consider providing a wrapper to turn an Erlang error into a Result. This would help reuse some of the erlang module functions that just blow up on bad inputs (I'm looking at you *_to_*).

Forbid function redefinitions in an OCaml module

This OCaml source code:

let f 0 = true
let f _ = false

should result in a compilation error explaining that module functions can only be defined once, and multi-clause functions are achievable by the use of the function | pattern_1 -> expr_1 | pattern_2 -> expr_2 syntax.

This currently yields a function redefinition:

f(0) -> true.
f(_) -> false.

Which is not valid Erlang.

All of the modules are available as we translate them (see OCaml_to_erlang.Fun.mk_functions) but unfortunately we do not do this with every function.

Instead we are mapping over them so they are unaware of each other.

Changing this to a fold would allow us to carry around the list of functions that have already been defined, and we could check that any is being defined twice.

Should && and || translate to short-circuiting andalso and orelse?

To Reproduce

Create a file main.ml with

let print_int number = Io.format "~0tp~n" [ number ]

let print_and_return_true number =
  print_int number ;
  true

let main _ =
  print_int 0 ;
  false && print_and_return_true 1 ;
  true || print_and_return_true 2 ;
  print_int 3

Run command caramel compile main.ml && escript main.erl
See result:

Expected behavior
If the operators should be short-circuiting, the correct result would be:

0
3

just like when running this ocaml file:

(* let print_int number = Io.format "~0tp~n" [ number ] *)

let print_and_return_true number =
  print_int number ;
  true ;;

let main _ =
  print_int 0 ;
  false && print_and_return_true 1 ;
  true || print_and_return_true 2 ;
  print_int 3 ;;

main ()

$ ocaml main.ml
0
3

If the operators should be short-circuiting, I think the correct Erlang to generate here might be:

-spec main(_) -> ok.
main(_) ->
  print_int(0),
  false andalso print_and_return_true(1),
  true orelse print_and_return_true(2),
  print_int(3).

Environment (please complete the following information):

OS: Ubuntu Linux 18.04
Caramel version: 0.1
Erlang version: 23.2.2

Support expression sequences

Describe the bug
When combining multiple OCaml expressions with ( expr1 ; expr2 ) or begin expr1 ; expr2 end, the resulting Erlang code is "flattened" and can have different result than expected or not compile at all.

To Reproduce

Create a file main.ml containing

let print thing = Io.format "~0tp~n" [ thing ]

let main _ =
  print
    ( print "hej" ;
      true )

or

let print thing = Io.format "~0tp~n" [ thing ]

let main _ =
  print
    begin
      print "hej" ;
      true
    end

Run command caramel compile main.ml && escript main.erl
See error:

main.erl:10: function print/2 undefined

This happens because of this incorrect generated Erlang code:

% Source code generated with Caramel.
-module(main).

-export([main/1]).
-export([print/1]).

-spec print(_) -> ok.
print(Thing) -> io:format(<<"~0tp~n">>, [Thing | []]).

-spec main(_) -> ok.
main(_) -> print(print(<<"hej">>),
true).

Expected behavior
Erlang gets generated with:

-spec main(_) -> ok.
main(_) -> print(begin print(<<"hej">>), true end).

Environment (please complete the following information):

OS: Ubuntu Linux 18.04
Caramel version 0.1
Erlang version 23.2.2

v0.1 - OCaml libraries to manipulate Erlang

Summary

Caramel currently supports generating Erlang source files from an OCaml typed-tree, and it aims to support type-checking a subset of Erlang using the OCaml type-checker as well.

To do this, we need to have good support of the Standard Erlang language at 3 levels: parsing, AST, and printing.

Parsing for reading the Erlang source files. AST for manipulating the structures (and converting from and to the OCaml ASTs). Printing to generate somewhat idiomatic, human readable .erl sources.

The result is a library for working with Erlang sources that is agnostic of the purposes of Caramel itself, and can be used to experiment with the language, build AST-aware transformations, a pretty printer, etc.

While I don't have an intention of doing all of those right now, I'd like to enable the OCaml and BEAM ecosystem to be able to do these things.

Checklist

The library should contain:

a definition of an AST for Erlang,
a parser that follows the Standard Erlang grammar, and
2 printers: a debugging S-expr printer, and a Standard Erlang printer

Language Support

This list and its completeness is a work in progress.

Preprocessor Language

Macro definition
Macro application

Module Language

Module attributes (built-in and custom ones)
Type declarations
Function declarations

Type Language

Pattern Language

Ignore pattern (_) -- AST, Printer
Constant matches (against literals) -- AST, Printer
Tuple patterns -- AST, Printer
List patterns -- AST, Printer
- Empty list -- AST, Printer
- Head-tail patterns -- AST, Printer
- Exact list patterns -- AST, Printer
Map patterns -- AST, Printer
Pattern binding -- AST, Printer

Expression Language

Operators

Tooling support

We'll need things like:

Formatting

Calling caramel fmt should reformat all inputs.

Doc Generation

Calling caramel docs *.mli *.rei --output ./docs should create docs from those interface files.

Language Server

What I'm considering atm is bundling Merlin within Caramel, so we have control over the subset of the language being accepted.

Calling caramel language-server should be the equivalent to booting a Merlin daemon, and we'll need a way to create the right .merlin files as well, possibly through a separate command that can be properly guided by a build system (e.g. caramel language-server setup ./path/to/lib/a ./path/to/lib/b ...)

REPL

TBD

Fix windows dependencies cache on CI

Describe the bug
Windows "Install Dependencies" step is taking minutes.

Expected behavior
Windows CI builds should be able to restore the cache and the "Install Dependencies" step should not install dependencies when cache was successfully restored.

I think the cache with that key for Windows may be just stuck for some reason and it needs to get kicked by manually adding some v-2 string to the key, which will hopefully fix it for future builds.

Set up tests to fuzz the compiler with AFL

We need a way of testing that the compilation is successfully verifiable for a large space of OCaml programs. Talking to @rgrinberg he suggested to have a look into fuzzing.

Thankfully, there is enough support in the ecosystem to set up a pipeline for fuzzing.

Support parsing of Erlang expression sequences

The following Erlang code:

A, B, C.

Should parse to the equivalent AST:

open Erlang.Std.Ast_helper

Expr.sequence
   (Expr.ident (Name.var "A"))
   (Expr.sequence
      (Expr.ident (Name.var "B"))
      (Expr.ident (Name.var "C")))

So this looks like a degenerate case of let binding, where the expression A is bound to _. At the moment that's how this is being worked around, so _ = A, can be parsed, and is equivalent to A.

There is definitely work to be done in ./src/erlang/erl_parser.mly, and for inspiration we can look at the way the OCaml language parser is written to handle them: https://github.com/ocaml/ocaml/blob/trunk/parsing/parser.mly#L2075-L2084

How does one run the compiled erlang in the README example?

I've been able to compile the example in the README, but when I attempt to run the generated erlang, I'm told that the function process:make does not exist. It seems this module is provided as part of caramel's stdlib, but it doesn't appear to be in erlang's load path. Is it expected that I somehow make this available before running my program? Should these definitions instead be inlined during compilation?

Am I missing some step that makes this "just work"?

Thanks!

Allow for arbitrary module attributes

While working on #19, the error traces pointed out that the Process.spawn function was shadowing Erlang's spawn. This can be overcomed with a compiler directive, which we can't write at the moment, such as -compile({no_auto_import, [spawn/1]})..

Allowing these compiler directives is useful for:

redefining functions with standard names
allowing NIF on_load functions to be defined
adding more module-level attributes such as author or vsn

One way to fix this is to just use the compiler annotations where we can embed a string of raw Erlang, parse it, and merge it with the translated AST.

[@caramel.raw {|
-compile({no_auto_import, [spawn/1]}).
|}]

Definitely not pretty, but should be enough to unblock this while we figure out what the proper syntax for this is. Another alternative would be:

Caramel.ModAttributes.[
  compile (no_auto_import [ funref `spawn 1 ]); 
] [@caramel.module_attributes]

but building a DSL for this feels like a lot more work right now.

Safe and named fresh type variables

In this commit: e916333 I've replaced all instances of fresh type variables with any, in an attempt to unblock #19 -- after all those type signatures are not necessary for the Erlang code to compile and execute.

However, I do want to have this feature available, but it has one problem: a signature like unit -> 'a is valid in OCaml, but fun( () -> A ) is not in Erlang.

Thus, to really fix this, the commit above should be reverted, and during translation time we should check if the fresh variable name is actually already bound before in the same type expression, or if it is the only instance of it.

If it is the only instance, then we can use any() instead.

To fix this, we should begin by applying this patch:

-    | Tvar None -> Type.any
+    | Tvar None -> Type.var (Name.var (Type_var_names.find type_expr))

At the end of this module we should map over the list of types and for every function type we should have a data structure with all the data we need in a fairly straightforward form:

Type_function { tyfun_args; tyfun_return; }

And if the return type is a type variable name that was not bound in the arguments, we can loosen it up with Type.any instead.

Function undefined for many externals from the Core module

Describe the bug
When using many of the automatically included functions from the Core module, the Caramel code compiles, but Erlang code does not compile with function x/N undefined errors.

To Reproduce

Create a file main.ml with

let print thing = Io.format "~0tp~n" [ thing ]

 let main _ =
  print_int 1 ;
  print_string "asdf" ;
  1 |> print ;
  print @@ 1 ;
  print (succ 1) ;
  print (pred 1) ;
  print (1 lsr 2) ;
  print (1.0 ** 2.0) ;
  print (expm1 2.0) ;
  print (log1p 2.0) ;
  print (hypot 2.0 2.0) ;
  print (copysign 2.0 2.0) ;
  print (mod_float 2.0 2.0) ;
  print (frexp 2.0) ;
  print (modf 2.0) ;
  print (int_of_char 'x') ;
  print (ignore 2) ;
  print (int_of_string "1") ;
  print (float_of_string "1.0") ;
  print (fst (1, 2)) ;
  print (snd (1, 2)) ;
  raise Exit ;
  raise_notrace Exit ;
  ()

Run command caramel compile main.ml && escript main.erl
See errors:

$ escript main.erl
[...]
main.erl:18:9: function '**'/2 undefined
main.erl:14:3: function '@@'/2 undefined
main.erl:22:9: function copysign/2 undefined
main.erl:19:9: function expm1/1 undefined
main.erl:29:9: function float_of_string/1 undefined
main.erl:24:9: function frexp/1 undefined
main.erl:30:9: function fst/1 undefined
main.erl:21:9: function hypot/2 undefined
main.erl:27:9: function ignore/1 undefined
main.erl:26:9: function int_of_char/1 undefined
main.erl:28:9: function int_of_string/1 undefined
main.erl:20:9: function log1p/1 undefined
main.erl:17:9: function lsr/2 undefined
main.erl:23:9: function mod_float/2 undefined
main.erl:25:9: function modf/1 undefined
main.erl:16:9: function pred/1 undefined
main.erl:11:3: function print_int/1 undefined
main.erl:12:3: function print_string/1 undefined
main.erl:32:3: function raise/1 undefined
main.erl:33:3: function raise_notrace/1 undefined
main.erl:31:9: function snd/1 undefined
main.erl:15:9: function succ/1 undefined
main.erl:13:3: function '|>'/2 undefined
escript: There were compilation errors.

Expected behavior
When Caramel code compiles, Erlang code compiles. It's possible that not all of the above functions are necessary to be implemented for now, but I think it wouldn't be too hard to get them working?

Environment (please complete the following information):

OS: Ubuntu Linux 18.04
Caramel version: 0.1
Erlang version: 23.2.2

Support parsing expression sequences

At the moment expression sequences can not be parsed directly unless they are bind patterns.

A work around for this is to use the ignore pattern:

f() ->
  _ = g(),
  _ = h(),
  ok.

Verify inferred typedtree is consistent with the original typedtree

One we can parse in the entire Erlang AST that we can generate from caramelc, the translation should end up in an OCaml Parsetree that we can typecheck successfully.

Even if we do typecheck correctly, we need to make sure that the inferred types are isomorphic to the source types used to generate the Erlang code to begin with.

So that the following identity law holds: types_of(parse(compile(ml))) = types_of(ml).

This may be doable at the source level as well.

As an example of the flow:

(* compile 2 modules *)
$ caramelc compile a.ml b.ml

(* perform an obviously wrong verification *)
$ caramelc verify a.erl b.ml
ERROR: inconsistent modules

(* perform a verification that should work *)
$ caramelc verify a.erl a.ml
OK

I need to do some thinking here, but this should be the intended flow.

Function references have the wrong arity

After the last refactor of the Erlang printer, we no longer look up the arity of a function at print time, and thus the default arity that's set to zero here is always printed out.

So this OCaml:

let f g = g ()
let rec g _ = f g

that should compile to this Erlang:

f(G) -> G().
g(_) -> f(fun g/1).

Compiles instead to this erlang:

f(G) -> G().
g(_) -> f(fun g/0).

Note the arity of g is obviously 1 since it takes one parameter, but the function reference does not have this information anymore.

To fix this we should change the way that function references are being created here and use the identifier found to lookup the arity of the function.

This could be done by:

Adding a Hashtbl.t from function name's to their a copy of the Erlang AST values
Carrying around an assoc. list of the functions that we've transformed so far.

The second approach would be a little cleaner (considering we've gone so far without any mutable state in the translation), and it happily relies on the assumption that every function will be defined before it is used.

The first approach might be easier to hack on.

Support record updates

The following OCaml currently does not compile:

type t = { b: int; c: bool }

let f () =
  let a = { b=1; c=true } in
  { a with b=2 }

Whereas it should compile to:

f() ->
  A = #{ b => 1, c => true },
  A#{ b := 2 }.

To get this done, we need to look into the extended_expression of the Texp_record value and if it is not None, then use that with the already existing Expr.map_update function from the Ast helpers.

Refactor Erlang AST and Parser to carry location information

In order to provide good error messages whenever there is a syntax error or a type error, we need to thread and carry this information on the AST.

leostera / caramel Goto Github PK

caramel's Introduction

🍬 Caramel

Feature Highlights

Install

caramel's People

Contributors

Stargazers

Watchers

Forkers

caramel's Issues

Other notes

Summary

Checklist

Usability and Errors

Language Support

Pattern Language

Expression Language

Libraries

Summary

Checklist

FFIs

Runtime

Other Notes

Summary

Checklist

Idioms

Module System

Type Language

Expressions

Patterns

Foreign Interfaces

Day 1

Day 2

Day 3

Summary

Checklist

Language Support

Preprocessor Language

Module Language

Type Language

Pattern Language

Expression Language

Operators

Formatting

Doc Generation

Language Server

REPL

Recommend Projects

Recommend Topics

Recommend Org