clojerl / clojerl

Clojure for the Erlang VM (unofficial)

clojerl

Clojure implemented on the Erlang VM.

Building

Building clojerl requires Erlang/OTP 21+ and rebar3.

git clone https://github.com/clojerl/clojerl
cd clojerl
make

On Windows:

git clone https://github.com/clojerl/clojerl
cd clojerl
rebar3 clojerl compile

Getting Started

Documentation and Resources

There is more information regarding Clojerl in clojerl.org, where you can find what features does Clojerl include and how it differs from Clojure.

Online REPL

To try it out and get a sense of what you can do, you can visit Try Clojerl.

Docker REPL

To quickly try out clojerl via docker you can make use of the docker image like so:

docker pull clojerl/clojerl
docker run -it clojerl/clojerl

Then you should be able to see the prompt:

Clojure 0.6.0
clje.user=>

Local REPL

Running make repl (on Windows first run rebar3 clojerl compile and then bin/clje.bat) will start the REPL and show its prompt:

Clojure 0.6.0
clje.user=>

From the REPL it's possible to start evaluating Clojure expressions:

clje.user=> (map inc (range 10))
(1 2 3 4 5 6 7 8 9 10)
clje.user=> (doc map)
-------------------------
clojure.core/map
([f] [f coll] [f c1 c2] [f c1 c2 c3] [f c1 c2 c3 & colls])
  Returns a lazy sequence consisting of the result of applying f to
  the set of first items of each coll, followed by applying f to the
  set of second items in each coll, until any one of the colls is
  exhausted.  Any remaining items in other colls are ignored. Function
  f should accept number-of-colls arguments. Returns a transducer when
  no collection is provided.
nil
clje.user=> (doc inc)
-------------------------
clojure.core/inc
([x])
  Returns a number one greater than num.
nil
clje.user=>

Code Examples

There are some very basic examples in the scripts/examples directory. These are meant to be references on how special forms in Clojure on the BEAM are used and how they sometimes differ from Clojure JVM.

Web Application Example

For a very basic example of a web project please check the example-web-app repository.

Building Your Own App

The build tool for Clojerl is the rebar3_clojerl plugin. rebar3 is the official build tool in the Erlang community.

The plugin provides helpful commands to:

• Build a basic directory scaffolding for a new project
• Compile
• Run tests
• Start a REPL

For more information on how to use this plugin please check the documentation in rebar3_clojerl.

Rationale

Erlang is a great language for building safe, reliable and scalable systems. It provides immutable, persistent data structures out of the box and its concurrency semantics are unequalled by any other language.

Clojure is a Lisp and as such comes with all the goodies Lisps provide. Apart from these Clojure also introduces powerful abstractions such as protocols, multimethods and seqs, to name a few.

Clojure was built to simplify the development of concurrent programs and some of its concurrency abstractions could be adapted to Erlang. It is fair to say that combining the power of the Erlang VM with the expressiveness of Clojure could provide an interesting, useful result to make the lives of many programmers simpler and make the world a happier place.

Goals

• Interoperability as smooth as possible, just like Clojure proper and ClojureScript do.
• Provide most Clojure abstractions.
• Provide all Erlang abstractions and toolset.
• Include a default OTP library in Clojerl.

Personal Goal

Learn more about Erlang (and its VM), Clojure and language implementation.

This project is an experiment that I hope others will find useful. Regardless of whether it becomes a fully functional implementation of Clojure or not, I will have learned a lot along the way.

QAs

What is Clojerl?

Clojerl is an experimental implementation of Clojure on the Erlang VM. Its goal is to leverage the features and abstractions of Clojure that we love (macros, collections, seq, protocols, multimethods, metadata, etc.), with the robustness the Erlang VM provides for building (distributed) systems.

Have you heard about LFE and Joxa?

Yes. LFE and Joxa were each created with very specific and different goals in mind. LFE was born to provide a LISP syntax for Erlang. Joxa was mainly created as a platform for creating DSLs that could take advantage of the Erlang VM. Its syntax was inspired by Clojure but the creators weren't interested in implementing all of Clojure's features.

Aren't the language constructs for concurrency very different between Clojure and Erlang?

Yes, they are. On one hand Clojure provides tools to handle mutable state in a sane way, while making a clear distinction between identity and state through reference types. On the other, concurrency in the Erlang VM is implemented through processes and message passing. The idea in Clojerl is to encourage the Erlang/OTP concurrency model, but support as many Clojure constructs as possible and as far as they make sense in the Erlang VM.

But... but... Rich Hickey lists here some of the reasons why he chose not to use the actor model in Clojure.

That is not a question, but I see what you mean :). The points he makes are of course very good. For example, when no state is shared between processes there is some communication overhead, but this isolation is also an advantage under a lot of circumstances. He also mentions here that building for the distributed case (a.k.a processes and message passing) is more complex and not always necessary, so he decided to optimise for the non-distributed case and add distribution to the parts of the system that need it. Rich Hickey calls Erlang "quite impressive", so my interpretation of these writings is that they are more about exposing the rationale behind the decisions and the trade-offs he made when designing Clojure (on the JVM), than about disregarding the actor model.

Will Clojerl support every single Clojure feature?

No. Some of Clojure's features are implemented by relying on the underlying mutability of the JVM and its object system. The Erlang VM provides very few mutability constructs and no support for defining new types. This makes it very hard or nearly impossible to port some features into Clojerl's implementation.

Can I reuse existing Clojure(Script) libraries?

Yes, but they will need to be ported, just like for ClojureScript. In fact, most of Clojure's core namespaces were ported from the original .clj files in the Clojure JVM repository.

Discussion

Join the conversation in the Clojerl mailing list or in the #clojerl Slack channel!

You can also find news and updates through @clojerl. Or if you have any questions you can find me @jfacorro or lurking on Clojure's and Erlang's mailing lists.

Any feedback, comment and/or suggestion is welcome!

clojerl's Issues

Use line and column information in all errors shown, when possible

clojerl.Var: maybe change internal representation of ns and name from symbol to binary

README

loop* & recur

Integrate with Travis CI

Possible when the repo goes public.

clojure.core/require

The function would cause the required module to be compiled and loaded in the runtime. If the compile-time Env is not modified then issue #53 should be implemented with this one so that a namespace can require another namespace and just use the vars defined in it.

lazy-seqs (?)

Add file information to errors

HOF that return anonymous functions not working

The following compiles but doesn't work:

(def comp
  (fn* [& fs]
       (fn* [& xs]
            (clojure.core/apply (clojure.core/first fs) xs))))

data structures: look into implementing vectors and/or sets with maps/tuples

Read http://www.cs.cmu.edu/~rwh/theses/okasaki.pdf once and for all!

Symbols that get resolved to an erl_fun might actually be vars that have not been defined yet

If the var contains a function, when the emitted erl_fun function value is applied or the emitted application expression is executed, it might result in an undefined function error. Or (even more confusing for the user) if the var defined after is a function with a variadic arity, then the application emitted has as many arguments as the ones provided in the original form, which might not match the ones the existing function should receive for its variadic application.

If the var contains a value then it will most likely result in an undefined function.

general: change keyword representation from clojerl.Keyword to plain Erlang atom

An Erlang atom is a lot closer to a Clojure keyword than to a symbol since their usage is a lot more common in Erlang as keys to maps and values used to tag data. The trade-off is that it won’t be possible for keywords to have any metadata associated to them. Additionally the Erlang atom undefined will be mapped to Clojure’s nil and vice versa.

Clojure symbols will still be a boxed type so that they can provide support for metadata.

Erlang atoms are a viable option for keywords and/or symbols.

The advantage of using atoms is that they are maintained in the atoms table so each time a symbol is created it would be equivalent to Java’s String.intern() (although it seems Clojure JVM is not interning the string for symbols and Keywords anymore).

The disadvantage is that a lot of name munging is involved when manipulating symbols and keyword. If atoms are used, a bunch of conversions between atom to binary and or binary to atom will be needed. For example, if the keyword clojure.core/str is represented by an atom, extracting the namespace and/or name from this keyword would involve converting to binary and splitting by the first “/” character found.

compiler: provide options such as the output dir and compiler flags

Compiler tests are failing because the output dir is hard-coded, by fixing this issue we could provide the output dir as an option.

If a var is not in the Env when analyzing try to get it from the module's attribute

analyzer: tests

Macros defined in a namespace can't currently be used in the same ns

Since all the module is now being compiled at once, the first time it is compiled all vars that are macros can't be executed because they don't exist yet. A possible solution would be to partition the forms in a module every time a macro is found.

analyzer: check implementation for analyze_seq

Compare the Clojerl implementation to tools.analyzer implementation, there might be some things missing.

Tuples should implement ISequential

emitter: let*

analyzer: add `context` to env and `top_level` to expressions when applicable

Remove erl_fun reader dispatch

The analyzer is currently returning an erl_fun node for all symbols that can't be resolved to locals or existing vars so it doesn't really make sense to have a specific reader that returns an Erlang fun.

Declare dummy ISequential behavior to avoid warning

Add test to compile .clj files in the priv folder

reader: keep track of line and column information

Var: dynamic bindings

Since we have no threads in Erlang the value would actually be associated to a process. This sounds like the process dictionary is a good fit for dynamic values to live in. We can reuse the clj_scope data structure to handle nested bindings.

def: add metadata from name symbol to var

analyzer: let*

analyzer: allow pattern matching wherever there is a binding form

This is more complex than just allowing arbitrary expressions as binding forms.

A possible approach would be translating Clojure's destructuring syntax to Erlang's valid patterns. We could cover the most basic patterns (i.e. lists and maps) this way and extend the destructuring syntax for all the Erlang pattern syntax not covered.

Another option would be to have both separately. Destructuring would be used in the default Clojure macros (i.e. defn, let, loop) and new equivalent macros would allow the usage of Erlang patterns when binding values (i.e. defn.e, let.e, loop.e).

analyzer: if

Integration with Travis CI

When the repo goes public.

analyzer: loop*

Change protocol implementation to avoid the generation of too many modules

The current approach forces the generation of a module for each implementation of any given protocol. The advantage of this is that it allows extending a data type without having to modify its module.

It could be changed so that the code for the protocol mechanism looks for the implementation in the data type module and then if it's not there then look for an implementation module. The problem with this approach is that for two different protocols with functions that share their names, there could only be a single implementation, unless the function name in the module has the protocol's name as a prefix. The prefix should actually need to be the protocol's fully qualified name to avoid collision between two protocols with the same name.

For example the following protocol generates a module clojerl.Counted with a callback count/0:

(ns clojerl)
(defprotocol Counted
  (count []))

If we want to implement the protocol for the type List, the current approach would mean generating a module clojerl.List.clojerl.Counted that implements the behavior. An alternative approach is to consolidate all implementations in the same module, either the protocol's or the type's module. This would mean loading the module's code and inserting the implementation function, which has challenges of its own.

Define clojure.core/defn- and take into account if a var is private

• When emitting check if the def'ed var is private and if so don't export it.
• When invoking it check if it is private and if so don't make a fully qualified function call.
• derefing the value of a private var should only work in the module the var is defined, but this won't work since the whole process of derefing involves other modules already (i.e. clojerl.Var).

analyzer: def

analyzer: do

reader: implement missing reader dispatches

The following reader dispatches are not implemented:

• syntax-quote (i.e. `sym)
• arg (i.e. %, %nand%&`)
• fn (i.e. #())
• eval (i.e. #=1)
• unreadable form
• cond (i.e. #?(:clj 1 :cljs 2))

emitter: invoke - figure out how functions and funs can map to defn and fn

One of the main issues is that Clojure functions can have a variadic number of arguments. This presents the challenge of detecting functions whose last argument is variadic while keeping interop as simple as possible.

In the function call (f 1 2 3), the symbol f could resolve to a var, local binding or nothing. If it's a var and its value is a clojerl function then it should be invoked depending on whether it's the variadic arity or not.

Analyze and emit var

Either emit the compile-time information of the var or get the run-time information for it.

Metadata: what happens if you use a symbol with metadata as a key in a map?

With the current metadata implementation, the same symbol with different metadata won't be mapped to that key 😞.

Tests for the emitter

analyzer: fn*

reader: implement reader tags

Test all data types

The idea would be to test their expected behavior implemented through the protocols they implement.

Value Types:

• clojerl.Boolean
• clojerl.String
• clojerl.Integer
• clojerl.Float
• clojerl.Nil

Named Types:

• clojerl.Symbol
• clojerl.Keyword
• clojerl.Var

Collection Types:

• clojerl.Map
• clojerl.List
• clojerl.Set

Erlang Types:

• clojerl.erlang.List
• clojerl.erlang.Map
• clojerl.erlang.Tuple
• clojerl.erlang.Fn

reader: tuples #[]

try..catch..finally

Macros

Avoid multiple passes for parse_def and parse_fn

Simple REPL

analyzer: quote

reader: should be able to read one form at a time

And fold over forms maintaining the Env. This is necessary to correctly keep track of the current namespace based on the analyzed/emitted/evaled forms so far.

The reader should provide a way to read one form and return the remaining unconsumed input.