lfe / try

The containerised service, REST API, and website for try.lfe.io (WIP!)

LFE

LFE, Lisp Flavoured Erlang, is a lisp syntax front-end to the Erlang compiler. Code produced with it is compatible with "normal" Erlang code. An LFE evaluator and shell is also included.

Building

To compile LFE, simple clone it and compile:

$ git clone https://github.com/lfe/lfe.git
$ cd lfe
$ make compile

LFE requires Erlang be installed on the system and that the erl binary is in $PATH.

Running the Tests

To run the full suite of tests for LFE, simply use the following:

make tests

Installation

Should you wish to have LFE available system-wide, you can run the following make target:

$ make install

By default this will create the programs lfe, lfec, lfedoc and lfescript in /usr/local/bin. This can be changed by defining the make variable PREFIX to point to the desired parent directory.

Note that the install target will also install the LFE man pages in the appropriate $(PREFIX)/share/man/man* directories. This can be changed by defining the make variable MANINSTDIR to point to the desired top man directory.

So:

$ make install PREFIX=/Users/rv/ MANINSTDIR=/Users/rv/man

will put the programs in /Users/rv/bin and the man pages in the /Users/rv/man/man* directories.

REPL

If you're running LFE from a git clone working dir, you can start the REPL like so after compiling:

$ ./bin/lfe

Erlang/OTP 26 [erts-14.0.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] [dtrace]

   ..-~.~_~---..
  (      \\     )    |   A Lisp-2+ on the Erlang VM
  |`-.._/_\\_.-':    |   Type (help) for usage info.
  |         g |_ \   |
  |        n    | |  |   Docs: http://docs.lfe.io/
  |       a    / /   |   Source: http://github.com/lfe/lfe
   \     l    |_/    |
    \   r     /      |   LFE v2.1.5 (abort with ^G)
     `-E___.-'

lfe>

If you have installed LFE, then you may start the REPL from any location:

$ lfe

Likewise, you may run an LFE shell script in the same style as shell scripts with:

$ ./bin/lfe script-name script-arg-1 ...

or

$ lfe script-name script-arg-1 ...

Usage

The docs site has several places to explore that will show you how to start using LFE. However, here's a quick taste:

• start up an LFE REPL as demonstrated above
• then, do something like this:

lfe> (* 2 (+ 1 2 3 4 5 6))
42
lfe> (* 2 (lists:foldl #'+/2 0 (lists:seq 1 6)))
42

Docker Support

LFE now supports Docker. To get started, simply do the following, once you have Docker set up on your machine:

$ docker pull lfex/lfe

Alternatively, you could build the image yourself:

$ cd lfe
$ docker build .

To bring up the LFE REPL:

$ docker run -it lfex/lfe

Documentation

• Quick Start
• LFE user guide
• version history

Files with more technical details:

• lfe.txt
• lfescript.txt
• lfe_bits.txt
• lfe_clj.txt
• lfe_comp.txt
• lfe_docs.txt
• lfe_gen.txt
• lfe_io.txt
• lfe_lib.txt
• lfe_macro.txt

If you would like to make changes to the LFE documentation and then regenerate the docs, you'll want to read the instructions here:

• Updating LFE Documentation

Join the Community

LFE on Slack, join by requesting an invite here

LFE Forum - Erlang Forums

Maintainers

Cutting Releases

Steps:

1. Update the version in src/lfe.app.src
2. Create the release tags
3. Create a release on Github
4. Publish to hex.pm

Once the app.src has been updated with the version for the release, you can create and push the tags (to Github) with the following:

make tags

That will create the number-only version as well as the "v"-prefixed version.

For now, the process of creating a release on Github is manual:

1. Go to https://github.com/lfe/lfe/releases
2. Click "Draft new release"
3. Select the correct tag from the drop-down "Choose a tag"
4. Click "Generate release notes"
5. Click "Publish release"

Lastly, to publish LFE to hex.pm, you need to have rebar3 installed on our system and an entry for the hex plugin in your system rebar.config file. With that in place, publish a new release to hex.pm requires only the following:

make hex-publish

try's Issues

Create custom (help) output

We shouldn't actually use the (help) from the LFE REPL. In addition to that having a couple of technical hurdles that would slow is down, it wouldn't even be that useful. This is going to be a very different REPL than the LFE REPL: there's a lot of stuff you simply won't be able to do here.

As such, we should create our own (help) (of course, modelled on the LFE one!)

Handle LFE bytes in REST handler

Task:

• Receive bytes in POST route
• Convert bytes to string
• Build list of all mod:func occurrences in string
• If any of those are not in the allowed/safe list of mod:funcs, reject the entire payload out of hand, returning an appropriate HTTP error and JSON error payload

Depends upon:

• #21

Create md5 hash from username

Upon clicking "submit" in the "login" modal, these things should happen:

• create a JS function which can POST to the REST server (can go in trylfe.js)
• username is posted to new "/login" URL
• function is called which extracts username from arg-data (see trylfe-rest.lfe for an example)
• implement this function in trylfe-rest.lfe
• also extract referrer IP address/hostname from arg-data (this may require pulling in a YAWS record from one of YAWS' .hrl files)
• calculate an md5 based upon these two data

Implement per-session environments

Whenever a new client IP address issues an eval request, a new session would be created and managed (see #8). Each session would get its own environment, allowing try.lfe.io users and *.lfe.io applications to define variables, functions, etc., and call those in their session.

Possible implementation: we could store a user's LFE environment in mnesia, key'ed off the md5 of host. Then store function and variable definitions, and perform lookups on a per-session basis.

This would mean that 100s or 1000s of users wouldn't have as big an impact on the system; we'd just need to set up mnesia.

Tasks (these are old and need to be revisited, per the description edits/updates):

• define record with "hash" key and "data" key (for now, just use the md5 of the IP address; when #15 and #16 are done, we can use the username too)
• define ETS table
• add wrapper functions for insert, get, delete, etc.
• query command state table when eval'ing expr from user
• save command state in table after successfully eval'ing expr from user for re-use in the next call; this save should over-write whatever data was associated with the user/ip hash key
• see if function definitions work
• see if +++ and *** work
• every time a user executes a command that goes back to the server, the expire time should be reset

Style the pages

For any static content we'll be using. At the very least, the index page.

Should match either the 1.0 or 2.0 docs.lfe.io style.

Deployable Proof of Concept

Features/Tasks:

• #12 - Create Dockerfile for use on the LFE domain
• Choose a deployment platform (e.g., AWS)
• Deploy (but only for short durations)
• Test like crazy
• Iterate until stable

Figure out running Docker behind HTTP

try.lfe.io:80 will go to a public-facing YAWS server; it will then forward requests to the docker instance on localhost running on a port of its own. This is essentially a ticket for exploring URL redirects in YAWS configurations.

Convert existing REST server to Nova

Tasks:

• Convert routes
• Add Nova dep and remove YAWS dep + make targets
• Add Nova boilerplate
• Convert current GET impl to use POST instead

Resources:

• http://www.novaframework.org/
• https://github.com/novaframework/nova

Implement session manager

The shell manager needs to create an LFE shell for each "user" (client IP address). The following capabilities would be needed:

• The ability to dynamically start a new "shell" process (maybe not a real LFE shell ...) per "user"
• A means of tracking how long a user has been active / idle
• #19 - The ability to prune idle processes (stale sessions)
• A way of recording the number of any "bad actor" calls made from any given IP address
• A way of recording the number of connection requests per second/minute/hour/etc made by any given IP address
• Putting "shell creation" requests into a "queue" with bad IP addresses getting put into exponentially longer waiting times before connection after recorded bad actions
• Watch for errors in "shells" (sessions) and kill processes with errors, transparently initiating a session re-creation the next time that IP address asks for an LFE code eval
• Track overall manager runtime and the ability to request self-restart after a fixed time

Note: the need for this might be obviated by the option proposed in ticket #14.

Create a modal dialog when the page loads

• have one input field for username
• provide a short help message for entering the name (this feature could confuse users, so a clear statement of purpose would be good)
• provide a "submit" button in the dialog
• have the modal be dark, but transparent
• have the dialog/modal go away after submitting
• save the username to an ETS table
  • create a record for username and date created
  • create db code for user operations
  • create a /login REST POST call
• set the username in the cookie
• update all in-session REST calls to also pass the username from the cookie data (thus any functions that now use query parameters will need to be updated to pass the user name to the functions they call)
• support logging out
  • add a "log out" link in the top bar
  • add /logout REST DELETE route
  • add functionality for destroying the trylfe cookie
  • add functionality for removing the user from the ETS table (the username will be passed in the query params; the arg data can be used to look up the user's IP address and we will assume that these uniquely identify the user
• update the prompt to be <username>|lfe>
  See ticket #16 for what should happen after clicking "submit".

Update existing try-lfe JS to use new Nova REST API

Depends upon #21

Prototype simple end-to-end example

Let's get something working locally, just enough to pass data from one component to the next. Components:

• A container
• A REST server (written in LFE)
• A JS client
• An mdbook with example LFE code on one of its pages

Feature-level work for proof-of-concept:

• #10 - a deployment approach and proof of concept (e.g., a container that could be deployed to AWS using ECS)
• #21 - a service, protocol, and proof of concept that could be used across *.lfe.io and from JS (e.g., REST & HTTP)
• #22 - update existing try-lfe JS to make sure we're moving in a good/supportable direction
• #20 - some JS hacking in mdbook's book.js to make sure we can properly render results from the REST service

We should be able to prove out a full set of round trips locally using Docker ... might have to do some hacking / tweaking for local CORS or cross-site scripting?

Investigate the level of effort for adding support for an "LFE eval service" in mdbook

https://rust-lang.github.io/mdBook/format/mdbook.html#inserting-runnable-rust-files

I don't think we'd need to recompile mdbook; we might just be able to:

1. reuse {{#playground file.rs}} and
2. maintain a modified book.js that calls the LFE service instead of the Rust playground

Need to look at the details of how mdbook reads the results:

• https://github.com/rust-lang/mdBook/blob/master/src/renderer/html_handlebars/hbs_renderer.rs
• https://github.com/rust-lang/mdBook/blob/master/tests/rendered_output.rs

Questions:

• Does mdbook do any file extension checks? Iow, will {{#playground file.lfe}} work?

Implement REST server

Try running in a docker instance

Create a Dockerfile for try.lfe.io

Then publish on DockerHub!

Build out skeleton REST/YAWS project

See if we can run LFE in restricted mode

Style the console

Fine-tune the tutorial text

Ship it!

Prepare for secure deployment

Features/tasks:

• #9 - Handle LFE bytes in REST handler
• #4 - No unsafe functions
• #8 - Support session management
• #14 - Implement per-session environments

Add a long-running processes that regularly checks for stale user sessions

• Try just using timer:apply_after/4
• Add a function (ets query) that gets all user rows older than X
• Add a function for bulk removal of user rows

Disallow running unsafe functions / allow only running safe functions

Proposed tasks:

• In actual source code, create a list of all allowed Erlang and LFE mod:func combinations
• During startup, read these into memory: as a list of strings (will this be needed), as a list of module atoms, and as a list of func atoms
• From the list of all mod:func occurrences in a POSTed payload:
  • extract the module and attempt to convert to atom with erlang:list_to_existing_atom -- if badarg, this isn't allowed
  • do the same for the function

The consuming REST functions can then return an appropriate HTTP error and JSON error payload upon error conditions.