hyperpape / mako

Java DSL that lets you write high level code, compile it to JVM ByteCode and execute it at runtime

Mako

Overview

Mako is a Java code generation library that produces Java classes/bytecode.

Mako features a high level DSL for defining methods/classes, with a focus on supporting complex or algorithmic code (loops, conditionals, switches, etc).

Mako supports generating and loading classes at runtime, or precompiling .class files that can be packaged with an application.

Classes implemented via Mako are fully interoperable with standard Java code. That is, they can call regular objects and methods and implement interfaces or inherit from classes defined in regular code.

To illustrate, here is how to specify the recursive fibonacci method:

        var classBuilder = new ClassBuilder("fibonacci2", "");
        classBuilder.addEmptyConstructor();
        var method = classBuilder.mkMethod("fibonacci", List.of("I"), "I", new GenericVars("x"));
        method.cond(eq(read("x"), 0)).withBody(List.of(
                returnValue(1)));
        method.cond(eq(read("x"), 1)).withBody(List.of(
                returnValue(1)));
        method.returnValue(plus(
                CodeElement.call("fibonacci", Builtin.I, thisRef(), sub(read("x"), 1)),
                CodeElement.call("fibonacci", Builtin.I, thisRef(), sub(read("x"), 2))));
        Class<?> cls = new ClassCompiler(classBuilder).generateClass();
        o = cls.getDeclaredConstructors()[0].newInstance();
        System.out.println(o.getClass().getDeclaredMethod("fibonacci", int.class).invoke(o, 5)); // prints 8

Status

This project is pre version 0.1 and has no users as of yet. Some important constructs are missing. My current approach is to reimplement needle using mako, which is uncovering missing language constructs.

Experiments using mako, feedback/suggestions, and bug reports are welcome.

Language

Semantically, Mako has many concepts in common with Java--its constructs compile straightforwardly to Java bytecode--but differs in a few ways. In general, it tends to implement less functionality than Java itself (i.e. no autoboxing, and restricted forms of looping).

Syntactically, Mako uses prefix notation (operators like eq or plus come before their arguments), like a lisp, but freely mixes method calls and static methods, according to what I find most natural to write. Many language constructs can be created using static methods, or fluent method calls off of the appropriate object (either a Method or CodeElement or the proper type).

Basic Expressions and Statements

Variables have function scope. They are defined by adding them to a Vars object.

Vars methodVars = new GenericVars("x", "y", "z");
// alternately
methodVars.add("a");
methodVars.add("b");

Variables are always non-final, and may be read/set. Setting a variable is a statement (does not produce a value).

method.set("x", literal(1));
method.get("x");

There are no compound assignment operators. You must read a variable, do addition, then set the variable.

method.set("x", plus((read("x"), literal(1))));

Arithmetic and equality expressions are written with the operator first:

plus(read("x"), literal(1));
sub(read("x"), 1); // calling literal is usually unnecessary in arithmetic contexts
mul(32, 2.5);
div(32, 2.5);
mod(8, 2);

There are standard comparisons that produce booleans:

gt(2, 1); // true
gte(1, 1); // true
lt(0, 1); // true
lte(0, 0); // true
eq(1, 1); // true
neq(1, 2); // true

Implicit numeric conversions for primitives are supported, but all other conversions must be explicit (no auto-boxing).

// define a method returning a long
Method method = new Method(TEST_METHOD, List.of(), Builtin.L, null);
method.returnValue(1);

Method calls put the method name first, followed by the receiver and arguments. There is a separate method to produce a static call:

call("toString", ReferenceType.of(String.class), read("myString"));
callStatic(CompilerUtil.internalName(Integer.class), "valueOf", 
        ReferenceType.of(Integer.class), literal(0));

The construct method is used for constructing new object instances.

construct(ReferenceType.of(Integer.class), literal(16);

Arrays

Standard array operations are supported. Array types can be constructed with ArrayType#of().

newArray(5, Builtin.I); // equivalent to int[5];
// note that we use Builtin.I, not ArrayType.of(Builtin.i), which would create int[][5];
arraySet(arrayExpression, 0, "abc"); // sets index 0 to "abc"
arrayRead(arrayExpression, 0); // retrieves the value at index 0;
arrayLength(arrayExpression); // equivalent of arrayExpression.length;

Note that arraySet is a statement, not an expression.

Conditionals

Conditionals are created using cond, then withBody. else if is done with the elseif method on a conditional. Else blocks can be added with orElse.

method.cond(eq(read("i"), 3))
.withBody(List.of(returnValue(3)))
.elseIf(eq(read("i"), 4)).withBody(list.of(returnValue(4)));
.orElse().withBody(List.of(returnValue(5)));

Loops

Mako has only while loops for now.

        var method = new Method(TEST_METHOD, List.of(), Builtin.I, new GenericVars("a"));
        method.set("a", 1);
        method.loop(lt(read("a"), 5),
                List.of(set("a", plus(read("a"), 1))));
        method.returnValue(read("a"));

Loops support break and continue, spelled escape and skip.

method.set("a", 1);
        method.loop(lt(read("a"), 5),
                List.of(set("a", plus(read("a"), 1)),
                        escape()));

method.loop(lt( read("a"), 5),
    List.of(set("a", plus(read("a"), 1)),
        skip()));

Defining Methods

Vars vars = new GenericVars("a", "b", "c", "d");
// Arguments to the method constructor are name, arguments, return type, the variables
Method method = new Method(TEST_METHOD, List.of(Builtin.I), Builtin.I, vars);
method.set("d", 2);
method.returnValue(read("d"));

Defining Classes

To define a class, create a ClassBuilder, add methods to it, then pass it to a ClassCompiler.

        // arguments are className, package, superclass, interfaces
        ClassBuilder cb = new ClassBuilder("TestClass", "", "java/lang/Object", new String[]{});
        cb.addEmptyConstructor();
        Method method = cb.mkMethod("foo", List.of("I"), "I", new GenericVars());
        Class<?> cls = new ClassCompiler(cb).generateClass();

Building

The compiler requires Java 11. Builds with maven. The generated classes should work with Java 8.

Mako, ByteBuddy and ASM

Mako is built on top of ASM, and provides a subset of ASM's capabilities. In particular, Mako only provides an API for generating classes from scratch, it does not provide an API for transforming them. What Mako attempts to add is a concise, higher level API for generating algorithmic code at runtime. For an example where this can be useful, see https://github.com/hyperpape/temporalformats/.

Compared to ByteBuddy, Mako is more verbose for simple use-cases, but handles some use-cases where the ByteBuddy based solution would be to use ASM.

mako's Issues

Support string concatenation

Right now, there’s no support other than constructing a string builder, which is quite verbose in mako. Since string concatenation is such a common operation, having a non-StringBuilder based api would be a significant quality of life win.

Observation: this doesn’t intrinsically require language support—static methods could emit string builder calls with opportunity to later refactor the underlying code to emit invokedynamic instructions the way javac does.

Redundant bytecodes

While working on needle, I saw a number of places where the bytecode of mako is larger than it should be. This isn't likely to be an issue for correctness, but it's uglier and harder to read. In addition, creating larger methods might sometimes inhibit inlining and cause worse performance.

For once example, while compiling the regex (ab|a|bcdef|g)+ the implementation at the time was producing output like:

       280: goto          283
       283: goto          286
       286: aload_0

In general, I remember having difficulty with the handling of nested control flow, so it's likely that code could be improved.

Current treatment of casts is not specified and inconsistent

Returning a value that can be cast will apply the cast. Otherwise, implicit casts are not done.

Fix or document this inconsistency.

Side note: inconsistently with the java language, a long can be implicitly cast to an int.

API to handle string constants

String constants are ubiquitous, and right now it requires significant ceremony to use them--you have to pass the constant from whatever method you're using from where you're computing it out to a static block where you initialize it.

It seems easy enough for the class builder to make this simpler. Could, for instance, have a method that can be called from anywhere, which handles setting up the constants, so that they can be used.

registerStringConstant("constantName", "someString");
set("x", getStatic("constantName", thisClassType, Type.of(String.class));

We could also support an api like:

set("x", stringConstant("constantName", "someString"));

Which could do much the same thing, but allow inline definition at the point of use. In this case, the stringConstant call would have a side-effect of registering the constant, and return a StaticFieldReference as if we'd called getStatic. The advantage would be that it's concise, but I'm not sure multiplying the ways to do things is worth it.

Add increment to language

The low level API supports it, no reason Mako shouldn't.

Current implementation supports dead return statements

See

mako/src/test/java/com/justinblank/classcompiler/lang/TestMethods.java

Line 46 in e6d7363

public static Method multiReturn() {

. This is an accidental feature, because we're not doing any analysis to detect dead return statements.

It's not clear to me that it's worth attempting to correct this, so just creating this as a tracking issue.

Support for generating source code

My original intent for mako was focused around my uses for it in compiling regexes. For that use case, there's no reason to want to generate Java source code. However, this is a use case that many people have.

Thinking about it, I don't see an obstacle to outputting Java source code from mako. I don't think I'm inclined to implement it now, but I'm creating this issue as pseudo-documentation, because my first thought is that it should be relatively tractable.