onelang / onelang

TL;DR

• V1 live demo on https://ide.onelang.io
• V2 is in progress

What is OneLang.io?

OneLang has a hard time defining itself.

Some may call it a transpiler, a special compiler which can compile source code from an input language to another. Those will have a hard time using it, as OneLang has its own rules and does not respect those of the input language, sometimes not even its syntax.

So the question arises, is OneLang a new programming language? Although we can talk about it with words like object-oriented or generic or strongly-typed, it has its own type system and AST, would you call something a programming language which does not have its own syntax?

Thus let's just define OneLang as the following for the time being: OneLang is a tool which helps writing code in multiple langauges at the same time.

It doesn't solve the problem for you, it just helps you a bit in it. You still need to master the target languages and OneLang to be able to produce anything usable.

Similar projects

• Haxe: https://haxe.org/
• progsbase: https://www.progsbase.com/
• Universal-transpiler: https://jarble.github.io/transpiler/
• pseudo: https://github.com/pseudo-lang/pseudo

Comparison: https://github.com/onelang/OneLang/wiki/OneLang-vs.-Haxe-vs.-Progsbase-comparison

onelang's Issues

Global function resolving

Add ability for native resolvers to resolve functions like PHP's count and others.

(Currently a workaround for this to use generic transformer. Maybe just generic transformers have to be more clever?)

They say, it takes a village to raise a child!

I am very interested in this project and I want to know how well does it compile to C language as I want to use it to transpile Python like code in C.

Lang: support float / double

Compare to https://kythe.io/

How does it work?

I'm curious about lots of things about this project, but a few jump out right away.

1. The video says I can use several input languages, but how? I don't see a way to choose anything but TypeScript for the top-left pane, and editing any other pane does not propagate changes to the others.
2. Does the Generator tab contain a complete description of how your internal representation is converted to text, or are there additional knobs we can't see?
3. The Generator tab appears to contain code in some kind of programming language, in curly braces. What language is this?
4. How does the reverse conversion (from text to the internal representation) work? I don't expect you can magically use the stuff in the "Generator tab" in reverse? And yet, somehow, you appear to have single-handedly written five "reverse generators"! How much time did that take?
5. Are you aware of the Loyc project?

As the inventor of LES I'm a bit biased, but I can't help but think that this project could benefit from using it. For example, compared to JSON or plain text, LES would provide a much more compact and intuitive way to express your ASTs.

Perhaps we could collaborate... for a brief time period starting right now, I have a lot of involuntary free time on my hands. Your primary language seems to be TypeScript, and it would be an interesting exercise to try to port my C#-based LES parser and printer to TypeScript... maybe using OneLang itself to help me do it! What do you think, if I did that, would you be interested in using LES somewhere in OneLang?

Then there's my Enhanced C# parser... if I could port that to TypeScript, you'd have access to a high-quality C# parser (though it doesn't quite support C# 7 yet.) Since the EC# and LES parsers were generated by the same parser generator (which I also wrote), if I can manage to port the LES parser to TypeScript, the EC# parser should be a pretty easy follow-up.

Compiler: entry point code at top-level

[WebIDE] Check out PhosphorJS

Maybe it's better than GoldenLayout?

Template: self-removing blocks

There are multiple cases where we only have to add eg. a method if it has content. This sometimes needs to be predicted beforehand, which requires some ugly hacks right now.

Most of the cases happens where we use an {{if}} inside a {{for}} and we don't know many elements will be printed out actually. But even if we use two of fors or ifs it complicates the "predictor" expression and looks ugly.

We also need an {{else}} option where we don't just remove the block but replace it with some other content (eg. with pass in Python)

Some places where we need this logic:

• every class.needsConstructor property usage
• Golang: init method
• Python: fill method body or use pass
• TODO: others...

How does it work?

If every child template gives empty result then the block is not showed (even if there is static content inside the block, eg. prefix / postfix text). If {{else}} is specified then instead of empty result, the contents of {{else}} is shown.

Proposed syntax

Python use pass if the method has no statements:

{{block}}
  {{genBody(method.body)}}
{{else}}
  pass
{{/block}}

Go / other langs: remove constructor if not used:

{{block}}
  func init() {
    {{for class in classes|sep=\n\n}}
      {{if class.reflect}}
        ...
      {{/if}}
    {{/for}}
{{/block}}

To-be-decided

What happens if we want to use that information whether a block was generated or not? Eg. only call an initialization method if it was generated beforehand? I don't know such case now, but we may run into one in the future...

Implement StringIndex type

You cannot index a string with a number in Swift.

Currently we are using a workaround which is really slow (indexing is O(N) instead of the expected O(1)) and really ugly (String(str[str.index(str.startIndex, offsetBy: i]) instead of str[i]).

So we have to introduce a StringIndex type which uses number underlying on every language expect Swift where it maps to String.Index type. We also need to add some helper methods eg. str.startIndex which gives 0 usually, except in Swift where it gives str.startIndex.

Adding Matlab

If you are interested in adding Matlab you might want to look at:https://misshit.org.
It appears to have a very comprehensive and effective parser for recent Matlab versions.

There is also the venerable SMOP https://github.com/victorlei/smop and an interesting derivative Mlabpy https://github.com/led02/mlabpy which manipulates the AST.

OneLang test issues

• Python: else indentation breaks logic
• Python: Tokenizer::offset is not initalized to 0 in __init__
• Python: static method StringHelper::starts_with_at_index has self argument and no @staticmethod attribute
• Swift: Tokenizer::init uses ! for setting operators field
• Swift: Tokenizer::tokenize: fix empty array init (var result = [] -> var result = [Token]())
• Swift: convert substrings to string (str[x ..< y] -> String(str[x ..< y]))
• Swift: do not force unwrap operators at foreach (for curr_op in self.operators! -> for curr_op in self.operators)
• Swift: static methods should be prefixed with class (func startsWithAtIndex -> class func startsWithAtIndex)
• Swift: nullability fixes... (eg. return [Token]? instead of [Token] and force unwrap every nullable value on access)
• Java: infer foreach item type (instead of Object)
• Java: use equals instead of == for comparison between strings (StringHelper.startsWithAtIndex)
• Ruby: create accessors for static fields

    class << self
      attr_accessor :identifier
    end

• Ruby: fix infinite loop (comparison issue?)
• C++: add ; after break
• C++: prefix static methods with static (make StringHelper::startsWithAtIndex static)
• C++: unify reference handling (make everything shared pointer? so sp<vector<sp<object>>>>???)
  • fix vector initialization
  • fix return value type
  • fix class field type
• Go: replace OneArray with concrete type (eg. []string or []Token)
• Go: use for instead of while
• Go: fix static methods
• Go: find out how to use array with custom class (eg. []*Token)
• JavaScript: initialize variable (Tokenizer::constructor: this.offset = 0;)
• JavaScript: use static prefix for static methods (StringHelper::startsWithAtIndex)
• Perl: fix array initializer
• Perl: fix infinite loop bug

Nullability

Nullability in target languages

In some languages any variable can be null, even primitive integer types. This is true for mostly dynamically-typed languages, eg. Ruby, Python, Perl, PHP, JavaScript.

In C# and Java most of the types (eg. String, Array, Map, user-defined types) can be null, but the primitive ones are not (in C# we can use the Nullable class / ? qualifier for that, in Java the boxed instances: eg Integer instead of int).

In the languages mentioned above there is no option the make the nullable types non-nullable.

In other languages nothing can be null by default, only if we make them nullable explicitly (eg. by using pointers in C++ and Go or the ? qualifier in Swift).

In TypeScript the default nullability depends on the strictNullChecks configuration.

Nullability in OneLang

The current implementation has a pretty adhoc nature in the nullability area. Generally primitive types (bool, int, float), string and enums are non-nullable, while everything else (Array, Map, user-defined types) is nullable.

At first string was nullable, but this leaded to some pretty ugly code eg. in C++ at string concatenation.

Nullability of Array and Map is still a question. null is a two-edged sword, it can be used as an easy way to indicate missing value, but allowing null makes the code more error-prone and can generate ugly code in languages which are non-null by default.

The best way would be a more clever compiler which can decide if a field / variable can be ever null, but this is not the goal of the PoC milestone, so we currently allow everything to be null, until this causes serious problems (even if it results in some really ugly generated code).

Enum support

Language	Native support
C++	✅
Go	❌
C#	✅
Java	✅
JavaScript	❌
Perl	❌
PHP	❌
Python	from 3.4
Ruby	❌
Swift	✅
TypeScript	✅

WebIDE: error window

Show every error (parsing, compilation, etc) in a distinct error window / panel.

Refactor template includes

Resolve the following issues:

• template includes only collected after template generation happens, so it needs to be split into a two stage process: generate code first, then generate includes
• testGenerator.includes does not work (e.g. C++)
• conditional includes should be introduced (e.g. in C++'s getType conditionally uses one:: namespace which requires OneLang.Core) - with e.g. a non-returning template action {{ includes("something") }}

Big integer support for C++, PHP and Swift

In #11 big integer support was implemented in language where it is supported natively.

The following languages require external libraries for big integer support: C++, PHP, Swift

JSON serialization via Reflection

Generated code issues - first round

• Go: non-primitive fields should be pointer types
• Java: one public class per file
• Java: fix fast compilation's loading issue
• Java: import java.util.ArrayList & import java.util.Arrays
• Java / Generic: use converted type: Onearray arr -> ArrayList<Integer>
• Java: arr[i] -> arr.get(i)
• Perl: arrays should be prefixed with @ and not $
• Swift: set field type
• Swift: find out how to handle optional values
• Swift: non-mutated fields should be let and not var (or disable warning?)
• Swift: i++ -> i += 1
• Swift: remove semicolons (;)?
• TS/JS: non-mutated fields should be const and not let (low prio)
• C++: non-primitive fields should be pointer types
• C++: auto arr = std::make_shared<std::vector<int>>(std::vector<int>({ 1, 2 }));
• C++: item access: (*arr)[i]
• Fix double semicolon (;) issues
• Perl, Python, Ruby: convert comments to #-type ones

Go: bitwise complement operator is ^, not ~

The following code:

var x = 3;
x = ~x;

must be translated to Go as:

x := 3;
x = ^x;  // Not ~x

See https://play.golang.org/p/CbJQ-gPguyG

Can't launch the project on OSX

./serve.py
-bash: ./serve.py: Permission denied

sudo ./serve.py
sudo: ./serve.py: command not found

Top-level functions

Parse and generate top-level functions.

In case of C# and similar languages put the top-level functions to the Program (/etc) class.

Compact generic transforms

Add an option to describe the transformation in the following form:

 - input: $object[$elementExpr] = $newValue
   output: $object.set($elementExpr, $newValue)

which will do the same as now this:

  - description: "$object[$elementExpr] = $newValue -> $object.set($elementExpr, $newValue)"
    input:
      exprKind: Binary
      operator: =
      left:
        exprKind: ElementAccess
        object: $object
        elementExpr: $elementExpr
      right: $newValue
    output:
      exprKind: Call
      method:
        exprKind: PropertyAccess
        object: $object
        propertyName: set
      arguments: [$elementExpr, $newValue]

Non-critical feature / bug list

• field / property / etc name casing depends on isStatic and visibility
• lang-specific keyword array, and template for escaping those variables (or a renaming scheme)
• fix code generator's line concatenating bug and remove unnecessary \<space>s
• if template name contains number then template compiler thinks it's a local variable
• string escaping differences
• Compiler: if StaticField.thisExpr is ThisReference then replace it with static ClassReference (so this.staticField will be resolved to SomeClass.staticField)

WebIDE: mark error positions in input files

Make text where the issue occurred underlined with a wavy red line and/or show warning icon on the left-side column in Ace for parsing or even runtime errors.

Standard libraries

The issue

OneLang does not really come with standard libraries (StdLibs) or more precisely wrappers around other languages' standard libraries.

One of the reasons is that providing a standard library is an enormous effort and requires a lot of time which is better spend on the compiler itself.

Other reason is that the choice is not clear in some cases: if the target language does not support a functionality then you have to use a library for that and if there are more wide-spread libraries for that purpose then it could be that OneLang would use one of them, and your project would use an other one and this could cause various conflict and issues (including you'd need to use two dependencies instead of one, etc.)

So I propose the following schema:

The plan

Creating (standard?) libraries for One will be a decentralized, community effort.

But we cannot expect from anyone to learn 11 languages and their standard libraries and create wrappers for all of them. I would expect most of the users will only use One for only two or three languages, so there would be incentive to create wrappers for just that many languages at once, not more.

So we split a StdLib wrappers into interfaces and implementations.

The interface and the implementation

The interface describes the classes, methods, parameters, input and output entities and their behaviors without a concrete implementation.

Emphasis is on the behavior as an interface comes with unit tests and these tests define the contract between the library's creator and the library's user: these show you what you can expect from an interface.

Instead of writing down in a textual form on a documentation website what this and that method should do if we pass this and that parameters, we write it down in tests!

If there is no test for an edge-case then it's an undefined behavior and you cannot assume that it will work as expected or work at all.

You can then publish an interface to a centralized repository (provided by onelang.io) where users can find your interface and start using it. The more users use your interface it has the better chance that others will create an implementation for it and vice versa: if you create an interface based on your implementation and your implementation becomes the popular one then you have a better chance that your interface will become the popular "standard" one as well and implementations supporting other languages will be created for your interface.

An implementation can implement one or more interfaces. It may depend on some external library which comes from the target language's dependency manager (if that exists), so its usage will be streamlined into the development process of the target language.

Also an implementation can only be published to the repository if it passes the tests specified by the interfaces it implements.

Wut

The whole concept may sound weird at first, but hopefully it will be an iterative process where we will converge to more and more better libraries.

It may come hard to some who are accustomed to a strong, centralized vendor who provides a well-tested, standard library for all their basic needs, but on the other hand, in some languages, it's usual that for even the basic needs (eg. JSON parsing) you need a third-party library and it's your decision and risk taking which one to use. I don't think this situation will be much different.

Some libraries will prefer staying backward-compatible and staying faithful to their existing users, while others will decide to give a better, cleaner interface and attract new users easier. Or to be frank they may kill two birds with one stone: they can implement both the old and the new interface while keeping the interfaces completely invisible to the other interface's user base.

State of affairs

These are just plans, nothing concrete have been implemented / tested.

Comments / questions are welcome.

Parser: PHP

StdLib: JSON load / save

Required basic features for code generator

• RegExp match
• RegExp test (?)
• RegExp replace
• String split
• String concat
• String join (?)
• Array SubArray (?)
• Any to string (?)
• JSON parse
• YAML parse
• Enum support
• Map support (contains key, remove, add, get values, get keys)
• Import tracking (causes Go error)

How to use it as a transpiler?

The documentation seems a bit odd

Run tests after every commit

...and send email / Slack notification about failures

Compiler: byte array support

Reflection

A rudimentary reflection had been implemented, but it's kind of mess and probably requires a full rewrite, but first we need to find out some overlooked aspects.

TODOs

• Option to create instance of a class (eg. by name)
• Publish full type system
• Add target language-specific and canonized names (and maybe even source lang names - as those will be used in strings for getting classes / fields / methods, but if the source code contains more than one compilation units, then parts can be compiled from different languages, so there is no "source lang")
• Support inheritance (if once we add inheritance to OneLang at first place)
• Make differences disappear between languages (eg. currently C# lists inherited Object methods in the GetMethods() call: Equals, ToString, etc)
• Canonized name (used by lookup functions) should handle the difference between field and _field (or the compiler should prevent generation of fields with conflicting names like these)
• Fix canonized name implementation (or the lack of) in some languages (lower casing is not enough, we have to remove underscores too!)
• Write tests

Using native reflection vs. lambdas

The current implementation tries to use native reflection where it can. Some languages does not have native reflection (eg. C++), or it allows only reflect instances, so no static field / method information is available (eg. Swift) or static is not understood in that language, OneLang uses fake static members (eg. Go).

The pros of using native reflection can be the source and compiled code size, maybe performance. As they usually don't require too much generated code, in the best case only the list of reflected classes have to be registered, so it's easy to manually modify the generated code.

On the other hand if we want to publish OneLang field / class "attributes" (attributes in C#, decorators in TypeScript, etc), and complete Type metadata (including canonized and language-specific names) then we have to generate tons of code anyway.

At the end maybe the only difference will be the setter / getter / caller / initiator lambdas. If we go that way, and use lambdas in every language then the whole reflection will work the same way and maybe it will be even faster (as the JIT can compile fast code instead of a dynamic, name-based field / method access code).

https://ide.onelang.io/ is down

https://ide.onelang.io/ is absent from the web,

Add a license

The package.json hints at this project being licensed under the ISC license, but since it is the default choice set by npm init, it may have limited legal value.

Please consider adding an open source license; see Choose a License for more informtaion. Thanks in advance 😃

Exceptions

Currently One does not support real exceptions, only raising one-shot ~fatal error.

To support exceptions the following scenarios have to be planned out:

• Support languages which does not have exceptions, but you have to return error code / object instead of throwing exception. This changes the code flow significantly.

• Track which methods throw exception, this probably requires an independent stage of some depth-first search with some logic for circle / recursion detection

StdLib - Stage #1

Epic: #9

Stage 1 - Split current libs into packages

As a first step remove standard library related entries from the language YAML files, more precisely the root > classes node.

Add a packages folder and various packages as subfolders, eg. One.Reflect-v0.1 and put a package.yaml file there with metadata and the current inline implementations (alternatively put inline implementation separately). Also put language-specific files there (eg. One.Reflect.cs).

It would be good if the WebIDE could download multiple files at once, so we don't have to execute thousands of requests. Conveniently it could be a JSON payload generated at a publish step (for now) + an endpoint on static serve.

In this stage we just concatenate the various StdLib parts into one file (order matters for eg. C++!).

Interfaces and implementations maybe should be already separated at this point already (but still without tests).

Preliminary big integer support