jzimmerman / langcc Goto Github PK

View Code? Open in Web Editor NEW

1.7K 1.7K 57.0 1.52 MB

langcc: A Next-Generation Compiler Compiler

Home Page: https://langcc.io

License: Apache License 2.0

Makefile 0.11% Shell 0.01% C++ 99.83% Python 0.03% Go 0.01%

langcc's People

Contributors

Stargazers

Watchers

Forkers

halosghost 5gapp nsauzede sentido-labs gmh5225 clmpo thearchiver snej clownsw dejan-stankovic brucerennie romatthe unlimitedsoftwareworks clayne 4144 liangtie jaketbrown tinker-hacker xunfeng1980 maifeeulasad sondro nitin-mane richardsonjf ivansprado language-lab redchew-fork santakd mbrukman doytsujin mu-l migzone linecode asdlei99 john-small tzyhsw jrcribb animesh hellcwang ocipower freakynit mfkiwl lzufalcon killvxk jddurand zmkeh feiyunwill leoadberg shonker b-xiang yibit koliyo bxlxx danlapid stjordanis caleblarsen gg-big-org yui-knk marcio-absmartly

langcc's Issues

Invalid tokens with some characters

I am experimenting a bit with langcc to generate a Smalltalk parser from one of its BNFs. However, I encounter an "Invalid token" problem with some characters, e.g. % < > =.

Ex:

langcc -vvvv vw.lang gen
[000:00:00.000510] Invalid token
[000:00:00.000510] Line 2, column 92:
[000:00:00.000510] 
[000:00:00.000510]       binary_character <- `+` | `/` | `\` | `*` | `~` | `=` | `@` | `|` | `?` | `!` | `,` | `%` | `<` | `>`;
[000:00:00.000510]                                                                                              ^               
[000:00:00.000510]

How can I do to escape them?

This is the .lang I wrote so far:
(The lexer and parser stanzas are simply copied from the manual because I have not yet reached that part :-)

tokens {
    binary_character <- `+` | `/` | `\` | `*` | `~` | `=` | `@` | `|` | `?` | `!` | `,` | `%` | `<` | `>`;
 
    digit <= `0`..`9`;
    digits <= digit*;
    letter <= `a`..`z`;
    id <= letter (letter | digit)*;

    named_literal <= `nil` | `true` | `false` | `super`;
    byte_array_literal <= `#` byte_array_literal_body;
    byte_array_literal_body <= `[` number* `]`;
    literal <= #L[e::`-`] number | named_literal | symbol_literal | character_literal | string array_literal | byte_array_literal;

    comment <= `"` (non_quote_character | `'` )* `"`;
    separator <= (whitespace_character | comment)+;

    big_digits <= (digit | letter)+;
    optional_fraction_and_exponent <= #L[e::`.` digits] #L[e::(`e` | `d` | `s`) #L[e::`-`] digits];
    number <= digits (`r` #L[e::`-`] big_digits | optional_fraction_and_exponent);
   
    extended_letter <= letter | `_`;
    identifier <= extended_letter (extended_letter | digit)*;
    whitespace_character <=  ` ` | `\n` | `\t`;
    
    keyword <= identifier `:`;
    block_argument <= `:` identifier;
    assignment_operator <= `:` `=`;
    binary_selector <= (`-` | binary_character) [binary_character];
    character_constant <= `$` (non_quote_character | `'` | `"`);    
    non_quote_character <= digit | letter | binary_character | whitespace_character | `[` | `]` | `{` | `}` | `(` | `)` | `_` | `^` | `;` | `$` | `#` | `:` | `.` | `-` | ``;
    string <= `'` (non_quote_character | `'` `'` | `"`)* `'`;
    token <= keyword | block_argument | assignment_operator | binary_selector | character_constant | string;

    unary_selector <= identifier;
    symbol <= identifier | binary_selector | keyword+;
    unary_message <= unary_selector;
    binary_message <= binary_selector primary unary_message*;
    keyword_message <= (keyword primary unary_message* binary_message*)+;
    cascaded_messages <= (`;` (unary_message | binary_message | keyword_message))*;
    messages <= unary_message+ binary_message* [keyword_message] | binary_message+ [keyword_message] | keyword_message;
    rest_of_expression <= #L[e::messages cascaded_messages];
    pseudovariable_name <= `self` | `thisContext`;
    primary <= extended_binding_name | binding_reference | pseudovariable_name | literal | block_constructor | `(` expression `)`;

    extended_binding_name <= binding_name #L[e::( `.` binding_name )* ];
    expression <= ( extended_binding_name | binding_reference ) (assignment_operator expression | rest_of_expression) | keyword `=` expression | primary rest_of_expression | `super` messages cascaded_messages;
    expression_list <= expression (`.` expression)* #L[e::`.`];

    binding_name <= identifier ( named_literal | pseudovariable_name );
    declared_variable_name <= binding_name;
    temporary_list <= declared_variable_name*;
    temporaries <= `|` temporary_list `|` | `||`;

    pragma <= `<` ( keyword literal )+ `>`;
    statements <= #L[e::`^` expression #L[e::`.`] | expression #L[e::`.` statements]];
    block_constructor <= `[` #L[e::block_declarations] statements `]`;
    block_declarations <= temporaries | block_argument+ (`|` [temporaries] | `||` temporary_list `|` | `|||`);
    message_pattern <= unary_selector | binary_selector declared_variable_name | (keyword declared_variable_name)+;
    method <= message_pattern pragma #L[e::temporaries] statements;

    top <= method;
}

lexer {
    main { body }

    mode body {
        top => { emit; }
        ws => { pass; }
        `"` => { push comment_single; pass; }
        eof => { pop; }
    }

    mode comment_single {
        `\n` => { pop_extract; }
        eof => { pop_extract; }
        _ => { pass; }
    }
}

parser {
    main { S }
}

'langcc_rt.hpp' file not found on make

Probably a minor issue when doing make from the examples/basic directory:

> make
langcc basic.lang gen
[000:00:00.001602] -- Performing initial validation and tabulation
[000:00:00.002307] -- Compiling lexer
[000:00:00.006472] -- Compiling parser: tabulating symbols
[000:00:00.006569] -- Compiling parser: inferring attributes
[000:00:00.006809] -- Compiling parser: symbol iteration 1 (0 triggers)
[000:00:00.008675] -- SLR(1) NFA: 10 vertices, 9 edges
[000:00:00.009147] -- SLR(1) subset NFA: 8 vertices, 7 edges
[000:00:00.009313] -- Compiling parser: final lookaheads
[000:00:00.009852] -- Compiling parser: constructing LR NFA
[000:00:00.010270] -- Compiling parser: LR NFA subset construction
[000:00:00.010744] -- Compiling parser: no LR conflicts
[000:00:00.010766] -- LR NFA: 10 vertices, 9 edges
[000:00:00.010783] -- LR DFA: 8 vertices, 7 edges
[000:00:00.011062] -- Generating AST datatype definitions
[000:00:00.031205] -- Generating AST writer definitions
[000:00:00.031948] -- Generating AST parser definitions
[000:00:00.039334] -- Generating global definitions
[000:00:00.046865] -- Running: clang++ -D __MACOS__ -D __MACOS_SDKROOT__=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o build/gen_test_bin/basic__gen_test -g -ggdb -g3 -std=c++17 -fno-omit-frame-pointer -mmacosx-version-min=12.0 -ldl -I ./gen -I ./src gen/basic__gen.cpp gen/basic__gen_test.cpp 
In file included from gen/basic__gen.cpp:1:
./gen/basic__gen.hpp:3:10: fatal error: 'langcc_rt.hpp' file not found
#include <langcc_rt.hpp>
         ^~~~~~~~~~~~~~~
1 error generated.
In file included from gen/basic__gen_test.cpp:1:
./gen/basic__gen.hpp:3:10: fatal error: 'langcc_rt.hpp' file not found
#include <langcc_rt.hpp>
         ^~~~~~~~~~~~~~~
1 error generated.
[000:00:00.344294]
[000:00:00.344294] Failure: clang++ -D __MACOS__ -D __MACOS_SDKROOT__=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o build/gen_test_bin/basic__gen_test -g -ggdb -g3 -std=c++17 -fno-omit-frame-pointer -mmacosx-version-min=12.0 -ldl -I ./gen -I ./src gen/basic__gen.cpp gen/basic__gen_test.cpp 

Stack trace:
  #1: main [??:0:0]
  #2: compile_lang_full [??:0:0]
  #3: AX [??:0:0]
  #4: AT [??:0:0]
  #5: void AT<..> [??:0:0]
Assertion failed
Assertion failed: (cond), function AT, file langcc_util.hpp, line 1236.
make: *** [gen/basic__gen.cpp] Abort trap: 6
make: *** Deleting file `gen/basic__gen.cpp'

Arch linux support

Hey,

Trying to get it running on arch linux, these are changes so far:

Here is my shell script for Arch deps:

#!/bin/bash
set -e
set -u
sudo pacman -S llvm crypto++ gperftools

Arch has different path than Ubuntu (and same as MacOS)
langcc_util.hpp

#include <cryptopp/sha.h>

There is some linking issues that I have to little experience to solve quickly, so maybe you can see yourself quickly if it is for example https://archlinux.org/packages/extra/x86_64/llvm/ (flagged as outdated as there is v15 out) or something else I need to change to get it working.

$ make -j8

mkdir -p build
mkdir -p build
mkdir -p build
mkdir -p build
clang++ -I./src -I./gen -g -ggdb -g3 -O3 -std=c++17 -fno-omit-frame-pointer -I/usr/include/llvm-c-14 -I/usr/include/llvm-14 -c -o build/datacc_main.o src/datacc_main.cpp
clang++ -I./src -I./gen -g -ggdb -g3 -O3 -std=c++17 -fno-omit-frame-pointer -I/usr/include/llvm-c-14 -I/usr/include/llvm-14 -c -o build/datacc.o src/datacc.cpp
clang++ -I./src -I./gen -g -ggdb -g3 -O3 -std=c++17 -fno-omit-frame-pointer -I/usr/include/llvm-c-14 -I/usr/include/llvm-14 -c -o build/cc__gen.o gen/cc__gen.cpp
clang++ -I./src -I./gen -g -ggdb -g3 -O3 -std=c++17 -fno-omit-frame-pointer -I/usr/include/llvm-c-14 -I/usr/include/llvm-14 -c -o build/data__gen.o gen/data__gen.cpp
clang++ -I./src -I./gen -g -ggdb -g3 -O3 -std=c++17 -fno-omit-frame-pointer -I/usr/include/llvm-c-14 -I/usr/include/llvm-14 -o build/datacc build/datacc_main.o build/datacc.o build/cc__gen.o build/data__gen.o  -L/usr/lib/llvm-14/lib -lLLVMSymbolize -lLLVMDemangle -lLLVMSupport -lcryptopp -lncurses -ltcmalloc
/usr/bin/ld: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../lib64/libLLVMSymbolize.a(SymbolizableObjectFile.cpp.o): in function `llvm::symbolize::SymbolizableObjectFile::getModulePreferredBase() const [clone .localalias]':
(.text._ZNK4llvm9symbolize22SymbolizableObjectFile22getModulePreferredBaseEv+0x10): undefined reference to `llvm::object::COFFObjectFile::getImageBase() const'
/usr/bin/ld: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../lib64/libLLVMSymbolize.a(SymbolizableObjectFile.cpp.o): in function `llvm::symbolize::SymbolizableObjectFile::addCoffExportSymbols(llvm::object::COFFObjectFile const*) [clone .localalias]':
(.text._ZN4llvm9symbolize22SymbolizableObjectFile20addCoffExportSymbolsEPKNS_6object14COFFObjectFileE+0x5e): undefined reference to `llvm::object::COFFObjectFile::export_directories() const'
/usr/bin/ld: (.text._ZN4llvm9symbolize22SymbolizableObjectFile20addCoffExportSymbolsEPKNS_6object14COFFObjectFileE+0xbb): undefined reference to `llvm::object::ExportDirectoryEntryRef::moveNext()'
/usr/bin/ld: (.text._ZN4llvm9symbolize22SymbolizableObjectFile20addCoffExportSymbolsEPKNS_6object14COFFObjectFileE+0xc7): undefined reference to `llvm::object::ExportDirectoryEntryRef::operator==(llvm::object::ExportDirectoryEntryRef const&) const'
/usr/bin/ld: (.text._ZN4llvm9symbolize22SymbolizableObjectFile20addCoffExportSymbolsEPKNS_6object14COFFObjectFileE+0xf7): undefined reference to `llvm::object::ExportDirectoryEntryRef::getSymbolName(llvm::StringRef&) const'
/usr/bin/ld: (.text._ZN4llvm9symbolize22SymbolizableObjectFile20addCoffExportSymbolsEPKNS_6object14COFFObjectFileE+0x116): undefined reference to `llvm::object::ExportDirectoryEntryRef::getExportRVA(unsigned int&) const'
/usr/bin/ld: (.text._ZN4llvm9symbolize22SymbolizableObjectFile20addCoffExportSymbolsEPKNS_6object14COFFObjectFileE+0x258): undefined reference to `llvm::object::COFFObjectFile::getImageBase() const'
/usr/bin/ld: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/12.2.0/../../../../lib64/libLLVMSymbolize.a(SymbolizableObjectFile.cpp.o): in function `llvm::symbolize::SymbolizableObjectFile::create(llvm::object::ObjectFile const*, std::unique_ptr<llvm::DIContext, std::default_delete<llvm::DIContext> >, bool)':
(.text._ZN4llvm9symbolize22SymbolizableObjectFile6createEPKNS_6object10ObjectFileESt10unique_ptrINS_9DIContextESt14default_deleteIS7_EEb+0x9e): undefined reference to `llvm::object::computeSymbolSizes(llvm::object::ObjectFile const&)'
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [Makefile:134: build/datacc] Error 1

Generate syntax highlighting files

Would be cool to generate some syntax highlighting files for web

Probably needs #9 first to identify what is what

Wishlist: use CI for validation

Proposal is to to use github worfklows

How to handle significant whitespace?

Hi! I'm interested in exploring how to use this parser system for a language that uses indentation to indicate scoping - which, notably, is not something a raw LR parser (or any strict context-free parser) can handle.

Normally, this would be handled in the lexer by consuming all the space/tab indentation in the lexer and emitting INDENT/DEDENT tokens for when the indentation is increased/decreased, respectively.

I'm puzzled as to how this works in the example py.lang grammar? The only mention of whitespace there seems to be immediately discarding it. I also don't see any mention in the technical paper on arxiv as to how this is handled.

Comparison to K framework

Hi, I'm on of the maintainers of cross-lang-and-cross-platform project. How does yours compare to K framework?

Compilation fail

I am trying to compile generated C++ file with g++ calc__gen_test.cpp but I have 'error: ‘std::filesystem’ has not been declared'.

I saw on the web that compile with -lstdc++fs -std=c++17 might be an issue, it works but now I have another error :

calc__gen.hpp:987:60: error: inlining failed in call to always_inline ‘lang_rt::ParserVertexId lang::calc::parser::vertex_dfa_step(lang_rt::ParserVertexId, lang_rt::ParserSymId, lang_rt::ParserAttrMask)’: function body not available
987 | attribute((always_inline)) lang_rt::ParserVertexId vertex_dfa_step(lang_rt::ParserVertexId v, lang_rt::ParserSymId sym, lang_rt::ParserAttrMask attr);

Requires 2 package managers on Mac / Hardcoded compiler flags

From the discussion on lobste.rs, by user alec:

By default it needs both Homebrew and Ports on Mac. I’m not going to install Ports just to get libcrypto++, so I manually installed it in /usr/local, but after adding the install path to the CFLAGS/LFLAGS in the root Makefile it still failed to find libcrypto at a later stage. It turns out the compiler arguments are hard-coded into the source code.

How exactly does langcc beat Knuth?

There seems to be no Web forum or mailing list, so I guess I should ask questions here?

It's well-known that any language parseable with a pushdown automaton is a deterministic context-free language. However, langcc seems to be able to generate parsers for some wider class of languages, so I'm interested in which of the results from your paper improve on the recognizing power of LR(k)/PDA?

If I wanted to make a strictly LR(1) parser generator based on your paper, which of your features/results should I ignore?

Enable GitHub Discussions for repo

I think it would be a good idea to enable GitHub discussions. Also perhaps have chat channel.

Also maybe this can be moved to an GtiHub org of its own.

UTF-8

Please add UTF-8 example

The example grammar from the langcc slides can't be parsed?

Consider the following:

// https://mdaines.github.io/grammophone/?s=UyAtPiBhIFggYSB8IGIgWCBiIHwgYSBZIGIgfCBiIFkgYS4KWCAtPiBjICJYJyIuClkgLT4gYyAiWSciLgoiWCciIC0+IGMuCiJZJyIgLT4gYy4=
// S -> a X a | b X b | a Y b | b Y a.
// X -> c "X'".
// Y -> c "Y'".
// "X'" -> c.
// "Y'" -> c.

tokens {
    ka <= `a`;
    kb <= `b`;
    kc <= `c`;
}

lexer {
    main { body }

    mode body {
        ka => { emit; }
        kb => { emit; }
        kc => { emit; }
        eof => { pop; }
    }
}

parser {
    main { S }
    S <- `a` X `a` | `b` X `b` | `a` Y `b` | `b` Y `a`;
    X <- `c` XP;
    Y <- `c` YP;
    XP <- `c`;
    YP <- `c`;
}

The grammar is from these slides.

I expected langcc to be able to parse it, because it is LR(1), but it seems like it can't:

[000:00:00.013584] -- Performing initial validation and tabulation
[000:00:00.022069] -- Compiling lexer
[000:00:00.029373] -- Compiling parser: tabulating symbols
[000:00:00.029843] -- Compiling parser: inferring attributes
[000:00:00.032254] -- Compiling parser: symbol iteration 1 (0 triggers)
[000:00:00.037547] -- SLR(1) NFA: 49 vertices, 50 edges
[000:00:00.039303] -- SLR(1) subset NFA: 34 vertices, 35 edges
[000:00:00.039861] -- Compiling parser: final lookaheads
[000:00:00.041901] -- Compiling parser: constructing LR NFA
[000:00:00.043192] -- Compiling parser: LR NFA subset construction
[000:00:00.044896] -- Compiling parser: detected LR conflicts; searching for traces
[000:00:00.045405] -- Compiling parser: tabulating conflicts
[000:00:00.046818] langcc compile error:
[000:00:00.046818] 
[000:00:00.046818]  ===== LR conflict 1 of 1
[000:00:00.046818] 
[000:00:00.046818]      &S              &S    
[000:00:00.046818]            RecurStep(S)    
[000:00:00.046818]     `a`             `a`    
[000:00:00.046818]                            
[000:00:00.046818]                Recur(X)    Recur(Y)
[000:00:00.046818]                            
[000:00:00.046818]                     `c`    `c`
[000:00:00.046818]                     `c`    `c`
[000:00:00.046818]                     `a`    `b`
[000:00:00.046818]                            
[000:00:00.046818] 
[000:00:00.046818] 
[000:00:00.046818] 
[000:00:00.046818] 
[000:00:00.046818]

Add support for semantic labelling of grammar

Would be cool if the grammar supported semantic labels, there is quite a few things you can do with that. Such as

https://microsoft.github.io/language-server-protocol/specifications/specification-3-16/#textDocument_semanticTokens

Additionally labels for entry/exit and open/closed for variables can be useful to do something with data flow. See paper for more details

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/07/hoopl-haskell10.pdf

Add quickcheck tests for printer and parser

Would be nice to automatically generate some tests with a quickcheck implementation such as https://github.com/emil-e/rapidcheck

One could generate some AST, print it, parse the source code and check if the AST still matches

Build failure (i386 issue?)

Hello, using clang++ on Debian GNU/Linux i386 platform langcc fails to compile:

make
mkdir -p build
clang++ -I./src -I./gen -g -ggdb -g3 -O3 -std=c++17 -fno-omit-frame-pointer  -D__CC__=clang++ -Wno-attributes -c -o build/datacc_main.o src/datacc_main.cpp
In file included from src/datacc_main.cpp:2:
In file included from ./src/datacc.hpp:4:
In file included from ./gen/data__gen.hpp:3:
In file included from ./src/langcc_rt.hpp:3:
./src/langcc_util.hpp:1007:13: error: redefinition of 'pr'
inline void pr(ostream& os, FmtFlags flags, size_t x) {
            ^
./src/langcc_util.hpp:963:13: note: previous definition is here
inline void pr(ostream& os, FmtFlags flags, u32 x) {
            ^
1 error generated.
make: *** [Makefile:119: build/datacc_main.o] Error 1

clang++ detail is:

clang++ -v
Debian clang version 14.0.6-2
Target: i386-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/i586-linux-gnu/4.8
Found candidate GCC installation: /usr/bin/../lib/gcc/i586-linux-gnu/4.8.4
Found candidate GCC installation: /usr/bin/../lib/gcc/i586-linux-gnu/4.9
Found candidate GCC installation: /usr/bin/../lib/gcc/i586-linux-gnu/4.9.2
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/10
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/12
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/6
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/6.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/7
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/7.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/8
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/9
Selected GCC installation: /usr/bin/../lib/gcc/i686-linux-gnu/12
Candidate multilib: .;@m32
Candidate multilib: 64;@m64
Candidate multilib: x32;@mx32
Selected multilib: .;@m32

Generating directly executable parsers

The old LR parsing and finite automata literature used to distinguish between two basic ways of implementing DFA: table-based and "directly executable".

Directly executable (or hardcoded) is when the tables are actually encoded in the generated C++ code, usually there would be a label and a block of generated code for each state, and goto would be used for jumping from one state to the other.

If I understood correctly, your approach was to start from table-based and then augment it with some "recursive-descent" stuff (I only skimmed your paper as of yet).

Is the directly executable/hardcoded apprach compatible with your work? Have you considered letting langcc generate such parsers?

Use system C++ compiler if possible

It would be nice if the makefile used the system’s default C++ compiler when possible. LLVM is a pretty hefty dependency to install.

Unless you’re using LLVM stuff other than Clang, it would IMHO be better just to defer to the developer and say “requires Clang vX or later”, and let them take care of installing it. In particular, most any Mac developer will already have Xcode or the Xcode CLI tools installed.

langc.cpp has the same issue: it invokes Clang with a whole bunch of hardcoded options — this is probably going to cause trouble at some point. (I’m not sure why the code has to run Clang? Other compiler-compilers I’ve used just generate source and leave it to the enclosing project to compile it.)

Wishlist: makefile portability

Proposal is to use cmake.

I have forked your repo and started working on that.

LALR instead of SLR for initializing LR(0) partitions.

Currently, langcc appears to use SLR followsets for initializing the partitions in the LR(0) NFA.

It looks to me like LALR followsets could be used instead.

Of course, calculating LALR followsets is more tricky than the standard followsets, but
LPG is a LALR(k) parser generator written in C++ and maybe adopting its approach wouldn't be too difficult.

I think it would be really interesting to see how big of an effect this could have on the size of the final tables.

py_standalone_test failure, go likewise

AR_ge 2 is wrong. 1 is enough.
a helptext would also be useful, that a num_iter arg is expected.
also that the cpython and go sources are expected there

$ build/py_standalone_test
#0 0x00005623ebdf6b81 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/rurban/Software/langcc/build/py_standalone_test+0x246b81)
#1 0x00005623ebbe1be0 void AT<int, int>(bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int) /home/rurban/Software/langcc/./src/langcc_util.hpp:1222:19
#2 0x00005623ebbcb373 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_data() const /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/basic_string.h:235:28
#3 0x00005623ebbcb373 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_is_local() const /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/basic_string.h:275:16
#4 0x00005623ebbcb373 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose() /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/basic_string.h:286:7
#5 0x00005623ebbcb373 std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() /usr/bin/../lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/basic_string.h:795:9
#6 0x00005623ebbcb373 void AR_ge<int, int>(int, int) /home/rurban/Software/langcc/./src/langcc_util.hpp:1266:9
#7 0x00005623ebbcb373 main /home/rurban/Software/langcc/src/py_standalone_test.cpp:23:5
#8 0x00007fcdf55b6d90 __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#9 0x00007fcdf55b6e40 call_init ./csu/../csu/libc-start.c:128:20
#10 0x00007fcdf55b6e40 __libc_start_main ./csu/../csu/libc-start.c:379:5
#11 0x00005623ebbcae15 _start (/home/rurban/Software/langcc/build/py_standalone_test+0x1ae15)
Assertion failed: comparison
Expected: >= 2
Received: 1

py_standalone_test: ./src/langcc_util.hpp:1228: void AT(bool, const std::string &, Ts...) [Ts = <int, int>]: Assertion `cond' failed.
Aborted (core dumped)

ubuntu 22.04 with all deps

Unresolvable simple epsilon-related local ambiguity.

Consider the following:

tokens {
    ka <= `a`;
    kb <= `b`;
    kd <= `d`;
    ws <- `\n` | `\r` | `\t` | ` `;
}

lexer {
    main { body }

    mode body {
        ws => { pass; }
        ka => { emit; }
        kb => { emit; }
        kd => { emit; }
        eof => { pop; }
    }
}

parser {
    main { S }
    S <- `a` B `b` `d`;
    B <- `b` | eps;
}

test {
    `a b b d` <<>>;
    `a b d` <<>>;
}

langcc is neither able to resolve the conflict here, nor provide any guidance that the conflict could be easily resolved by inlining B.

I'm wondering: are conflicts of this kind intentionally not being handled by langcc (because there might be a good reason not to?) or are they just something that were not considered a high priority?

tests bog down, get stuck after xfail_lexer_conflict (macOS)

On my M1 MacBook Pro (16GB RAM, macOS 12.6), the tests that are run as part of ./bootstrap.sh get stuck for quite a while, at least 10 minutes, after the xfail_lexer_conflict success is logged. The entire OS gets very sluggish, and ps awx showed 300 clang processes.

The test output is:

[000:00:02.439655] >>> Launched 153 tests
[000:00:02.441458] [success] xfail_parser_alias_cycle
[000:00:02.451620] [success] xfail_lexer_nested
[000:00:02.451684] [success] xfail_lexer_cycle
[000:00:02.451721] [success] xfail_parser_compound_id
[000:00:02.451759] [success] xfail_lexer_multi_push
[000:00:02.451792] [success] xfail_conflict_sub
[000:00:02.451830] [success] xfail_parser_non_lr
[000:00:02.451857] [success] xfail_lexer_conflict
^CStack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  unittest_lang            0x000000010536e5a8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 76
1  unittest_lang            0x0000000104cfc650 sig_handler(int) + 12
2  libsystem_platform.dylib 0x000000019a24b4a4 _sigtramp + 56
3  libsystem_c.dylib        0x000000019a102fc8 nanosleep + 220
4  libsystem_c.dylib        0x000000019a102ee0 usleep + 68
5  unittest_lang            0x0000000104cfa860 run_unit_tests() + 1352
6  unittest_lang            0x0000000104cf8fb8 main + 1596
7  dyld                     0x0000000105cd508c start + 520
./bootstrap.sh: line 9:  4429 Abort trap: 6           build/unittest_lang

Here’s a snapshot of top just before I killed the tests. Note that RAM is full and there is a lot of swapping.

Processes: 896 total, 187 running, 49 stuck, 660 sleeping, 3459 threads                    11:25:30
Load Avg: 367.04, 229.32, 112.28  CPU usage: 9.77% user, 85.57% sys, 4.64% idle
SharedLibs: 456M resident, 96M data, 24M linkedit.
MemRegions: 229241 total, 9026M resident, 371M private, 3033M shared.
PhysMem: 15G used (3086M wired), 37M unused.
VM: 339T vsize, 3831M framework vsize, 18370824(192938) swapins, 19266739(207417) swapouts.
Networks: packets: 310947792/383G in, 152899790/118G out.
Disks: 74762492/1405G read, 22120795/924G written.

PID    COMMAND      %CPU      TIME     #TH    #WQ   #PORTS MEM    PURG   CMPRS  PGRP  PPID  STATE
0      kernel_task  331.7     15:34:36 564/12 0     0      34M+   0B     0B     0     0     running
967    WindowServer 20.8      18:22:58 24     7     5279-  1863M+ 8704K- 556M+  967   1     sleeping
4429   unittest_lan 15.7      00:57.72 1/1    0     10     5297K  0B     5024K  3818  3818  running
30996  com.apple.We 14.0      01:08.16 7/1    4/1   102    344M+  0B     155M-  30996 1     running
35306  Xcode        11.5      07:35.23 24/1   10/2  2068-  1522M- 0B     1479M+ 35306 1     running
6392   Safari       8.4       41:07.10 26/5   20/4  4046-  833M-  0B     810M+  6392  1     running
3697   sysmond      8.3       06:38.39 3      2     29     2657K  0B     928K-  3697  1     sleeping
6573   Code Helper  7.7       30:37.49 20/1   2     417+   348M+  0B     307M-  6395  6395  running
3985   top          6.9       00:20.79 1/1    0     485    11M+   0B     9328K- 3985  3958  running
91640  Activity Mon 6.5       01:44.08 8      6     424+   96M+   0B     63M-   91640 1     sleeping
4876   clang-15     5.6       00:12.87 1      0     10     97M+   0B     77M+   3818  4875  stuck
4851   clang-15     5.5       00:13.05 1/1    0     10     100M+  0B     78M-   3818  4846  running
6568   Code Helper  5.2       18:47.75 14/3   5     185-   313M+  0B     66M+   6395  6395  running
4843   clang-15     4.7       00:13.07 1      0     10     102M+  0B     85M+   3818  4838  sleeping
4827   clang-15     4.4       00:12.76 1/1    0     10     99M+   0B     78M+   3818  4819  running
4817   clang-15     4.4       00:12.74 1      0     10     99M+   0B     79M-   3818  4800  stuck
4841   clang-15     4.3       00:12.79 1      0     10     98M+   0B     79M+   3818  4831  sleeping
4801   clang-15     4.3       00:12.78 1      0     10     100M+  0B     81M-   3818  4766  sleeping
4856   clang-15     4.3       00:12.79 1      0     10     98M+   0B     80M-   3818  4847  stuck
4813   clang-15     4.3       00:12.71 1      0     10     99M+   0B     78M-   3818  4786  sleeping
4803   clang-15     4.2       00:12.71 1/1    0     10     100M+  0B     81M-   3818  4755  running
4864   clang-15     4.2       00:12.92 1/1    0     10     99M+   0B     80M-   3818  4862  running
4863   clang-15     4.2       00:12.96 1/1    0     10     101M+  0B     80M-   3818  4855  running
4814   clang-15     4.2       00:12.73 1/1    0     10     99M+   0B     82M+   3818  4794  running
4811   clang-15     4.2       00:12.66 1      0     10     100M+  0B     82M-   3818  4783  sleeping
4865   clang-15     4.2       00:12.80 1/1    0     10     100M+  0B     80M-   3818  4857  running

Build process completed with 3 tests failed

I'm using macOS 11.7 first so that could be an issue?
My environment details:

$ clang --version
Homebrew clang version 15.0.0
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /usr/local/opt/llvm/bin

$ > llvm-config --version --prefix --includedir --libdir
15.0.0
/usr/local/Cellar/llvm/15.0.0
/usr/local/Cellar/llvm/15.0.0/include
/usr/local/Cellar/llvm/15.0.0/lib

HOMEBREW_VERSION: 3.6.2-26-g4441452
ORIGIN: https://github.com/Homebrew/brew
HEAD: 4441452f4e7a7efdd5ae8cc98466d8cf6fa410ab
Last commit: 2 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 44f9ecb1cc912260152fd6221fe012dbfca1af90
Core tap last commit: 6 hours ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 2.6.8 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8_1/bin/ruby
CPU: quad-core 64-bit broadwell
Clang: 13.0.0 build 1300
Git: 2.37.3 => /usr/local/bin/git
Curl: 7.64.1 => /usr/bin/curl
macOS: 11.7-x86_64
CLT: 13.2.0.0.1.1638488800
Xcode: 12.5.1 => /Users/mvs/Downloads/Xcode.app/Contents/Developer

$ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables | grep version
version: 13.2.0.0.1.1638488800

$ xcode-select -p
/Library/Developer/CommandLineTools

The build:

$ ./deps_macos.sh
$ make -j8

...
clang++ -c -o build/go__gen.o -I./src -I./gen -g -ggdb -g3 -O3 -std=c++17 -fno-omit-frame-pointer -D__HAS_LLVM_SYMBOLIZER__ -D__LLVM_SYMBOLIZER_PATH__=/usr/local/opt/llvm/bin/llvm-symbolizer -D__MACOS__ -D__MACOS_SDKROOT__=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -mmacosx-version-min=12.0 gen/go__gen.cpp
mkdir -p build
clang++ -o build/py_standalone_test -I./src -I./gen -g -ggdb -g3 -O3 -std=c++17 -fno-omit-frame-pointer -D__HAS_LLVM_SYMBOLIZER__ -D__LLVM_SYMBOLIZER_PATH__=/usr/local/opt/llvm/bin/llvm-symbolizer -D__MACOS__ -D__MACOS_SDKROOT__=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -mmacosx-version-min=12.0 src/py_standalone_test.cpp build/py__gen.o -L/opt/local/lib -L/usr/local/opt/gperftools/lib -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -ldl -ltcmalloc
ld: warning: directory not found for option '-L/opt/local/lib'
dsymutil build/py_standalone_test
warning: (x86_64) /var/folders/hj/wvd354ps77s53v3r5tw3_6kr0000gn/T/py_standalone_test-b42d74.o unable to open object file: No such file or directory
mkdir -p build
clang++ -o build/go_standalone_test -I./src -I./gen -g -ggdb -g3 -O3 -std=c++17 -fno-omit-frame-pointer -D__HAS_LLVM_SYMBOLIZER__ -D__LLVM_SYMBOLIZER_PATH__=/usr/local/opt/llvm/bin/llvm-symbolizer -D__MACOS__ -D__MACOS_SDKROOT__=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -mmacosx-version-min=12.0 src/go_standalone_test.cpp build/go__gen.o -L/opt/local/lib -L/usr/local/opt/gperftools/lib -L/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib -ldl -ltcmalloc
ld: warning: directory not found for option '-L/opt/local/lib'
dsymutil build/go_standalone_test
warning: (x86_64) /var/folders/hj/wvd354ps77s53v3r5tw3_6kr0000gn/T/go_standalone_test-c14e11.o unable to open object file: No such file or directory

Seems the problem is the "ld: warning: directory not found for option '-L/opt/local/lib'" part but the binaries were built:

> tree build
build
├── cc__gen.o
├── common__data_gen.o
├── data__gen.o
├── datacc
├── datacc.dSYM
│   └── Contents
│       ├── Info.plist
│       └── Resources
│           └── DWARF
│               └── datacc
├── datacc.o
├── datacc_main.o
├── gen_test_bin
│   ├── go__gen_test
│   ├── go__gen_test.dSYM
│   │   └── Contents
│   │       ├── Info.plist
│   │       └── Resources
│   │           └── DWARF
│   │               └── go__gen_test
│   ├── py__gen_test
│   └── py__gen_test.dSYM
│       └── Contents
│           ├── Info.plist
│           └── Resources
│               └── DWARF
│                   └── py__gen_test
├── go__gen.o
├── go_standalone_test
├── go_standalone_test.dSYM
│   └── Contents
│       ├── Info.plist
│       └── Resources
│           └── DWARF
│               └── go_standalone_test
├── langcc
├── langcc.dSYM
│   └── Contents
│       ├── Info.plist
│       └── Resources
│           └── DWARF
│               └── langcc
├── langcc.o
├── langcc_emit.o
├── langcc_encode.o
├── langcc_lexer.o
├── langcc_main.o
├── langcc_normalize.o
├── langcc_parser_attr.o
├── langcc_parser_cps.o
├── langcc_parser_flatten.o
├── langcc_parser_lr.o
├── meta__gen.o
├── py__gen.o
├── py_standalone_test
├── py_standalone_test.dSYM
│   └── Contents
│       ├── Info.plist
│       └── Resources
│           └── DWARF
│               └── py_standalone_test
├── unittest_lang
├── unittest_lang_main.o
├── unittest_lib
└── unittest_lib_main.o

The make install ran without problems

> sudo make install
Password:
mkdir -p /usr/local/bin
mkdir -p /usr/local/include
cp build/datacc /usr/local/bin/
cp build/langcc /usr/local/bin/
cp src/langcc_util.hpp /usr/local/include/
cp src/langcc_rt.hpp /usr/local/include/

Then most tests passed but 3 which failed:

[000:04:49.628031] [success] attr_basic1
[000:04:49.890931] [success] unreach2
[000:04:50.303145] [success] write_concat_alt2
[000:04:52.536100] [success] gen_alt_basic03
[000:05:03.958407] [success] golang_assoc_mini6
[TIMEOUT] golang_assoc_mini7_opaque[TIMEOUT] right_rec_nested_concat[TIMEOUT] golang_field_star2_succ[000:05:03.971889]
[000:05:03.971889]  ===== Summary: 82 succeeded, 3 failed.
[000:05:03.971889]
[000:05:03.972140] --  ===== Failure(timeout): golang_assoc_mini7_opaque
[000:05:03.972140]
[000:05:03.972163] --  ===== Begin stdout[golang_assoc_mini7_opaque]
[000:05:03.972163]
[000:05:03.972336]
[000:05:03.972336] --  ===== End stdout[golang_assoc_mini7_opaque]
[000:05:03.972361] --  ===== Begin stderr[golang_assoc_mini7_opaque]
[000:05:03.972361]    [000:04:49.638483] -- Performing initial validation and tabulation
[000:05:03.972361]    [000:04:49.639976] -- Compiling lexer
[000:05:03.972361]    [000:04:49.646063] -- Compiling parser: tabulating symbols
[000:05:03.972361]    [000:04:49.646272] -- Compiling parser: inferring attributes
[000:05:03.972361]    [000:04:49.649761] -- Compiling parser: symbol iteration 1 (0 triggers)
[000:05:03.972361]    [000:04:49.674939] -- SLR(1) NFA: 78 vertices, 102 edges
[000:05:03.972361]    [000:04:49.683206] -- SLR(1) subset NFA: 32 vertices, 40 edges
[000:05:03.972361]    [000:04:49.683635] -- Compiling parser: symbol iteration 2 (4 triggers)
[000:05:03.972361]    [000:04:49.716314] -- SLR(1) NFA: 102 vertices, 138 edges
[000:05:03.972361]    [000:04:49.731255] -- SLR(1) subset NFA: 41 vertices, 64 edges
[000:05:03.972361]    [000:04:49.731744] -- Compiling parser: symbol iteration 3 (8 triggers)
[000:05:03.972361]    [000:04:49.764352] -- SLR(1) NFA: 103 vertices, 135 edges
[000:05:03.972361]    [000:04:49.782810] -- SLR(1) subset NFA: 42 vertices, 71 edges
[000:05:03.972361]    [000:04:49.785625] -- Compiling parser: final lookaheads
[000:05:03.972361]    [000:04:49.802990] -- Compiling parser: constructing LR NFA
[000:05:03.972361]    [000:04:49.814310] -- Compiling parser: LR NFA subset construction
[000:05:03.972361]    [000:04:49.829737] -- Compiling parser: no LR conflicts
[000:05:03.972361]    [000:04:49.829837] -- LR NFA: 103 vertices, 135 edges
[000:05:03.972361]    [000:04:49.829887] -- LR DFA: 42 vertices, 71 edges
[000:05:03.972361]    [000:04:49.834801] -- Generating AST datatype definitions
[000:05:03.972361]    [000:04:49.963097] -- Generating AST writer definitions
[000:05:03.972361]    [000:04:49.968080] -- Generating AST parser definitions
[000:05:03.972361]    [000:04:50.087304] -- Generating global definitions
[000:05:03.972361]    [000:04:50.130529] -- Running: clang++ -D __MACOS__ -D __MACOS_SDKROOT__=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o build/gen_test_bin/golang_assoc_mini7_opaque__gen_test -g -ggdb -g3 -std=c++17 -fno-omit-frame-pointer -mmacosx-version-min=12.0 -ldl -I ./build/gen_test_src -I ./src build/gen_test_src/golang_assoc_mini7_opaque__gen.cpp build/gen_test_src/golang_assoc_mini7_opaque__gen_test.cpp
[000:05:03.972361]
[000:05:03.972628] --  ===== End stderr[golang_assoc_mini7_opaque]
[000:05:03.972656] --  ===== Failure(timeout): golang_field_star2_succ
[000:05:03.972656]
[000:05:03.972677] --  ===== Begin stdout[golang_field_star2_succ]
[000:05:03.972677]
[000:05:03.972696] --  ===== End stdout[golang_field_star2_succ]
[000:05:03.972714] --  ===== Begin stderr[golang_field_star2_succ]
[000:05:03.972714]    [000:04:52.543648] -- Performing initial validation and tabulation
[000:05:03.972714]    [000:04:52.545163] -- Compiling lexer
[000:05:03.972714]    [000:04:52.550179] -- Compiling parser: tabulating symbols
[000:05:03.972714]    [000:04:52.550434] -- Compiling parser: inferring attributes
[000:05:03.972714]    [000:04:52.551872] -- Compiling parser: symbol iteration 1 (0 triggers)
[000:05:03.972714]    [000:04:52.558818] -- SLR(1) NFA: 30 vertices, 29 edges
[000:05:03.972714]    [000:04:52.561307] -- SLR(1) subset NFA: 20 vertices, 19 edges
[000:05:03.972714]    [000:04:52.561520] -- Compiling parser: symbol iteration 2 (2 triggers)
[000:05:03.972714]    [000:04:52.568145] -- SLR(1) NFA: 32 vertices, 31 edges
[000:05:03.972714]    [000:04:52.570685] -- SLR(1) subset NFA: 20 vertices, 19 edges
[000:05:03.972714]    [000:04:52.571452] -- Compiling parser: final lookaheads
[000:05:03.972714]    [000:04:52.574728] -- Compiling parser: constructing LR NFA
[000:05:03.972714]    [000:04:52.577612] -- Compiling parser: LR NFA subset construction
[000:05:03.972714]    [000:04:52.580907] -- Compiling parser: no LR conflicts
[000:05:03.972714]    [000:04:52.581012] -- LR NFA: 32 vertices, 31 edges
[000:05:03.972714]    [000:04:52.581067] -- LR DFA: 20 vertices, 19 edges
[000:05:03.972714]    [000:04:52.582440] -- Generating AST datatype definitions
[000:05:03.972714]    [000:04:52.701561] -- Generating AST writer definitions
[000:05:03.972714]    [000:04:52.704663] -- Generating AST parser definitions
[000:05:03.972714]    [000:04:52.733748] -- Generating global definitions
[000:05:03.972714]    [000:04:52.754709] -- Running: clang++ -D __MACOS__ -D __MACOS_SDKROOT__=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o build/gen_test_bin/golang_field_star2_succ__gen_test -g -ggdb -g3 -std=c++17 -fno-omit-frame-pointer -mmacosx-version-min=12.0 -ldl -I ./build/gen_test_src -I ./src build/gen_test_src/golang_field_star2_succ__gen.cpp build/gen_test_src/golang_field_star2_succ__gen_test.cpp
[000:05:03.972714]
[000:05:03.972989] --  ===== End stderr[golang_field_star2_succ]
[000:05:03.973017] --  ===== Failure(timeout): right_rec_nested_concat
[000:05:03.973017]
[000:05:03.973041] --  ===== Begin stdout[right_rec_nested_concat]
[000:05:03.973041]
[000:05:03.973061] --  ===== End stdout[right_rec_nested_concat]
[000:05:03.973080] --  ===== Begin stderr[right_rec_nested_concat]
[000:05:03.973080]    [000:04:50.308527] -- Performing initial validation and tabulation
[000:05:03.973080]    [000:04:50.310751] -- Compiling lexer
[000:05:03.973080]    [000:04:50.315677] -- Compiling parser: tabulating symbols
[000:05:03.973080]    [000:04:50.315891] -- Compiling parser: inferring attributes
[000:05:03.973080]    [000:04:50.317060] -- Compiling parser: symbol iteration 1 (0 triggers)
[000:05:03.973080]    [000:04:50.322055] -- SLR(1) NFA: 22 vertices, 22 edges
[000:05:03.973080]    [000:04:50.324101] -- SLR(1) subset NFA: 15 vertices, 15 edges
[000:05:03.973080]    [000:04:50.324791] -- Compiling parser: final lookaheads
[000:05:03.973080]    [000:04:50.327382] -- Compiling parser: constructing LR NFA
[000:05:03.973080]    [000:04:50.329500] -- Compiling parser: LR NFA subset construction
[000:05:03.973080]    [000:04:50.331776] -- Compiling parser: no LR conflicts
[000:05:03.973080]    [000:04:50.331879] -- LR NFA: 22 vertices, 22 edges
[000:05:03.973080]    [000:04:50.331936] -- LR DFA: 15 vertices, 15 edges
[000:05:03.973080]    [000:04:50.333164] -- Generating AST datatype definitions
[000:05:03.973080]    [000:04:50.474111] -- Generating AST writer definitions
[000:05:03.973080]    [000:04:50.478388] -- Generating AST parser definitions
[000:05:03.973080]    [000:04:50.498812] -- Generating global definitions
[000:05:03.973080]    [000:04:50.519858] -- Running: clang++ -D __MACOS__ -D __MACOS_SDKROOT__=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o build/gen_test_bin/right_rec_nested_concat__gen_test -g -ggdb -g3 -std=c++17 -fno-omit-frame-pointer -mmacosx-version-min=12.0 -ldl -I ./build/gen_test_src -I ./src build/gen_test_src/right_rec_nested_concat__gen.cpp build/gen_test_src/right_rec_nested_concat__gen_test.cpp
[000:05:03.973080]
[000:05:03.973279] --  ===== End stderr[right_rec_nested_concat]
[000:05:03.973304]
[000:05:03.973304]  ===== Summary: 82 succeeded, 3 failed.
[000:05:03.973304]
[000:05:03.973367] Succeeded:
[000:05:03.973391]   arith1_prec1
[000:05:03.973408]   arith_naive2_alt
[000:05:03.973423]   arith_prec1
[000:05:03.973440]   attr_basic1
[000:05:03.973454]   attr_cps
[000:05:03.973469]   attr_iter_cps2
[000:05:03.973483]   attr_nested_pos1b
[000:05:03.973496]   attr_nested_pos1c
[000:05:03.973511]   attr_nested_pos2
[000:05:03.973525]   attr_nested_pos3
[000:05:03.973538]   attr_nested_pos4
[000:05:03.973551]   attr_predict_multi
[000:05:03.973566]   attr_single
[000:05:03.973580]   basic_a
[000:05:03.973595]   basic_ab_concat
[000:05:03.973610]   basic_block_comment1
[000:05:03.973624]   basic_cps1
[000:05:03.973638]   basic_cps2b
[000:05:03.973651]   basic_optional
[000:05:03.973664]   basic_parens
[000:05:03.973677]   basic_rep_list1
[000:05:03.973691]   basic_rep_list6
[000:05:03.973704]   basic_rep_list7
[000:05:03.973718]   basic_rep_nonzero
[000:05:03.973733]   comment
[000:05:03.973747]   cpp_sub_prec_unary2
[000:05:03.973762]   cpp_sub_typed_decl1
[000:05:03.973776]   expr_alias_name1
[000:05:03.973791]   flatten_basic01
[000:05:03.973805]   gen_alt_basic02
[000:05:03.973820]   gen_alt_basic03
[000:05:03.973834]   gen_alt_basic04
[000:05:03.973849]   gen_alt_basic05
[000:05:03.973865]   gen_alt_basic08
[000:05:03.973878]   gen_alt_basic09
[000:05:03.973893]   golang_assoc_mini2_space1
[000:05:03.973909]   golang_assoc_mini3
[000:05:03.973924]   golang_assoc_mini4
[000:05:03.973940]   golang_assoc_mini5
[000:05:03.973954]   golang_assoc_mini6
[000:05:03.973969]   golang_assoc_mini7_alias1
[000:05:03.973986]   golang_chan
[000:05:03.974001]   golang_field_star1
[000:05:03.974016]   golang_if_func_assoc2
[000:05:03.974031]   ident_ab_kw
[000:05:03.974046]   inline_nested2
[000:05:03.974061]   inline_perf1a
[000:05:03.974078]   inline_perf2a
[000:05:03.974093]   left_rec
[000:05:03.974107]   left_rec_01_mutual
[000:05:03.974122]   left_rec_03_mutual_quasi
[000:05:03.974138]   left_rec_04_prec_basic
[000:05:03.974156]   left_rec_nested
[000:05:03.974170]   lexer_adv00_pop_emit
[000:05:03.974185]   lexer_adv01_match_history
[000:05:03.974203]   lexer_adv02_match_history
[000:05:03.974219]   mem_branch2
[000:05:03.974233]   mem_branch3
[000:05:03.974246]   mem_branch4
[000:05:03.974260]   mem_branch5
[000:05:03.974275]   mem_branch_iter
[000:05:03.974289]   mem_branch_multi1
[000:05:03.974304]   prec_attr_ctx2
[000:05:03.974320]   prec_basic_paren_fail
[000:05:03.974337]   prec_basic_succ1
[000:05:03.974351]   prec_large
[000:05:03.974365]   prec_mixed_fixity2_rev
[000:05:03.974381]   prec_mixed_fixity3_fail
[000:05:03.974398]   prec_with_attr1
[000:05:03.974412]   rep2
[000:05:03.974426]   repcount
[000:05:03.974441]   right_rec_alt
[000:05:03.974455]   unreach1
[000:05:03.974469]   unreach2
[000:05:03.974484]   write_concat_alt1
[000:05:03.974499]   write_concat_alt2
[000:05:03.974514]   ws_sig_01_comment_lit
[000:05:03.974530]   xfail_conflict_sub
[000:05:03.974544]   xfail_lexer_conflict
[000:05:03.974559]   xfail_lexer_multi_push
[000:05:03.974577]   xfail_parser_compound_id
[000:05:03.974593]   xfail_parser_non_lr
[000:05:03.974615]
[000:05:03.974615] Failed:
[000:05:03.974631]   golang_assoc_mini7_opaque
[000:05:03.974648]   golang_field_star2_succ
[000:05:03.974815]   right_rec_nested_concat

Didn't tried to generate something yet.
Any suggestions should I check?

[QUESTION] Do you support "look ahead" operation?

Eg. Match everything till '\n'|'*/'|eof

Generate compiler sources in Rust

People using Rust to create their compilers would appreciate this tool generating sources files in the Rust language, instead of C++.

[manual] `-` missing.

It looks like the - syntax that can be found, for example, here:

https://github.com/jzimmerman/langcc/blob/6601d082cfa402cd02a5f2852da3a2603dfe8791/examples/lox/lox.lang#LL17C8-L17C8

is not being explained by the manual. I feel like it can be used for disambiguation purposes on the lexical level, but my attempts to use it have not been successful yet.

I think it would be great if the manual could explain the purpose of the - construct.

Consider LALR lookahead sets for determining CPS triggers.

langcc/src/langcc_parser_lr.cpp

Line 1843 in 6601d08

// Iteratively determine CPS triggers by SLR conflicts.

Similar to #41, it seems like this would benefit from considering LALR lookahead sets.

Related work: BNF Converter

When I stumbled across this project I got reminded of BNFC. I couldn't find any reference to BNFC in the manual or technical report, so I thought I'd mention it here.

-ltcmalloc

I hid this behind a macro, so I can build with make LIBTCMALLOC=

tcmalloc.patch.gz

libtcmalloc is in libgoogle-perftools-dev

How to create an instruction to print a variable ?

All is in the title.