mna / pigeon

Command pigeon generates parsers in Go from a PEG grammar.

pigeon - a PEG parser generator for Go

The pigeon command generates parsers based on a parsing expression grammar (PEG). Its grammar and syntax is inspired by the PEG.js project, while the implementation is loosely based on the parsing expression grammar for C# 3.0 article. It parses Unicode text encoded in UTF-8.

See the godoc page for detailed usage. Also have a look at the Pigeon Wiki for additional information about Pigeon and PEG in general.

Releases

• v1.0.0 is the tagged release of the original implementation.
• Work has started on v2.0.0 with some planned breaking changes.

Github user @mna created the package in April 2015, and @breml is the package's maintainer as of May 2017.

Release policy

Starting of June 2023, the backwards compatibility support for pigeon is changed to follow the official Go Security Policy.

Over time, the Go ecosystem is evolving.

On one hand side, packages like golang.org/x/tools, which are critical dependencies of pigeon, do follow the official Security Policy and with pigeon not following the same guidelines, it was no longer possible to include recent versions of these dependencies and with this it was no longer possible to include critical bugfixes. On the other hand there are changes to what is considered good practice by the greater community (e.g. change from interface{} to any). For users following (or even enforcing) these good practices, the code generated by pigeon does no longer meet the bar of expectations. Last but not least, following the Go Security Policy over the last years has been a smooth experience and therefore updating Go on a regular bases feels like duty that is reasonable to be put on users of pigeon.

This observations lead to the decision to follow the same Security Policy as Go.

Installation

Provided you have Go correctly installed with the $GOPATH and $GOBIN environment variables set, run:

$ go get -u github.com/mna/pigeon

This will install or update the package, and the pigeon command will be installed in your $GOBIN directory. Neither this package nor the parsers generated by this command require any third-party dependency, unless such a dependency is used in the code blocks of the grammar.

Basic usage

$ pigeon [options] [PEG_GRAMMAR_FILE]

By default, the input grammar is read from stdin and the generated code is printed to stdout. You may save it in a file using the -o flag.

Example

Given the following grammar:

{
// part of the initializer code block omitted for brevity

var ops = map[string]func(int, int) int {
    "+": func(l, r int) int {
        return l + r
    },
    "-": func(l, r int) int {
        return l - r
    },
    "*": func(l, r int) int {
        return l * r
    },
    "/": func(l, r int) int {
        return l / r
    },
}

func toAnySlice(v any) []any {
    if v == nil {
        return nil
    }
    return v.([]any)
}

func eval(first, rest any) int {
    l := first.(int)
    restSl := toAnySlice(rest)
    for _, v := range restSl {
        restExpr := toAnySlice(v)
        r := restExpr[3].(int)
        op := restExpr[1].(string)
        l = ops[op](l, r)
    }
    return l
}
}


Input <- expr:Expr EOF {
    return expr, nil
}

Expr <- _ first:Term rest:( _ AddOp _ Term )* _ {
    return eval(first, rest), nil
}

Term <- first:Factor rest:( _ MulOp _ Factor )* {
    return eval(first, rest), nil
}

Factor <- '(' expr:Expr ')' {
    return expr, nil
} / integer:Integer {
    return integer, nil
}

AddOp <- ( '+' / '-' ) {
    return string(c.text), nil
}

MulOp <- ( '*' / '/' ) {
    return string(c.text), nil
}

Integer <- '-'? [0-9]+ {
    return strconv.Atoi(string(c.text))
}

_ "whitespace" <- [ \n\t\r]*

EOF <- !.

The generated parser can parse simple arithmetic operations, e.g.:

18 + 3 - 27 * (-18 / -3)

=> -141

More examples can be found in the examples/ subdirectory.

See the package documentation for detailed usage.

Contributing

See the CONTRIBUTING.md file.

License

The BSD 3-Clause license. See the LICENSE file.

pigeon's Issues

Document global state handling in combination with recover/throw

As mentioned in #52 (comment), the documentation should reflect the fact, that if global state is used in grammars with recover/throw, it is the responsibility of the author/grammar to set the global state back to a valid state in the case of a recovery.

Feature request: aggregate values for labels appearing multiple times in expression

A lot of boilerplate code is needed to extract a list of elements that appear in different parts of an expression.

If the same label appears in more than once in an expression, the underlying value of that label should be an []interface{}, where each element is the value of the labeled field in the order it appears in the text.

Additionally, if a label appears within a repeating subexpression, the resulting list should contain the repeating elements (as opposed to a list of the repeating elements).

For example, the rule:

non_empty_string_array <- "[" elms:StringLit (_ "," _ elms:StringLit)* _ "]" _ ";" {
return elms, nil}

when parsing this input:

[ "a", "b", "c" ]

should return a slice of length 3.

This would make it possible to solve a lot of very common problems without any code at all, and make this great project even more powerful.

wrong pigeon gets installed?

Sometimes when I install pigeon, I get the wrong version. "pigeon -h" talks about goimports and the generated parser fails to compile with a number of standard libraries not found.

I don't understand why this happens nor how to force the newer pigeon. Can you please point me?

Help Required for developing expression evaluation parser!

Hello,

Hope you are doing great! Thank you very much for developing such a wonderful library.

For one of our project (https://github.com/maniartech/x/), we are developing expression evaluator in Golang. We have used expr library for now. But would like to replace with our own evaluator/parser. We do not want to write from the scratch. The pigeon seems to provide perfect solution for our requirements. This seems to be easy. I started with changing calculator parser provided with the pigeon library. However, it is not working out.

I am new to PEG ecosystem, can anyone help me develop this parser! I am open to start a new MIT licensed open source project for this expression evaluator.

Thanks and Regards.

The expression evaluator has got following syntax!

-a-----a-----b-----b----------c----d------------l-------c----e---
SUM(AVERAGE(varA, obj.key, 10 + (2 + 3), [1, 23, 4]), 10 - 2, x)
-------------e-----e----------e--------------e-----------e----e---

-a---o-k---v----k---v----k------v--------
KEYS({ a: true, b: false, c: [1, 2, 3] })
------------t---------t-------l------------

- a: Function Name
- b: Variables, it should be JSON like variable with dot notation support
- c: Expression, Could be mathematical (+ - * /), comparison(== != > etc..), boolean (| & ~) etc...
- d: Bracketed Expression
- e: Comma separated function parameters
- o: JSON like object with nested object and array support
- k: The object key
- v: The object value
- l: The list (array) value in object 
- t: The boolean value

Accessing filename during parsing

I have a use case where I want to propagate the filename into the AST nodes generated by my parser.

Is there a way to access the filename during parsing? It seems it would be easy enough to put the filename on the current struct. Any suggestions?

Thanks!

Support annotating return types next to rules

the proposed syntax would be:

RuleName TYPES <- "grammar"

where TYPES is anything that is a valid return type for a go function. e.g. (string,error), int16 or even nothing (like in func main()).

This return type would then be copied exactly to the generated code.

[Feature Request] Add Expression name to wanted when parsing error occured

When I use pigeon, parsing error information is very important to me.

Metric "METRIC" = left:ValidString+ "::" right:ValidString+ {
// some code...
return metric, nil
}

ValidString = ':'? [a-zA-Z0-9_.-]+ {
return string(c.text), nil
}

For my use case, when I type something not correct, I want to get the expression name "METRIC". And then I will give a metric list to user. The metric list is about to change all the time so it can't be added to peg file.
The generated program will give detailed literal options in the expected list. But I want the optional expression name when parsing error occured.

Thanks in advance.

ParseFile swallows parse errors

This line causes the err return value of .parse() below it to be overridden with the result of closing the file.

Not sure what the best fix would be. It could either ignore the result, or only set err if it's nil to begin with, or return both errors glued together but that seems a bit overkill.

Great project, by the way - it's saved me a ton of time and is very easy to work with!

Accessing labels defined in parent expression from subexpressions

Hi,

I'm having trouble acccessing a label defined in the parent expression from a subexpression.
I read another issue where it is mentioned that:
"...labels are allowed in subexpressions, but they are available to code blocks in those subexpressions only. It's very similar to lexical scoping in programming languages, if that helps. Open a new scope and declare a var, that var is visible within that scope, but not in its parent scope."

So I'm trying to use this scoping intuition as follows:

{
    package main
}
A <- 
    ("B" b:B
        ( "C" c:C
            {return b + c, nil}
            /
          "D" d:d
            {return b + d, nil}
        ))
/
    ( "E"  )
{
    return "E", nil
}
B <- "B"{return "B", nil}
C <- "C"{return "C", nil}

But the generated "onA..." function doesn't have a 'b' parameter:

func (c *current) onA7(c interface{}) (interface{}, error) {
return b + c, nil
}

Is there a way to access the value of b from this subexpression?

No match found when it should match a later type

Hello! I have built a (relatively simple) parser to provide a shorthand syntax that expands to JSON or YAML for command-line utilities using Pigeon. Keep in mind I've never made a parser before! 😄

The PEG file essentially takes key/value pairs and turns them into a list of AST-like nodes. The value can consist of Null / Bool / Float / Int / String and it works great except for one strange case that I have discovered and noted in danielgtaylor/openapi-cli-generator#13:

# Works fine
$ j foo: bar.baz.blah
{
  "foo": "bar.baz.blah"
}

# Works fine
$ j foo: 1.2
{
  "foo": 1.2
}

# Fails!
$ j foo: 1.2.3
panic: stdin:1:9 (8): no match found, expected: ",", [ \t\r\n], [0-9] or EOF

goroutine 1 [running]:
main.main.func1(0xc0000c8c80, 0xc00009a400, 0x2, 0x2)
	/Users/dan/Projects/src/github.com/danielgtaylor/openapi-cli-generator/j/main.go:26 +0x2ec
github.com/spf13/cobra.(*Command).execute(0xc0000c8c80, 0xc000092190, 0x2, 0x2, 0xc0000c8c80, 0xc000092190)
	/Users/dan/Projects/pkg/mod/github.com/spf13/[email protected]/command.go:766 +0x2cc
github.com/spf13/cobra.(*Command).ExecuteC(0xc0000c8c80, 0x122527a, 0x20, 0xc000090a80)
	/Users/dan/Projects/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2fd
github.com/spf13/cobra.(*Command).Execute(0xc0000c8c80, 0x121e86b, 0x6)
	/Users/dan/Projects/pkg/mod/github.com/spf13/[email protected]/command.go:800 +0x2b
main.main()
	/Users/dan/Projects/src/github.com/danielgtaylor/openapi-cli-generator/j/main.go:53 +0x2f2

What I don't understand is how this is happening.

Float definition: Float ← [0-9]+ '.' [0-9]+
Int definition: Int ← [0-9]+
String definition: String ← [^,}]*

Based on the grammar I would think that the float begins to match, then fails at the second .. The int will match until the first ., and then it should fall back to a normal string. Any help you can give would be greatly appreciated.

Invalid detection replacement char AnyMatcher

replacement char incorrectly detected as an encoding error in AnyMatcher.
Peg: Test <- "[" s:. "]"
Input: [�]
Expect: Match
Output: 2016/09/08 23:05:22 stdin:1:1 (0): no match found
Debug trace:

1:1:0: parseRule Test [U+005B '[']
1:1:0: parseActionExpr [U+005B '[']
1:1:0: parseSeqExpr [U+005B '[']
> 1:1:0: parseLitMatcher [U+005B '[']
< 1:2:1: parseLitMatcher [U+FFFD '?']
> 1:2:1: parseLabeledExpr [U+FFFD '?']
> 1:2:1: parseAnyMatcher [U+FFFD '?']
< 1:2:1: parseAnyMatcher [U+FFFD '?']
< 1:2:1: parseLabeledExpr [U+FFFD '?']
> 1:2:1: restore [U+FFFD '?']
< 1:1:0: restore [U+005B '[']
< 1:1:0: parseSeqExpr [U+005B '[']
< 1:1:0: parseActionExpr [U+005B '[']
< 1:1:0: parseRule Test [U+005B '[']

Probably problem with replacement chat also exist in CharClassMatcher, but need tests

[Feature Request] Operator Precedence Climbing

{
//------ start
package main

type CompExpr struct {
    left string
    op string
    right string
}

type LogicExpr struct {
    left interface{}
    op string
    right interface{}
}

func main() {
    if len(os.Args) != 2 {
        log.Fatal("Usage: calculator 'EXPR'")
    }
    got, err := ParseReader("", strings.NewReader(os.Args[1]))
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%#v\n", got)
}

// ------ end
}

Input <- expr:Expr EOF {
    return expr, nil
}

Expr <- LogicExpr / Atom

LogicExpr <- _ atom:Atom _ op: LogicOp _ expr: Expr _ {
    return  LogicExpr {left : atom, op : op.(string), right: expr}, nil
}

Atom <- '(' expr:Expr ')' {
    return expr, nil
} / _ field: Ident _ op:BinOp _ value:Value _{
    return CompExpr{left : field.(string), op: op.(string), right : value.(string)}, nil
}

LogicOp <- ("and" / "or"){
    return string(c.text), nil
}

BinOp <- ("!=" / ">=" / "<=" / "=" / "<>" / ">" / "<") {
    return string(c.text),nil
}

Ident <- [a-zA-Z][a-zA-Z0-9]* {
    return string(c.text),nil
}

Value <- [0-9]+ {
    return string(c.text),nil
}

_ "whitespace" <- [ \n\t\r]*

EOF <- !.

The right recursion grammar will auto-generate right association

If pigeon can implement precedence climbing or something else, it will be very convenient ~

Build failure only when --optimize-grammar is set; grammar with not referenced rule

The following grammar results in build failure, if generated with --optimize-grammar. The same grammar builds with out problem, if the option is not set.

Minimal example:

X ← z:Z
Y ← Z
Z ← ('Z' { return nil, nil })?

Build error:

(*parser).callonZ3 undefined (type *parser has no method callonZ3)

Branch wip-vm

@mna There is an stale branch wip-vm and I am wondering, what this is all about and if it would be useful to reactivate the work on this branch again. What is your opinion on this? Do you mind to give some insights?

UTF-8 replacement character blocks code generation and runtime

I am parsing biological scientific names detected in Biodiversity Heritage Library. Due to OCR errors quite many of them contain 'bad' characters that had been replaced by '�', so I need to add the character to a parsing rule.

However when I try to generate the Go code with pigeon, the process stalls indefinitely. If I try to add the '�' to the rule in already generated code, the runtime execution in turn gets stalled indefinitely. Setting AllowInvalidUTF8(bool) option does not make a difference. The rule in question looked like this:

NameLowerChar <- LowerChar / LowerCharExtended / MiscodedChar

MiscodedChar <- '�' {
  addWarn(c, OCRErrorWarn)
  return c.text, nil
}

LowerCharExtended <- [æœſàâåãäáçčéèíìïňññóòôøõöúùüŕřŗššşž] {
  addWarn(c, badCharsWarn)
  return c.text, nil
}

LowerChar <- [a-zë] {
  return c.text, nil
}

How to achieve quantity {n} in regular expression?

I found that there are only ? * + in grammar/pigeon.peg.SuffixedExpr, how to achieve quantity {n}?

Now I use ? to achieve, is there a better way? Such as using \d\d\d\d?\d?to achieve \d{3,5}.

The exact count: {5}
\d{5} denotes exactly 5 digits, the same as \d\d\d\d\d.

The example below looks for a 5-digit number:

alert( "I'm 12345 years old".match(/\d{5}/) ); // "12345"
We can add \b to exclude longer numbers: \b\d{5}\b.

The range: {3,5}, match 3-5 times
To find numbers from 3 to 5 digits we can put the limits into curly braces: \d{3,5}

alert( "I'm not 12, but 1234 years old".match(/\d{3,5}/) ); // "1234"
We can omit the upper limit.

Then a regexp \d{3,} looks for sequences of digits of length 3 or more:

alert( "I'm not 12, but 345678 years old".match(/\d{3,}/) ); // "345678"
Let’s return to the string [phone].

A number is a sequence of one or more digits in a row. So the regexp is \d{1,}:

 let str = "[phone]";

let numbers = str.match(/\d{1,}/g);

alert(numbers); // 7,903,123,45,67

From https://javascript.info/regexp-quantifiers

Should not allow left recursion grammar to pass conversion

eg:

{
//------ start
package main

func main() {
    if len(os.Args) != 2 {
        log.Fatal("Usage: calculator 'EXPR'")
    }
    got, err := ParseReader("", strings.NewReader(os.Args[1]))
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%#v\n", got)
}

// ------ end
}

Input <- expr:Expr EOF {
    return expr, nil
}

Expr <- _ Expr _ LogicOp _ Expr _/ _ Value _

LogicOp <- ("and" / "or") {
    return string(c.text), nil
}

Value <- [0-9]+ {
    return string(c.text),nil
}

_ "whitespace" <- [ \n\t\r]*

EOF <- !.

go run main.go "1 and 1"

Will cause dead loop

Document incompatibility of global state with memoization

Update documentation to reflect the incompatibility of global state with memoization.

see #52 (comment)

Line numbers in errNoMatch

Hi Martin,

The line numbers in errNoMatch errors do not make sense to me. Any help or pointers would be appreciated.

I started to reproduce this with a minimal test case but then realized that I could use an invalid .peg file and run pigeon over it to produce the same error...

For example...

$ cat blah.peg

Output:

xxx

(yes, a bunch of newlines followed by "xxx" followed by newline).

Then run pigeon over the file to generate the parser...

$ pigeon blah.peg
parse error(s):
 blah.peg:2:0 (0): no match found

Line 2 doesn't seem to be the problem here. I must be missing something!

Cut a new release

Update: Looks like my problem was just running go get -u github.com/mna/pigeon. That brings up another request which is to cut a new release of pigeon so that users can pin to a specific version instead of master.

The peg file in test/thrownrecover does not appear to be valid. I was trying to implement similar error handling and it was failing. I then tried to run pigeon thrownrecover.peg within that directory and I get the following error:

parse error(s):
 thrownrecover.peg:1:1 (0): no match found

Partial matches keep captured variables on the stack

Hello,

First, Pigeon is awesome. The most well documented and maintained PEG generator for Go that I've found. Thanks!

The following example produces foo.foo, where (IIUC) it should produce (nil).foo:

{
package main

func main() {
  ast, err := Parse("STDIN", []byte("foo"))
  if err != nil {
    fmt.Printf("error: %s\n", err)
    return
  }
  fmt.Printf("%+v\n", ast)
}
}

TableRef <- (database:Id '.')? table:Id    { return fmt.Sprintf("%v.%v", database, table), nil }
Id <- [a-z]+                               { return c.text, nil }

I'm assuming this occurs because the optional sub-expression partially matches, populates the stack, but does not clean up when the rest of the match fails.

Thanks,
Alec

How to pass in data into parser

In peg.js I'm able to pass data into the parser via an options variable. Is there an example of how I could do the same here?

Backtrack-aware state does not restore correctly

Hello,

As I understand it, the backtrack-aware state store (current.state) should follow the parser's backtracking and by the end of the parsing, it should only hold the data written in state code blocks of valid matches (#{ ... }).

However, currently pigeon will only restore the state in a handful of cases:

• After an Action code block
• After an And or Not predicate code block
• After a Choice expression that did not match

The first two seem correct, as the state store is read-only in all code blocks but the state-specific one. However, the third is insufficient to rollback the state in all cases where the parser backtracks.

A minimal example of how it can fail is provided in https://github.com/mna/pigeonstatebug (you can just run make and it will run with the sample input "f#\n" and should count 8 newlines).

This is because the parser can backtrack in any kind of rule, e.g. given that rule:

// match any whitespace followed by a 'Z'
_ <- [ \t\n]*
Z_ <- _ 'Z'

With this input: " A", the _ rule will match, but not the 'Z', so the parser will backtrack but not the state (because this is a sequence expression).

I'm working on it today and will try to provide a pull request. I think the state should clone and restore everywhere where the parser saves and restores the position (so that it truly follows the backtracking of the parser - in addition to restoring after the non-state code blocks of course).

Thanks,
Martin

Incorrect memoization for predicate conditional rules

If rule contain code predicate returning everytime different result on one text position then memoize can break logic

TestCase peg : https://github.com/RedSkotina/pigeon/blob/bugfix-memoization-predicate/test/memoize_predicate/memoize_predicate.peg
Input string: "aa"
Expected output: ["cond2" "cond1"]
Actual output(with enabled memoize ): no match found

as i see solutions here:

• not memoize rule which has predicate code as childrens according grammar
• not memoize rule if some flag returned through parse call on children rules.

For implementation may require redesign of parse code, so i decide ask your opinion.

Encoding error REPLACEMENT CHARACTER

i have rule
Comment = (("//" (!NL .)* &NL) / ("/" (!"/" .)* "*/"))
and take error rule Comment: invalid encoding on source which contain REPLACEMENT CHARACTER (utf-8 0xEF 0xBF 0xBD) (�)

I look up workaround.

Remove -track-fail flag from usage text

The usage text for the cli shows a -track-fail flag that doesn't exists

Proposal on How to Maintain the State of the Parser in Pigeon

Current Situation

State within the Parser

Originally the state of the parser mainly consisted of the current position of the cursor within the parsed data. Additionally, a list of errors, which occurred during the parsing, is kept by the parser, which could be seen as state as well.

With the addition of the GlobalStore feature (current.globalStore), a central place to maintain arbitrary data during parsing was added. For example this could be used to provide the parser with configuration data.

With the current implementation of Pigeon it is not possible to parse inputs, where some kind of state has to be maintained. The most prominent example is: significant white space parsing (e.g. Python source code).

Go Code Blocks in the Parser

Currently, Pigeon allows to embed Go code in the following ways:

1. Initializer Code Block: Go code block, which is copied as-is into the generated parser, required to generate proper Go source files (as a minimum requirement, this code block has to provide the package statement).
2. Action Expression: contains Go code to control what value is returned as the result of a parsing rule. This code block is turned into a method on the *current type.
3. AndCode Expression / NotCodeExpression: the code predicate are similar to the normal predicate expressions (&{} and !{}). Rather than performing a look-ahead, they execute Go code to evaluate, if the processing of the current parsing tree should continue or if the parser should backtrack. This code blocks are turned into methods on the *current type.

With the existing code blocks it is possible to initialize the parser, specify the result of an expression or a rule and to control the flow of the parser. What currently is missing is, a way to change the state of the parser.

Currently, to workaround this limitation, the AndCodeExpr (&{}) is used in the examples for Global State as well as in the PR for the labeled failures, which is semantically not the right way to do this.

Proposal

State within the Parser

• current.globalStore — is a general global store for the user to store arbitrary key, value pairs that they need to manage and that they do not want tied to the backtracking of the parser. This is only modified by the user and never rolled back by the parser. It is always up to the user to keep this in a consistent state.
• current.state — is a store for arbitrary key,value pairs that the user wants to be tied to the backtracking of the parser. This is always restored if a parsing rule fails.
• current.pos, current.text — are variables managed only by the parser and the user is only allowed to read but not update. This is always restored and managed by the parser. The user is not allowed to modify this.
• parser.errs (ErrList) — list of errors, occurred during parsing. Errors are added by the parser (e.g. "grammar has no rule", "invalid encoding") as well as by the user (e.g. return true, errors.New(...) in AndCodeExpression). This is never rolled back by the parser.

State Change Expression

A new expression type State Change Expression (#{}) is introduced. The state change expression operates on *current and is allowed to change the internal state of the parser. This includes to:

• Change the global store (add, modify and remove values)
• Change the global state (add, modify and remove values)
• Add/raise parsing errors (without resulting the current parsing rule to fail, the same as &{ return true, errors.New("New error") } )

With this new expression in place, the modifiable elements of the parser state must no longer be changed by the AndCodeExpressions, NotCodeExpressions and ActionExpression, as the state is considered read-only in those expressions. Therefore those changes are reverted.

Backtracking / Restore old State / Memoization

If the parser performs a backtracking, the following elements of the state are restored to their old values before an alternative parsing tree is tried:

• Current position of the parser (current.pos) and the respective data (current.text)
• Global state (current.state)

This also has to work, if memoization is used.

Open Questions

• Should the state change expression be used to add/raise parsing errors?
• Should the ErrList be restored as well, if a backtracking is performed? (The farthestFailure must not be restored, because to keep this information is the whole purpose of this feature).
• What is the best way to return from the parser, if the parsing it self is successful, but still there are errors reported (this is possible with the labeled failures PR, especially if used for robust parsing).

Unicode character class match

Hi, I'm probably doing something wrong, but couldn't find any reference in the docs. I've added a rule to match any Unicode character class, digits or hyphens:

Term ← _ [\p{L}\d_-]+ _ {
  println(string(c.text))
}

But when running pigeon -o example.peg.go example.peg I got the error:

rule CharClassEscape: invalid escape character

If I try to escape the regex (adding an extra slash) it does compile, but the resulted file doesn't generate the expected match for Unicode characters. And as I don't need to escape \n it shouldn't be an issue I think.

Any ideas on what I'm doing wrong? 😄

Wish error output format were recognized by emacs compilation-mode & next-error

Some seem to be but others aren't. I haven't figured out the pattern nor how to fix it.

These are the errors from the generated parser, btw.

cloneState/restoreState reuses state if no keys have been set

This test fails since the same empty state is returned from cloneState multiple times.

func TestState(t *testing.T) {
	var p parser
	p.emptyState = make(storeDict)
	p.cur.state = make(storeDict)
	p.restoreState(p.cloneState())
	c := &p.cur

	backup := p.cloneState()

	c.state["foo"] = true

	p.restoreState(backup)
	if len(p.cur.state) > 0 {
		t.Fatalf("leaking state! %#v", p.cur.state)
	}
}

No stack trace for panics in handlers

I have a bug somewhere in my code resulting in a nil pointer dereference, but unfortunately I have no idea where it is because pigeon converts panics into errors:

%!s(PANIC=runtime error: invalid memory address or nil pointer dereference)

Some way of not capturing panics would be nice I think.

Latest pigeon fails to parse examples/indentation/indentation.peg

I ran:

go get -u github.com/mna/pigeon
curl -s https://raw.githubusercontent.com/mna/pigeon/master/examples/indentation/indentation.peg | pigeon

and I got the output:

parse error(s):
 stdin:1:1 (0): no match found

This is surprising, because I expect the examples provided with pigeon to parse correctly with pigeon.

(Context: I was trying to use the #{} state-change block syntax, and I couldn't get it to work, so I thought I'd look at one of the examples that use it; only the example didn't work either. This may or may not be related to the failure to parse 🤷‍♂️)

Proposed optimization for parseLitMatcher

Hi there,

I use pigeon for philandstuff/dhall-golang and I started doing some profiling since I noticed that some expressions were quite slow to parse.

After running a profiler, I noticed that a lot of time was being spent in parseLitMatcher, and in particular in strconv.AppendQuote. I manually hacked my generated parser to cache the AppendQuote result on the litMatcher struct:

type litMatcher struct {
	pos        position
	val        string
	ignoreCase bool
	want_      string // added here
}

// ...

func (lit *litMatcher) want() string {
	if lit.want_ != "" {
		return lit.want_
	}
	ignoreCase := ""
	if lit.ignoreCase {
		ignoreCase = "i"
	}
	val := string(strconv.AppendQuote([]byte{}, lit.val)) + ignoreCase // wrap 'lit.val' with double quotes
	lit.want_ = val
	return val
}

func (p *parser) parseLitMatcher(lit *litMatcher) (interface{}, bool) {
	start := p.pt
	for _, want := range lit.val {
		cur := p.pt.rn
		if lit.ignoreCase {
			cur = unicode.ToLower(cur)
		}
		if cur != want {
			p.failAt(false, start.position, lit.want())
			p.restore(start)
			return nil, false
		}
		p.read()
	}
	p.failAt(true, start.position, lit.want())
	return p.sliceFrom(start), true
}

This cut my test case runtime by nearly a third, from 5.99s to 4.18s, and specifically cut my parseLitMatcher runtime from 2.43s to 0.09s.

Would you accept a PR with this optimization? I'm not sure how easily it will generalize so I'm happy to hear critical feedback.

Export parserError and errList

I have a package with the parser generated by pigeon and in several separate (main) packages the tools, using the generated parser.

My problem now is, parserError and errList are not exported, which prevents me from using the suggested (https://godoc.org/github.com/PuerkitoBio/pigeon#hdr-Error_reporting) error handling code in my main packages.

In https://godoc.org/github.com/PuerkitoBio/pigeon#hdr-API_stability the returned error types are defined, which prevents to change these types (and their fields) to be exported.

What do you suggest in this case? Are there other solutions?

Syntax Highlighting in Vim

I've created a syntax highlighting Vim plugin for pigeon grammar files, https://github.com/jasontbradshaw/pigeon.vim:

I thought it might be useful to point others to it, possibly by adding it to the README if anyone finds it interesting and/or useful.

If not, no worries! Just wanted to throw it out there.

Here's a sample of what it looks like in the jellybeans color scheme:

errors during compilation of grammar.go are reported as grammar.go rather than grammar.peg

This is a nuisance when used, eg, with emacs compilation mode.

lex/yacc/byacc/bison/et all use line number preprocessor lines in the generated output to refer back to the original source file even when it's the C compiler that is reporting the error. I don't know how to do that in go but I presume that it must be possible. If not, it should be.

Certain inputs take an extremely long time to parse

Hello!

First of all, thank you very much for maintaining this project!

I'm hoping that someone can provide a bit of guidance. I apologize in advance for not having a minimal test case to reproduce this issue.

The issue

I've been doing some fuzz testing on OPA and I ran into one case where certain inputs would cause the program to hang and then crash. Here's a snippet of the crash:

program hanged (timeout 10 seconds)

SIGABRT: abort
PC=0x45766f m=0 sigcode=0

goroutine 1 [running]:
runtime.aeshashbody()
        /tmp/go-fuzz-build473666132/goroot/src/runtime/asm_amd64.s:917 +0x5f fp=0xc0041a78f8 sp=0xc0041a78f0 pc=0x45766f
runtime.mapassign_faststr(0x764fc0, 0xc0041a7a20, 0xc0005d1fb0, 0x3, 0xc0118f8d68)
        /tmp/go-fuzz-build473666132/goroot/src/runtime/map_faststr.go:202 +0x62 fp=0xc0041a7960 sp=0xc0041a78f8 pc=0x4135d2
github.com/open-policy-agent/opa/ast.(*parser).parse(0xc00049a180, 0xa53e00, 0x0, 0x0, 0x0, 0x0)
        /tmp/go-fuzz-build473666132/gopath/src/github.com/open-policy-agent/opa/ast/parser.go:4362 +0x272 fp=0xc0041a7b50 sp=0xc0041a7960 pc=0x6dc782
github.com/open-policy-agent/opa/ast.Parse(0x0, 0x0, 0xc0001c95e8, 0x8, 0x8, 0xc000527cb8, 0x2, 0x2, 0xc0002f4460, 0x0, ...)
        /tmp/go-fuzz-build473666132/gopath/src/github.com/open-policy-agent/opa/ast/parser.go:3784 +0x98 fp=0xc0041a7ba8 sp=0xc0041a7b50 pc=0x6da558
github.com/open-policy-agent/opa/ast.ParseStatements(0x0, 0x0, 0xc0001c95e0, 0x8, 0xc0001c95e0, 0x8, 0x200000003, 0xc000000300, 0xc000022000, 0xc000527df8, ...)
        /tmp/go-fuzz-build473666132/gopath/src/github.com/open-policy-agent/opa/ast/parser_ext.go:468 +0x173 fp=0xc0041a7d50 sp=0xc0041a7ba8 pc=0x6e62b3
github.com/open-policy-agent/fuzz-opa.Fuzz(0x7f734c798000, 0x8, 0x200000, 0x3)

The crash above occurs here: https://github.com/open-policy-agent/opa/blob/master/ast/parser.go#L4362

I modified the code to print the size of the maxFailExpected slice and found that it grew to very large sizes in pathological cases. For example the input {{{{{{{{ takes 3.5s to parse (error) and the slice holds around 3,000,000 elements.

Expected behaviour

It's not clear whether much can be done about this. In the case of OPA, we don't display the expected values (because we found them too noisy to be helpful) so disabling the code that generates them is an option, however, I'm not sure that would resolve the problem because valid inputs with a similar structure also take a very long time to parse (e.g., {{{{{{{{}}}}}}}} takes ~1.5s before succeeding.)

The PEG file is here: https://github.com/open-policy-agent/opa/blob/master/ast/rego.peg

The vendored version is bb0192c.

Any suggestions would be appreciated.

Debugging?

Is there a description or a walkthrough of the output of the Debug() option anywhere?

I'm going crazy trying to understand what this thing is doing. I really miss the shift/reduce tables from yacc. They were tedious but the inevitably led to the problem. With pigeon, I'm doing a lot of guessing and a very large number of fmt.Fprintf's trying to guess and weed out possibilities for where the problem might lie.

Today I'm going to have to start creating secondary test grammers in an attempt to try to isolate the problem. Some debug help would certainly be useful.

Choice expression limit

Hi all,

Apologies, I haven't tracked down this issue exactly. I have a choice expression with about 35 choices. The parser is able to match up to about the 24th choice, but doesn't match anything further. See the example code below.

On this line in += "<F7><bS><HOMEOn><insert><rightsuper>", note that <insert> is matched correctly, whereas <rightsuper> is not. The choice expression in question is SpecialKey. If you instead remove a few of the expressions, then <rightsuper> matches correctly.

Please let me know if I can provide any more information, or if anyone has any convenient work-arounds. I've tested that this exists with v1.0.0 and latest. Thanks!

{
package main

func main() {
    in := "<wait><wait10><wait1s><wait1m2ns>"
    in += "foo/bar > one"
    in += "<fOn> b<fOff>"
    in += "<F7><bS><HOMEOn><insert><rightsuper>"
    got, err := ParseReader("", strings.NewReader(in))
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("%s\n", got)
}
}

Input <- expr:Expr EOF {
    return expr, nil
}

Expr <- l:( Wait / CharToggle / Special / Literal)+ {
    return l, nil
}

ExprEnd = ">"

ExprStart = "<"

Wait = ExprStart "wait" duration:( Duration / Integer )? ExprEnd {
    switch t := duration.(type) {
    case time.Duration:
        return t, nil
    case int64:
        return time.Duration(t) * time.Second, nil
    default:
        return time.Second, nil
    }
}

CharToggle = ExprStart lit:(Literal) t:(On / Off) ExprEnd {
    return fmt.Sprintf("%s(%s)", t, lit), nil
}

Special = (SpecialToggle / SpecialExpr)

SpecialExpr = ExprStart s:SpecialKey ExprEnd {
    return fmt.Sprintf("(%s)", s), nil
}

SpecialToggle = ExprStart s:(SpecialKey) t:(On / Off) ExprEnd {
    return fmt.Sprintf("S%s(%s)", t, s), nil
}

Number = '-'? Integer ( '.' Digit+ )? {
    return string(c.text), nil
}

Integer = '0' / NonZeroDigit Digit* {
    return strconv.ParseInt(string(c.text), 10, 64)
}

Duration = ( Number TimeUnit )+ {
    return time.ParseDuration(string(c.text))
}

SpecialKey = "bs"i / "del"i / "enter"i / "esc"i / "f1"i / "f2"i / "f3"i / "f4"i
            / "f5"i / "f6"i / "f7"i / "f8"i / "f9"i / "f10"i / "f11"i / "f12"i / "return"i
            / "tab"i / "up"i / "down"i / "left"i / "right"i / "spacebar"i / "insert"i
            / "home"i / "end"i / "pageUp"i / "pageDown"i / "leftAlt"i / "leftCtrl"i
            / "leftShift"i / "rightAlt"i / "rightCtrl"i / "rightShift"i / "leftSuper"i
            / "rightSuper"i

NonZeroDigit = [1-9]
Digit = [0-9]
TimeUnit = ("ns" / "us" / "µs" / "ms" / "s" / "m" / "h")
Literal = .
On = "on"i
Off = "off"i

_ "whitespace" <- [ \n\t\r]*

EOF <- !.

output:

$ pigeon -o boot_command.go boot_command.pigeon; go run boot_command.go
[1s 10s 1s 1m0.000000002s f o o / b a r   >   o n e On(f)   b Off(f) (F7) (bS) SOn(HOME) (insert) < r i g h t s u p e r >]

Bad peg or bad input causes out of memory error

I spent the morning experimenting with pigeon. Very cool. My toy was a skeleton XML parser / pretty printer. I hacked on it enough to get a feel for pigeon.

One problem I ran into was if I fed an incomplete XML document into it (such as "bar"), it would crash with:

fatal error: runtime: out of memory

This was my peg file:

{
package main
}

Document <- el:Element EOF {
  return el, nil
}

Element <- _ oTag:OpeningTag body:(Element / TextNode)* cTag:ClosingTag _ {
  el := Element{name: oTag.(string)}

  for _, c := range body.([]interface{}) {
    el.children = append(el.children, c.(prettyPrinter))
  }
  return el, nil
}

OpeningTag <- '<' tag:TagName '>' {
  return tag, nil
}

ClosingTag <- "</" tag:TagName '>' {
  return tag, nil
}

TagName <- [a-z]i [a-z0-9]i* {
  return string(c.text), nil
}

TextNode <- [^<]+ {
  return TextNode(c.text), nil
}

_ "whitespace" <- [ \n\t\r]*

EOF <- !.

I don't know if this is a bug in my peg file or a bug in pigeon, but it really doesn't like running into an unexpected EOF. If it is a bad peg file, some sort of error or warning would be nice.

If it would help, you can see the whole repo here: https://github.com/jackc/pigeon-exp-xmlpp

Suggestion: Continuous Fuzzing

Hi, I'm Yevgeny Pats Founder of Fuzzit - Continuous fuzzing as a service platform.

We have a free plan for OSS and I would be happy to contribute a PR if that's interesting.
The PR will include the following

• go-fuzz fuzzers (This is generic step not-connected to fuzzit)
• Continuous Fuzzing of master branch which will generate new corpus and look for new crashes
• Regression on every PR that will run the fuzzers through all the generated corpus and fixed crashes from previous step. This will prevent new or old bugs from crippling into master.

You can see our basic example fuzzitdev/example-go and you can see an example of "in the wild" integration google/syzkaller.

Let me know if this is something worth working on.

CCing @mna

Cheers,
Yevgeny

Build failure only when --optimize-grammar is set

Hello,

When building a particular grammar generated with --optimize-grammar, it may result in build failures. The same grammar builds correctly when this option is not set. I know this flag is documented as "experimental", but still, this is probably fixable.

I have tried to reduce the grammar to the minimum required to reproduce the issue, and the result is in the github repo https://github.com/mna/pigeon-bug-optimize-grammar .

The grammar looks like this:

Start <- List         // NOTE: changing List to X here makes the bug disappear
List <- X ( ',' X )*  // NOTE: changing X to Y here makes the bug disappear
X <- 'X' Y?           // NOTE: removing the "?" makes the bug disappear
Y <- 'Y'
  {
    // NOTE: code block required for the bug, gets rendered twice with --optimize-grammar
    return nil, nil
  }

And with the --optimize-grammar option set, the output of go build (go1.10.2 on macOS for me) is:

bug/grammar.go:43:22: (*parser).callonList4 undefined (type *parser has no method callonList4)
bug/grammar.go:94:6: (*current).onList10 redeclared in this block
	previous declaration at bug/grammar.go:82:6
bug/grammar.go:100:6: (*parser).callonList10 redeclared in this block
	previous declaration at bug/grammar.go:88:6

I'll investigate and see if I can find a fix and send a PR, but I wanted to open the issue first in case you have ideas on what may cause this.

Thanks!
Martin

Significant Whitespace Parsing

Good day, @PuerkitoBio.

Are there any plans to implement either parametrisable rules [1] or a global state [2] [3] [4]?

unable to parse curly braces

Hi!

I think that the grammar is not handling correctly curly brackets.
I created a basic example:

{
package mypegtest

var ContentString string = ""
}
Search <- .+ {
    return strings.Contains(ContentString, string(c.text)), nil
}

EOF <- !.

compile it with pigeon and test it with:

package mypegtest

import (
	"bytes"
	"testing"

	"github.com/notdodo/something/mypegtest"
)

func TestMatchSimple(t *testing.T) {
	m := [1]string{"{test}"}
	mypegtest.ContentString = "my {test}"
	_, err := mypegtest.ParseReader("", bytes.NewBufferString(m[0]))
	if err != nil {
		t.Error("failed")
	}
}

running the test I get this stacktrace:

``` runtime: goroutine stack exceeds 1000000000-byte limit runtime: sp=0xc020160370 stack=[0xc020160000, 0xc040160000] fatal error: stack overflow

runtime stack:
runtime.throw(0x54defc, 0xe)
/usr/lib/go/src/runtime/panic.go:1116 +0x72
runtime.newstack()
/usr/lib/go/src/runtime/stack.go:1034 +0x6ce
runtime.morestack()
/usr/lib/go/src/runtime/asm_amd64.s:449 +0x8f

goroutine 6 [running]:
runtime.mallocgc(0x10, 0x518e80, 0x1, 0x0)
/usr/lib/go/src/runtime/malloc.go:891 +0x9bb fp=0xc020160380 sp=0xc020160378 pc=0x40c47b
runtime.convTstring(0x54c0fd, 0x1, 0x0)
/usr/lib/go/src/runtime/iface.go:367 +0x5b fp=0xc0201603b0 sp=0xc020160380 pc=0x409d6b
github.com/notdodo/pegparse/mypegtest.(*parser).parseLitMatcher(0xc00011a000, 0x6467e0, 0x1, 0xaaaaaaaaaaaaaaaa, 0x38)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1310 +0x58 fp=0xc0201604e0 sp=0xc0201603b0 pc=0x500d88
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5119a0, 0x6467e0, 0xc00006e3a8, 0x3, 0x203001)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1144 +0xab fp=0xc020160538 sp=0xc0201604e0 pc=0x4fe24b
github.com/notdodo/pegparse/mypegtest.(*parser).parseActionExpr(0xc00011a000, 0x6467a0, 0xc007d05a10, 0xc020160640, 0x447d9c)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1171 +0x75 fp=0xc0201605f8 sp=0xc020160538 pc=0x4fe875
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5117a0, 0x6467a0, 0xc007d05a40, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1130 +0x542 fp=0xc020160650 sp=0xc0201605f8 pc=0x4fe6e2
github.com/notdodo/pegparse/mypegtest.(*parser).parseSeqExpr(0xc00011a000, 0x646720, 0x0, 0x0, 0x40bd00)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1394 +0x111 fp=0xc020160730 sp=0xc020160650 pc=0x5020c1
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b60, 0x646720, 0x0, 0x0, 0xc007d05900)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1156 +0x586 fp=0xc020160788 sp=0xc020160730 pc=0x4fe726
github.com/notdodo/pegparse/mypegtest.(*parser).parseActionExpr(0xc00011a000, 0x6466e0, 0x0, 0x0, 0xc007d05900)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1171 +0x75 fp=0xc020160848 sp=0xc020160788 pc=0x4fe875
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5117a0, 0x6466e0, 0x0, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1130 +0x542 fp=0xc0201608a0 sp=0xc020160848 pc=0x4fe6e2
github.com/notdodo/pegparse/mypegtest.(*parser).parseChoiceExpr(0xc00011a000, 0x646660, 0x53ef00, 0x1, 0xc007d059b0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1285 +0x10f fp=0xc020160918 sp=0xc0201608a0 pc=0x50079f
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5118e0, 0x646660, 0x40bdd6, 0xc00006edd0, 0x30)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1140 +0x286 fp=0xc020160970 sp=0xc020160918 pc=0x4fe426
github.com/notdodo/pegparse/mypegtest.(*parser).parseRule(0xc00011a000, 0x64a880, 0x54c645, 0x6, 0xc00006ee70)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1113 +0x10a fp=0xc0201609c0 sp=0xc020160970 pc=0x4fdf2a
github.com/notdodo/pegparse/mypegtest.(*parser).parseRuleRefExpr(0xc00011a000, 0x645ce0, 0x53ef00, 0xc00006e301, 0xc007d05980)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1386 +0x82 fp=0xc020160a28 sp=0xc0201609c0 pc=0x501e02
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b20, 0x645ce0, 0x1, 0x1, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1154 +0x2d2 fp=0xc020160a80 sp=0xc020160a28 pc=0x4fe472
github.com/notdodo/pegparse/mypegtest.(*parser).parseLabeledExpr(0xc00011a000, 0x646ae0, 0x0, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1296 +0xc8 fp=0xc020160ae8 sp=0xc020160a80 pc=0x500aa8
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511960, 0x646ae0, 0x0, 0x0, 0xc007d05800)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1142 +0x412 fp=0xc020160b40 sp=0xc020160ae8 pc=0x4fe5b2
github.com/notdodo/pegparse/mypegtest.(*parser).parseActionExpr(0xc00011a000, 0x6464e0, 0x0, 0x0, 0xc007d05800)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1171 +0x75 fp=0xc020160c00 sp=0xc020160b40 pc=0x4fe875
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5117a0, 0x6464e0, 0x0, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1130 +0x542 fp=0xc020160c58 sp=0xc020160c00 pc=0x4fe6e2
github.com/notdodo/pegparse/mypegtest.(*parser).parseChoiceExpr(0xc00011a000, 0x6461a0, 0x4a8a2a, 0xc00006e380, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1285 +0x10f fp=0xc020160cd0 sp=0xc020160c58 pc=0x50079f
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5118e0, 0x6461a0, 0x1, 0xc00006edd0, 0x73)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1140 +0x286 fp=0xc020160d28 sp=0xc020160cd0 pc=0x4fe426
github.com/notdodo/pegparse/mypegtest.(*parser).parseRule(0xc00011a000, 0x64a820, 0x54c2c4, 0x4, 0xc00006ee68)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1113 +0x10a fp=0xc020160d78 sp=0xc020160d28 pc=0x4fdf2a
github.com/notdodo/pegparse/mypegtest.(*parser).parseRuleRefExpr(0xc00011a000, 0x645a20, 0xc00011a000, 0x6467e0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1386 +0x82 fp=0xc020160de0 sp=0xc020160d78 pc=0x501e02
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b20, 0x645a20, 0x0, 0x40, 0x40)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1154 +0x2d2 fp=0xc020160e38 sp=0xc020160de0 pc=0x4fe472
github.com/notdodo/pegparse/mypegtest.(*parser).parseLabeledExpr(0xc00011a000, 0x6469e0, 0x516360, 0x678ba0, 0x447d01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1296 +0xc8 fp=0xc020160ea0 sp=0xc020160e38 pc=0x500aa8
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511960, 0x6469e0, 0x516360, 0x678ba0, 0xc020161001)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1142 +0x412 fp=0xc020160ef8 sp=0xc020160ea0 pc=0x4fe5b2
github.com/notdodo/pegparse/mypegtest.(*parser).parseSeqExpr(0xc00011a000, 0x645fe0, 0x1, 0xc007c99a39, 0x1)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1394 +0x111 fp=0xc020160fd8 sp=0xc020160ef8 pc=0x5020c1
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b60, 0x645fe0, 0x646620, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1156 +0x586 fp=0xc020161030 sp=0xc020160fd8 pc=0x4fe726
github.com/notdodo/pegparse/mypegtest.(*parser).parseActionExpr(0xc00011a000, 0x645fa0, 0x203001, 0x203001, 0x203001)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1171 +0x75 fp=0xc0201610f0 sp=0xc020161030 pc=0x4fe875
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5117a0, 0x645fa0, 0x1, 0xc00006edd0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1130 +0x542 fp=0xc020161148 sp=0xc0201610f0 pc=0x4fe6e2
github.com/notdodo/pegparse/mypegtest.(*parser).parseRule(0xc00011a000, 0x64a760, 0x54c290, 0x4, 0xc00006ee60)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1113 +0x10a fp=0xc020161198 sp=0xc020161148 pc=0x4fdf2a
github.com/notdodo/pegparse/mypegtest.(*parser).parseRuleRefExpr(0xc00011a000, 0x645de0, 0xd600000000000000, 0x0, 0xd625710ded2bf5e4)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1386 +0x82 fp=0xc020161200 sp=0xc020161198 pc=0x501e02
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b20, 0x645de0, 0x0, 0xc007d03168, 0x1)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1154 +0x2d2 fp=0xc020161258 sp=0xc020161200 pc=0x4fe472
github.com/notdodo/pegparse/mypegtest.(*parser).parseLabeledExpr(0xc00011a000, 0x646b60, 0x516360, 0x678ba0, 0x447d01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1296 +0xc8 fp=0xc0201612c0 sp=0xc020161258 pc=0x500aa8
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511960, 0x646b60, 0x516360, 0x678ba0, 0x645c01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1142 +0x412 fp=0xc020161318 sp=0xc0201612c0 pc=0x4fe5b2
github.com/notdodo/pegparse/mypegtest.(*parser).parseSeqExpr(0xc00011a000, 0x646560, 0x0, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1394 +0x111 fp=0xc0201613f8 sp=0xc020161318 pc=0x5020c1
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b60, 0x646560, 0x0, 0x0, 0xc007d05700)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1156 +0x586 fp=0xc020161450 sp=0xc0201613f8 pc=0x4fe726
github.com/notdodo/pegparse/mypegtest.(*parser).parseActionExpr(0xc00011a000, 0x646520, 0x0, 0x0, 0xc007d05700)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1171 +0x75 fp=0xc020161510 sp=0xc020161450 pc=0x4fe875
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5117a0, 0x646520, 0x0, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1130 +0x542 fp=0xc020161568 sp=0xc020161510 pc=0x4fe6e2
github.com/notdodo/pegparse/mypegtest.(*parser).parseChoiceExpr(0xc00011a000, 0x6461a0, 0x4a8a2a, 0xc00006e380, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1285 +0x10f fp=0xc0201615e0 sp=0xc020161568 pc=0x50079f
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5118e0, 0x6461a0, 0x1, 0xc00006edd0, 0x73)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1140 +0x286 fp=0xc020161638 sp=0xc0201615e0 pc=0x4fe426
github.com/notdodo/pegparse/mypegtest.(*parser).parseRule(0xc00011a000, 0x64a820, 0x54c2c4, 0x4, 0xc00006ee68)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1113 +0x10a fp=0xc020161688 sp=0xc020161638 pc=0x4fdf2a
github.com/notdodo/pegparse/mypegtest.(*parser).parseRuleRefExpr(0xc00011a000, 0x645a20, 0xc00011a000, 0x6467e0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1386 +0x82 fp=0xc0201616f0 sp=0xc020161688 pc=0x501e02
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b20, 0x645a20, 0x0, 0x40, 0x40)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1154 +0x2d2 fp=0xc020161748 sp=0xc0201616f0 pc=0x4fe472
github.com/notdodo/pegparse/mypegtest.(*parser).parseLabeledExpr(0xc00011a000, 0x6469e0, 0x516360, 0x678ba0, 0x447d01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1296 +0xc8 fp=0xc0201617b0 sp=0xc020161748 pc=0x500aa8
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511960, 0x6469e0, 0x516360, 0x678ba0, 0xc020161901)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1142 +0x412 fp=0xc020161808 sp=0xc0201617b0 pc=0x4fe5b2
github.com/notdodo/pegparse/mypegtest.(*parser).parseSeqExpr(0xc00011a000, 0x645fe0, 0x1, 0xc007c99a2c, 0x1)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1394 +0x111 fp=0xc0201618e8 sp=0xc020161808 pc=0x5020c1
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b60, 0x645fe0, 0x646620, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1156 +0x586 fp=0xc020161940 sp=0xc0201618e8 pc=0x4fe726
github.com/notdodo/pegparse/mypegtest.(*parser).parseActionExpr(0xc00011a000, 0x645fa0, 0x203001, 0x203001, 0x203001)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1171 +0x75 fp=0xc020161a00 sp=0xc020161940 pc=0x4fe875
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5117a0, 0x645fa0, 0x1, 0xc00006edd0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1130 +0x542 fp=0xc020161a58 sp=0xc020161a00 pc=0x4fe6e2
github.com/notdodo/pegparse/mypegtest.(*parser).parseRule(0xc00011a000, 0x64a760, 0x54c290, 0x4, 0xc00006ee60)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1113 +0x10a fp=0xc020161aa8 sp=0xc020161a58 pc=0x4fdf2a
github.com/notdodo/pegparse/mypegtest.(*parser).parseRuleRefExpr(0xc00011a000, 0x645de0, 0x6400000000000000, 0x0, 0x64012d7df1d65cbd)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1386 +0x82 fp=0xc020161b10 sp=0xc020161aa8 pc=0x501e02
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b20, 0x645de0, 0x0, 0xc007d03048, 0x1)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1154 +0x2d2 fp=0xc020161b68 sp=0xc020161b10 pc=0x4fe472
github.com/notdodo/pegparse/mypegtest.(*parser).parseLabeledExpr(0xc00011a000, 0x646b60, 0x516360, 0x678ba0, 0x447d01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1296 +0xc8 fp=0xc020161bd0 sp=0xc020161b68 pc=0x500aa8
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511960, 0x646b60, 0x516360, 0x678ba0, 0x645c01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1142 +0x412 fp=0xc020161c28 sp=0xc020161bd0 pc=0x4fe5b2
github.com/notdodo/pegparse/mypegtest.(*parser).parseSeqExpr(0xc00011a000, 0x646560, 0x0, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1394 +0x111 fp=0xc020161d08 sp=0xc020161c28 pc=0x5020c1
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b60, 0x646560, 0x0, 0x0, 0xc007d05500)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1156 +0x586 fp=0xc020161d60 sp=0xc020161d08 pc=0x4fe726
github.com/notdodo/pegparse/mypegtest.(*parser).parseActionExpr(0xc00011a000, 0x646520, 0x0, 0x0, 0xc007d05500)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1171 +0x75 fp=0xc020161e20 sp=0xc020161d60 pc=0x4fe875
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5117a0, 0x646520, 0x0, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1130 +0x542 fp=0xc020161e78 sp=0xc020161e20 pc=0x4fe6e2
github.com/notdodo/pegparse/mypegtest.(*parser).parseChoiceExpr(0xc00011a000, 0x6461a0, 0x4a8a2a, 0xc00006e380, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1285 +0x10f fp=0xc020161ef0 sp=0xc020161e78 pc=0x50079f
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5118e0, 0x6461a0, 0x1, 0xc00006edd0, 0x73)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1140 +0x286 fp=0xc020161f48 sp=0xc020161ef0 pc=0x4fe426
github.com/notdodo/pegparse/mypegtest.(*parser).parseRule(0xc00011a000, 0x64a820, 0x54c2c4, 0x4, 0xc00006ee68)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1113 +0x10a fp=0xc020161f98 sp=0xc020161f48 pc=0x4fdf2a
github.com/notdodo/pegparse/mypegtest.(*parser).parseRuleRefExpr(0xc00011a000, 0x645a20, 0xc00011a000, 0x6467e0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1386 +0x82 fp=0xc020162000 sp=0xc020161f98 pc=0x501e02
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b20, 0x645a20, 0x0, 0x40, 0x40)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1154 +0x2d2 fp=0xc020162058 sp=0xc020162000 pc=0x4fe472
github.com/notdodo/pegparse/mypegtest.(*parser).parseLabeledExpr(0xc00011a000, 0x6469e0, 0x516360, 0x678ba0, 0x447d01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1296 +0xc8 fp=0xc0201620c0 sp=0xc020162058 pc=0x500aa8
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511960, 0x6469e0, 0x516360, 0x678ba0, 0xc020162201)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1142 +0x412 fp=0xc020162118 sp=0xc0201620c0 pc=0x4fe5b2
github.com/notdodo/pegparse/mypegtest.(*parser).parseSeqExpr(0xc00011a000, 0x645fe0, 0x1, 0xc007c99a20, 0x1)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1394 +0x111 fp=0xc0201621f8 sp=0xc020162118 pc=0x5020c1
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b60, 0x645fe0, 0x646620, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1156 +0x586 fp=0xc020162250 sp=0xc0201621f8 pc=0x4fe726
github.com/notdodo/pegparse/mypegtest.(*parser).parseActionExpr(0xc00011a000, 0x645fa0, 0x203001, 0x203001, 0x203001)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1171 +0x75 fp=0xc020162310 sp=0xc020162250 pc=0x4fe875
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5117a0, 0x645fa0, 0x1, 0xc00006edd0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1130 +0x542 fp=0xc020162368 sp=0xc020162310 pc=0x4fe6e2
github.com/notdodo/pegparse/mypegtest.(*parser).parseRule(0xc00011a000, 0x64a760, 0x54c290, 0x4, 0xc00006ee60)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1113 +0x10a fp=0xc0201623b8 sp=0xc020162368 pc=0x4fdf2a
github.com/notdodo/pegparse/mypegtest.(*parser).parseRuleRefExpr(0xc00011a000, 0x645de0, 0x6a00000000000000, 0x0, 0x6aceeb1cd9a1b72f)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1386 +0x82 fp=0xc020162420 sp=0xc0201623b8 pc=0x501e02
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b20, 0x645de0, 0x0, 0xc007d02f28, 0x1)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1154 +0x2d2 fp=0xc020162478 sp=0xc020162420 pc=0x4fe472
github.com/notdodo/pegparse/mypegtest.(*parser).parseLabeledExpr(0xc00011a000, 0x646b60, 0x516360, 0x678ba0, 0x447d01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1296 +0xc8 fp=0xc0201624e0 sp=0xc020162478 pc=0x500aa8
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511960, 0x646b60, 0x516360, 0x678ba0, 0x645c01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1142 +0x412 fp=0xc020162538 sp=0xc0201624e0 pc=0x4fe5b2
github.com/notdodo/pegparse/mypegtest.(*parser).parseSeqExpr(0xc00011a000, 0x646560, 0x0, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1394 +0x111 fp=0xc020162618 sp=0xc020162538 pc=0x5020c1
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b60, 0x646560, 0x0, 0x0, 0xc007d05300)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1156 +0x586 fp=0xc020162670 sp=0xc020162618 pc=0x4fe726
github.com/notdodo/pegparse/mypegtest.(*parser).parseActionExpr(0xc00011a000, 0x646520, 0x0, 0x0, 0xc007d05300)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1171 +0x75 fp=0xc020162730 sp=0xc020162670 pc=0x4fe875
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5117a0, 0x646520, 0x0, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1130 +0x542 fp=0xc020162788 sp=0xc020162730 pc=0x4fe6e2
github.com/notdodo/pegparse/mypegtest.(*parser).parseChoiceExpr(0xc00011a000, 0x6461a0, 0x4a8a2a, 0xc00006e380, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1285 +0x10f fp=0xc020162800 sp=0xc020162788 pc=0x50079f
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5118e0, 0x6461a0, 0x1, 0xc00006edd0, 0x73)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1140 +0x286 fp=0xc020162858 sp=0xc020162800 pc=0x4fe426
github.com/notdodo/pegparse/mypegtest.(*parser).parseRule(0xc00011a000, 0x64a820, 0x54c2c4, 0x4, 0xc00006ee68)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1113 +0x10a fp=0xc0201628a8 sp=0xc020162858 pc=0x4fdf2a
github.com/notdodo/pegparse/mypegtest.(*parser).parseRuleRefExpr(0xc00011a000, 0x645a20, 0xc00011a000, 0x6467e0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1386 +0x82 fp=0xc020162910 sp=0xc0201628a8 pc=0x501e02
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b20, 0x645a20, 0x0, 0x40, 0x40)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1154 +0x2d2 fp=0xc020162968 sp=0xc020162910 pc=0x4fe472
github.com/notdodo/pegparse/mypegtest.(*parser).parseLabeledExpr(0xc00011a000, 0x6469e0, 0x516360, 0x678ba0, 0x447d01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1296 +0xc8 fp=0xc0201629d0 sp=0xc020162968 pc=0x500aa8
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511960, 0x6469e0, 0x516360, 0x678ba0, 0xc020162b01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1142 +0x412 fp=0xc020162a28 sp=0xc0201629d0 pc=0x4fe5b2
github.com/notdodo/pegparse/mypegtest.(*parser).parseSeqExpr(0xc00011a000, 0x645fe0, 0x1, 0xc007c99a13, 0x1)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1394 +0x111 fp=0xc020162b08 sp=0xc020162a28 pc=0x5020c1
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b60, 0x645fe0, 0x646620, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1156 +0x586 fp=0xc020162b60 sp=0xc020162b08 pc=0x4fe726
github.com/notdodo/pegparse/mypegtest.(*parser).parseActionExpr(0xc00011a000, 0x645fa0, 0x203001, 0x203001, 0x203001)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1171 +0x75 fp=0xc020162c20 sp=0xc020162b60 pc=0x4fe875
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5117a0, 0x645fa0, 0x1, 0xc00006edd0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1130 +0x542 fp=0xc020162c78 sp=0xc020162c20 pc=0x4fe6e2
github.com/notdodo/pegparse/mypegtest.(*parser).parseRule(0xc00011a000, 0x64a760, 0x54c290, 0x4, 0xc00006ee60)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1113 +0x10a fp=0xc020162cc8 sp=0xc020162c78 pc=0x4fdf2a
github.com/notdodo/pegparse/mypegtest.(*parser).parseRuleRefExpr(0xc00011a000, 0x645de0, 0x9900000000000000, 0x0, 0x99aab2f2ba5011bf)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1386 +0x82 fp=0xc020162d30 sp=0xc020162cc8 pc=0x501e02
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b20, 0x645de0, 0x0, 0xc007d02e08, 0x1)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1154 +0x2d2 fp=0xc020162d88 sp=0xc020162d30 pc=0x4fe472
github.com/notdodo/pegparse/mypegtest.(*parser).parseLabeledExpr(0xc00011a000, 0x646b60, 0x516360, 0x678ba0, 0x447d01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1296 +0xc8 fp=0xc020162df0 sp=0xc020162d88 pc=0x500aa8
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511960, 0x646b60, 0x516360, 0x678ba0, 0x645c01)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1142 +0x412 fp=0xc020162e48 sp=0xc020162df0 pc=0x4fe5b2
github.com/notdodo/pegparse/mypegtest.(*parser).parseSeqExpr(0xc00011a000, 0x646560, 0x0, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1394 +0x111 fp=0xc020162f28 sp=0xc020162e48 pc=0x5020c1
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x511b60, 0x646560, 0x0, 0x0, 0xc007d05200)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1156 +0x586 fp=0xc020162f80 sp=0xc020162f28 pc=0x4fe726
github.com/notdodo/pegparse/mypegtest.(*parser).parseActionExpr(0xc00011a000, 0x646520, 0x0, 0x0, 0xc007d05200)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1171 +0x75 fp=0xc020163040 sp=0xc020162f80 pc=0x4fe875
github.com/notdodo/pegparse/mypegtest.(*parser).parseExpr(0xc00011a000, 0x5117a0, 0x646520, 0x0, 0x0, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1130 +0x542 fp=0xc020163098 sp=0xc020163040 pc=0x4fe6e2
github.com/notdodo/pegparse/mypegtest.(*parser).parseChoiceExpr(0xc00011a000, 0x6461a0, 0x4a8a2a, 0xc00006e380, 0x0)
/home/notdodo/.go/src/github.com/notdodo/pegparse/mypegtest/mypegtest.go:1285 +0x10f fp=0xc020163110 sp=0xc020163098 pc=0x50079f
...additional frames elided...
created by testing.(*T).Run
/usr/lib/go/src/testing/testing.go:1042 +0x357

goroutine 1 [chan receive]:
testing.(*T).Run(0xc000114120, 0x54eb2e, 0x12, 0x555cb8, 0x4762c6)
/usr/lib/go/src/testing/testing.go:1043 +0x37e
testing.runTests.func1(0xc000114000)
/usr/lib/go/src/testing/testing.go:1284 +0x78
testing.tRunner(0xc000114000, 0xc000076e10)
/usr/lib/go/src/testing/testing.go:991 +0xdc
testing.runTests(0xc00000c060, 0x645220, 0x1, 0x1, 0x0)
/usr/lib/go/src/testing/testing.go:1282 +0x2a7
testing.(*M).Run(0xc000112000, 0x0)
/usr/lib/go/src/testing/testing.go:1199 +0x15f
main.main()
_testmain.go:44 +0x135
FAIL _/home/notdodo/Desktop/Edo@rdo/pegparse/mypegtest/testing 2.718s
FAIL

</details>

Just removing the curly brackets in `m` the test pass.

Feature proposal: alternate entrypoints

Hello,

At the moment, pigeon hard-codes the entrypoint (starting rule) of the generated parsers - the first rule in the grammar serves as the entrypoint. One area where this constraint is particularly painful is for tests: imagine for example that you want to test in isolation a specific area of the grammar (e.g. string literals, numbers, etc.). Currently you're left with a few less-than-ideal solutions:

• generate a fully valid input that includes the rule to test (depending on the grammar, may require non-trivial unnecessary input to be maintained in addition to the input to test, and is not an isolated test)
• split the grammar in various files (e.g. "main" grammar and as many "entrypoint" partial grammar files as necessary, and $ cat entrypoint_test1.peg main_grammar.peg > complete_grammar.peg them to generate as many different parsers as necessary). This works at the cost of many additional files and relies heavily on Makefiles.
• hack the internals of pigeon (identify the index of the "rule under test" in g.rules and swap it with the rule at index 0) - which doesn't play well with -optimize-grammar (the "rule-under-test" may have been optimized out) and is not very flexible (maybe we want to test a combination of existing rules, or make sure that the test matches the full input - that is, a given rule followed by EOF).

I'd like to propose a new command-line flag and a new Parse option to support alternate entrypoints natively. This implementation supports:

• A single grammar file (no need to split into multiple entrypoints with subsequent concatenation)
• Support for -optimize-grammar (the alternate entrypoints are treated the same as the first rule)
• Flexibility of the alternate entrypoints (not limited to other existing rules, can be an otherwise non-referenced rule)
• Opt-in feature (if no alternate entrypoints are specified, works the same as today, with no performance impact)
• Only rules specified when the parser is generated can be used as entrypoints (i.e. non-referenced rules that are not valid alternate entrypoints are still optimized out)

I have a working implementation in branch wip-testrules. As you can see among the noise of the generated files, the changes are quite targeted and do not spread all over the codebase, there's the command-line flag, the Entrypoint option, the Optimize call and the parser picks the right rule at the start of the parsing.

Let me know your thoughts about this and any comments you may have on the current implementation. For example, I chose to call the flag -alternate-entrypoints so that the first rule is always implicitly a valid entrypoint. I think that makes the whole API cleaner and has less impact on backwards compatibility (e.g. otherwise if the flag is entrypoints, it is special when not set - implies first rule - but when set it would need to include the first rule if desired), but that's just my point of view.

Thanks,
Martin

Add type safety

Dealing with interfaces can be quite burdensome. Why not add type safety to the project?

The data matched by a single character matcher type byte, and if you are matching multiple items it's always an [] of whatever type underlies it. And instead of using simple { } go blocks, have it with a bit more detail, but details that define types. Something like this: [int, error] { } that defines we'll be returning an int, and an error

Not getting []interface{} for star, plus expressions

Hi,

{
    package main

}

A <- (  "A " first:B rest1:( "," B )* rest2:C+ rest3:[a-z]+ )
{
    for i, el := range rest {
        fmt.Printf( "%d: %+v\n", i, el )
    }
    return string(c.text), nil
}

B <- "B " val:([a-z]+) {
    return string(c.text), nil
}

C <- "C " val:([0-9]+) {
    return string(c.text), nil
}

I get:

func (c *current) onA1(first, rest1, rest2, rest3 interface{}) (interface{}, error) {
    for i, el := range rest {
        fmt.Printf( "%d: %+v\n", el )
    }
    return string(c.text), nil
}

I was expecting rest1, rest2, rest3 to be slices. What am I doing wrong?

Unable to generate optimized grammar

Hello,

First, thanks a lot for maintaining this project, it's a great library! I'm currently using it in https://github.com/bytesparadise/libasciidoc and it's working really well 🙌

However, since 9fec389 was merged, I've been getting the following error when running the command below in the project's root:

$ pigeon -optimize-grammar -alternate-entrypoints PreparsedDocument,InlineElementsWithoutSubtitution,VerbatimBlock -o ./pkg/parser/asciidoc_parser.go  ./pkg/parser/asciidoc-grammar.peg
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1284d49]

goroutine 1 [running]:
main.main.func1(0x13d3500, 0xc000072020, 0xc0005fde50)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/main.go:87 +0x13d
panic(0x12f2440, 0x15f2fd0)
        /usr/local/Cellar/go/1.12.1/libexec/src/runtime/panic.go:522 +0x1b5
github.com/mna/pigeon/ast.(*grammarOptimizer).optimizeRule(0xc0004bfec0, 0x13d0d80, 0xc0003e2300, 0xc00045d080, 0xc0003e4d20)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_optimize.go:243 +0x309
github.com/mna/pigeon/ast.(*grammarOptimizer).optimize(0xc0004bfec0, 0x13d0d60, 0xc0003e4f00, 0x13d0e40, 0xc0004bfec0)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_optimize.go:182 +0x27ec
github.com/mna/pigeon/ast.(*grammarOptimizer).Visit(0xc0004bfec0, 0x13d0d60, 0xc0003e4f00, 0xc0003e4d20, 0xc0004bfec0)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_optimize.go:39 +0x3e
github.com/mna/pigeon/ast.Walk(0x13d0e40, 0xc0004bfec0, 0x13d0d60, 0xc0003e4f00)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_walk.go:20 +0x55
github.com/mna/pigeon/ast.Walk(0x13d0e40, 0xc0004bfec0, 0x13d0c80, 0xc0004dba90)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_walk.go:41 +0x43b
github.com/mna/pigeon/ast.Optimize(0xc0004dba90, 0xc0001044e0, 0x3, 0x3)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_optimize.go:464 +0x14e
main.main()
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/main.go:120 +0x106f

The grammar in my project is already quite big: https://github.com/bytesparadise/libasciidoc/blob/master/pkg/parser/asciidoc-grammar.peg and unfortunalely the stack trace does not give much information about the rule(s) that cause the error, so I can't really narrow down the grammar to a simpler form :/

Note: building and running pigeon with the previous commit works like a charm.

[Feature Request] Output rules used during parsing

Sometimes with really large grammars (such as the wikitext/parsoid peg definitions) it's really hard to debug which rules are actually being used to output the final result. It'd be really nice if pigeon could return a tree of which rules are being used to generate the final result.

`go get github.com/mna/pigeon/...` is failing

go get github.com/mna/pigeon/... is failing with the following error message (go 1.9):

# github.com/mna/pigeon/test/runeerror
runtime.main_main·f: relocation target main.main not defined
runtime.main_main·f: undefined: "main.main"