tunnelvisionlabs / antlr4cs

The original, highly-optimized C# Target for ANTLR 4

C# target for ANTLR 4

Getting Started

Step 1: Install Java (recommended)

⚠️ Starting with release 4.5.0-alpha003, users are no longer required to install the Java Runtime in order to compile .NET applications using ANTLR 4. However, installing Java will dramatically improve the performance of the code generation process. It is highly recommended, especially on developer machines where background code generation is used for IntelliSense functionality.

The C# target for ANTLR 4 uses Java for compiling applications. The resulting compiled C# applications will not require Java to be installed. You can install any of the following versions of Java to use this target.

If you already have one of the following installed, you should check to make sure the installation is up-to-date.

• Java 7 runtime environment (x86 or x64)
• Java 7 development kit (x86 or x64, provided that the JRE option is also installed during the development kit installation)
• Java 6 runtime environment (x86 or x64)
• Java 6 development kit (x86 or x64, provided that the JRE option is also installed during the development kit installation)

If no suitable version of Java could be located on the machine, the build tools will automatically fall back to using the new C# implementation of the code generation tool instead.

Step 2: Install ANTLR Language Support for Visual Studio (optional)

This step is optional, but highly recommended for users working with a version of Visual Studio that the extension supports. If you have one of the Express Editions of Visual Studio, or would like to skip this step, move on to the following step.

1. Open Visual Studio
2. Select Tools → Extensions and Updates...
3. In the left pane, select Online
4. In the top right, type ANTLR to search for extensions
5. If your version of Visual Studio is supported, the results pane will show the extension ANTLR Language Support by Sam Harwell. You can click the result and then select Download to install the extension.
6. Restart Visual Studio after installing the extension

Step 3: Update the NuGet Package Manager

For nearly all Visual Studio 2010 and newer installations

1. Open Visual Studio
2. Select Tools → Extensions and Updates...
3. In the left pane, select Updates, then select Product Updates
4. Wait for ~10 seconds while the application checks for updates (it might not notify you that it's checking in the background)
5. If an update for NuGet Package Manager is listed in the results, click to update it
6. Repeat steps 3-5 for any other sections listed under Updates in the left pane
7. If you updated NuGet, restart Visual Studio before continuing to Step 4 below

For Visual Studio 2008, and Visual C# 2010 Express Edition

These versions of Visual Studio do not support the NuGet Package Manager extension, but the C# target for ANTLR 4 does support .NET 2.0 and higher so you should be able to use the command line NuGet utility instead of working directly within Visual Studio.

Step 4: Install ANTLR 4 support in a C# project

For Visual Studio 2017

1. Create or open a C# project which will use ANTLR
2. Right click the top-level solution node in the Solution Explorer window and select Manage NuGet Packages for Solution...
3. In the upper left, choose Browse and then choose nuget.org as the Package source
4. Next to the Search box, check Include prerelease
5. In the Search box, type Antlr4 to search for the package
6. In the search results, locate and select the package called Antlr4. Verify that the name is listed as exactly Antlr4.
7. In the right pane, select the C# projects you want to use ANTLR4 by clicking their checkboxes
8. Click Install under the list of projects
9. Approve changes and accept license agreements, if prompted.

For nearly all Visual Studio 2010 - 2015 installations

1. Create or open a C# project which will use ANTLR
2. Right click the top-level solution node in the Solution Explorer window and select Manage NuGet Packages for Solution...
3. In the left pane, select Online, then select nuget.org
4. At the top of the middle pane, if a drop down says Stable Only, change the selection to Include Prerelease
5. In the top right, type Antlr4 to search for the package
6. In the search results, locate and select the package called ANTLR 4. In the right pane, verify that the Id is listed as exactly Antlr4.
7. Click install on the search result
8. Select the C# projects you want to add support for ANTLR 4 to, and click OK to update those projects

For Visual C# 2010 Express Edition

TODO: This section needs more detail.

Use the NuGet Package Manager Console to install the latest version of the Antlr4 package, as described on the following page.

ANTLR 4

For Visual Studio 2008

TODO: This section needs more detail.

A special NuGet package is available for installation in Visual Studio 2008 projects. You should be able to use the NuGet Package Manager Console to install the latest version of the Antlr4.VS2008 package, as described on the following page.

ANTLR 4 (Visual Studio 2008)

Working with ANTLR 4 in Visual Studio

The sections below may be performed multiple times in any order as necessary to add and configure the grammars used in your project.

Add a new grammar to the project

Using templates provided by the ANTLR Language Support extension

1. Right click the project (or subfolder) in Solution Explorer where the new grammar should be placed
2. Select Add → New Item...
3. In the left pane, expand Visual C# Items and select ANTLR
4. Select one of the ANTLR 4 templates and give it a name, and click Add to create the new grammar file and add it to the project

Without using the ANTLR Language Support extension

1. Right click the project (or subfolder) in Solution Explorer where the new grammar should be placed

2. Select Add → New Item...

3. In the left pane, expand Visual C# Items and select General

4. In the middle pane, select Text File

5. In the name box, type the complete name of the grammar file (including the .g4 extension), e.g. CustomLanguage.g4, and click Add to create the new file and add it to the project

6. Select File → Advanced Save Options...

7. For Encoding, select Unicode (UTF-8 without signature) - Codepage 65001

8. Click OK, and then save the file

9. Add the grammar declaration at the top of the file, e.g. the following for a grammar named CustomLanguage.g4

   grammar CustomLanguage;

10. Follow step 2 of the Add an existing grammar to the project section below to configure the build properties of the newly added grammar file

Add an existing grammar to the project

The steps for adding an existing grammar to a project are similar for users with and without the ANTLR Language Support extension installed. The primary difference between these cases involves configuring the grammar to generate C# code during the build process, a step which is automated when the ANTLR Language Support extension is installed.

1. Add the *.g4 grammar file to the C# project using one of the following methods.
   • Right click the project or folder in Solution Explorer and click Add → Existing Item...
   • Select the project in Solution Explorer, and click the toolbar button for Show All Files. Then locate the *.g4 grammar file within the project structure, right click the file, and select Include In Project.
2. If you have the ANTLR Language Support extension installed, this step is automatically performed. Otherwise, you will need to right click the grammar file in Solution Explorer and select Properties. In the properties window, configure the following items.
   • Build Action: Antlr4
   • Custom Tool: MSBuild:Compile
   • Custom Tool Namespace: The complete name of the namespace you want the generated classes to be located within. The ANTLR Language Support extension configures this field according to the root namespace configured for the C# project combined with the subfolder within the project where the grammar is located.

Configure the compile-time options for a grammar in the project

The code generation process for ANTLR grammars is configured by setting specific MSBuild properties on the grammar files included in the C# project file. The ANTLR Language Support extension for Visual Studio exposes the customizable properties for easy configuration in the Properties pane of Visual Studio, but they can also be manually configured by editing the project file.

Note: this section assumes you have already added the grammar file(s) you wish to customize to the project via the steps described above.

The following table describes the properties available for customizing the code generation process for grammars in C# projects.

1. MSBuild Property: This is the name of the XML element in the project file responsible for customizing the property
2. Display Name: This is the name of the property as it appears in the Visual Studio Properties window for users with the ANTLR Language Support extension installed
3. Value: The allowed value(s) for the property
4. Details: A description of the impact of the property on the build process

MSBuild Property	Display Name	Value	Details
Abstract	Abstract Grammar	True or False	When True, the generated lexer and/or parser classes are marked as abstract.
ForceAtn	Force ATN	True or False	When True, the generated parser will use AdaptivePredict for all decisions, including LL(1) decisions.
Listener	Generate Listener	True or False	When True, a parse tree listener interface and base class will be generated for the parLitser.
Visitor	Generate Visitor	True or False	When True, a parse tree visitor interface and base class will be generated for the parser.

Using the ANTLR Language Support extension

1. Right click the grammar file in Solution Explorer and select Properties
2. Locate the property to customize according to the Display Name column in the table above
3. Update the value as appropriate

Manually editing the project file

1. Right click the project in Solution Explorer and select Unload Project

2. Right click the project in Solution Explorer and select Edit ProjectName.csproj

3. Locate the XML element for the ANTLR grammar project item, which should look like the following

   <Antlr4 Include="CustomLanguage.g4">
     <Generator>MSBuild:Compile</Generator>
     <CustomToolNamespace>MyProject.Folder</CustomToolNamespace>
   </Antlr4>

4. Locate an existing XML element according to the MSBuild Property column in the table above, or add one if it does not already exist. For example, to generate both the parse tree listener and visitor interfaces and base classes for your parser, update the project item to resemble the following.

   <Antlr4 Include="CustomLanguage.g4">
     <Generator>MSBuild:Compile</Generator>
     <CustomToolNamespace>MyProject.Folder</CustomToolNamespace>
     <Listener>True</Listener>
     <Visitor>True</Visitor>
   </Antlr4>

5. Save and close the project file

6. Right click the project in Solution Explorer and select Reload Project

Assembly Strong Name Policy

Starting with the beta release of version 4.4.1 of the C# target, a new strong name policy is used for this target. The new policy is designed to make it easy to write libraries referencing a specific target framework (e.g. .NET 4.0), and then replace the Antlr4.Runtime.dll assembly at runtime built for a newer target framework (e.g. .NET 4.5). For authors of intermediate library, this dramatically simplifies the process of distributing libraries targeting many target frameworks. The following graph show the supported replacements which retain binary compatibility. Note that the dashed line represents a currently-untested link in the graph (see #91).

Pre-release Builds

To ensure the highest level of reliability for developers targeting final releases of particular versions of the ANTLR 4 Runtime, pre-release builds use a different strong name key than final builds. Library authors are encouraged to use strong-named assemblies when referencing the final builds, especially when code is executing in an environment where other libraries may be referencing ANTLR 4 (e.g. Visual Studio extensions).

While the strong name keys used for the build are included in the repository, users are strongly discouraged from distributing builds of the ANTLR 4 Runtime using a strong name produced by antlr4.snk. Failure to follow this rule will compromise the reliability of the millions of other users using applications that reference ANTLR on a daily basis.

Example

Suppose a library writer wishes to use ANTLR 4, and produce output which supports the .NET Framework 3.0 and newer, including the wide selection of targets offered by the Portable Class Library profile 328. In this example library, also suppose that all referenced dependencies are present in both the .NET Framework 3.0 and Profile 328, so the same code compiles in both cases without problems. This library writer only needs to produce and distribute two builds of the library: a net30 build which is built against the net30 build of ANTLR 4, and a portable-net40 build which is built against the portable-net40 build of ANTLR 4.

Now suppose an application developer wishes to use the library described previously. This application is a desktop application targeting .NET 4.5, and the application developer wishes to use the highest-performing available build of ANTLR 4. By simply installing the previous library from NuGet, an additional reference will automatically be added to the Antlr4.Runtime package. Without additional configuration, the application will build against the portable-net40 build of the library and the net45 build of ANTLR 4. This situation is fully supported (and recommended), because the Framework Compatibility graph above includes a chain extending from portable-net40 to net45.

Grammars

TODO

Custom Token Specifications (*.tokens)

TODO

Generated Code

During the build, generated code for a grammar is placed in the intermediate output directory. By default, this is obj\Debug for debug builds, and obj\Release for release builds. These generated files are managed during the build process:

• The generated files for the current build configuration are automatically included during the compilation process
• If the grammar is renamed or removed, old generated files will be deleted during the next build
• The generated files are removed during a 'clean' operation

The generated files are not intended to be included in source control, so there is no need to copy them to another location or manually include them in the project. Since each generated type is declared with the partial modifier, users wishing to extend the behavior of the generated code can do so by including a separate file in the project which has another definition for the partial type. When adding files to a project using the templates in the ANTLR Language Support extension, empty code files for this purpose are created and added to the project automatically.

Extra Features in the C# Target

TODO

Example Grammars

TODO

antlr4cs's Issues

Support running the RuleDependencyChecker as a post-build action

The Java version of the tool includes RuleDependencyProcessor, an annotation processor that statically validates the accuracy of the rule dependency annotations used throughout Java code. This processor produces compile-time errors if the dependencies are violated.

Currently, the C# target for ANTLR 4 does not provide this feature. Instead, users are required to use the RuleDependencyChecker at runtime to validate the attributes used in the assembly. The target would benefit from the following additions:

• Provide the ability to validate the rule dependency attribute usage at compile time, producing compile-time errors if the attributes are misused.
• Provide the ability to report misused rule dependency attributes in the IDE even before the project is explicitly built.

C# compiler can't compile generated lexer

CSC gives me following error without name of the file, line and column, only project name:

Error   4   An expression is too long or complex to compile

I found the reason for this error:

Unfortunately, there's not much we can do here without isolating the issue further. Usually, this error results from a deeply chained expression in your source, such as a long chain of string concatenations ("aaaa" + "bbbb" + ...). Such expressions are parsed and bound recursively in the compiler, and there is a depth at which the compiler will run out of stack space to continue parsing. The exact expression complexity supported depends on the data stashed in the compiler's stack frames, which can vary subtly between updates - if an expression in your solution was right at the boundary before, it may have tipped over.

https://connect.microsoft.com/VisualStudio/feedback/details/785173/got-error-cs1647-an-expression-is-too-long-or-complex-to-compile-in-vs2012

There is a huge string in generated lexer, over 5000 lines, which looks like this:

    public static readonly string _serializedATN =
        "\x3\xAF6F\x8320\x479D\xB75C\x4880\x1605\x191C\xAB37\x2\x37D\x2D79\b\x1"+
        "\x4\x2\t\x2\x4\x3\t\x3\x4\x4\t\x4\x4\x5\t\x5\x4\x6\t\x6\x4\a\t\a\x4\b"+
        "\t\b\x4\t\t\t\x4\n\t\n\x4\v\t\v\x4\f\t\f\x4\r\t\r\x4\xE\t\xE\x4\xF\t\xF"+
        "\x4\x10\t\x10\x4\x11\t\x11\x4\x12\t\x12\x4\x13\t\x13\x4\x14\t\x14\x4\x15"+
        "\t\x15\x4\x16\t\x16\x4\x17\t\x17\x4\x18\t\x18\x4\x19\t\x19\x4\x1A\t\x1A"+
        "\x4\x1B\t\x1B\x4\x1C\t\x1C\x4\x1D\t\x1D\x4\x1E\t\x1E\x4\x1F\t\x1F\x4 "+
        "\t \x4!\t!\x4\"\t\"\x4#\t#\x4$\t$\x4%\t%\x4&\t&\x4\'\t\'\x4(\t(\x4)\t"+
............

Any workaround would be very helpful.

Can't get the token name from XXXLexer.ruleNames with ITerminalNode.Symbol.Type

I wrote the following code.
But it can't show a correct token name if the lexer has fragment tokens.
It seems that the ruleNames field includes fragment tokens but the Type field, which derives from constant integers on the head of the lexer class, excludes them.
How can I get a correct token name?

public override object VisitTerminal(Antlr4.Runtime.Tree.ITerminalNode node) {
    if (node.Symbol.Type > 0) {
        var name = XXXLexer.ruleNames[node.Symbol.Type - 1];
        var text = node.Symbol.Text;
        Console.WriteLine(name + " : " + text);
    }
    return base.VisitTerminal(node);
}

Unable to build project

mvn fails with:

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building ANTLR 4 Target C# 4.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[WARNING] The POM for com.tunnelvisionlabs:antlr4:jar:4.0.1-SNAPSHOT is missing,
no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.288s
[INFO] Finished at: Tue Jul 30 20:37:33 CDT 2013
[INFO] Final Memory: 6M/122M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project antlr4-csharp: Could not resolve depen
dencies for project com.tunnelvisionlabs:antlr4-csharp:jar:4.0.1-SNAPSHOT: Failu
re to find com.tunnelvisionlabs:antlr4:jar:4.0.1-SNAPSHOT in https://oss.sonatyp
e.org/content/repositories/snapshots was cached in the local repository, resolut
ion will not be reattempted until the update interval of sonatype-nexus-snapshot
s has elapsed or updates are forced -> [Help 1]

Follow Visual Studio conventions for generated code.

In holding with convention (that a lot of static analysis and style analyzers use) generated code files should be started with one of the following comment blocks:

/// <auto-generated />

/// <auto-generated>
/// Any needed data
/// </auto-generated>

Also, generated code should have the filename "filename.g.cs"

These two things are key to allowing static analysis tools to ignore tool generated code.

Xml comments in Visitor classes are not accurate

This is the generated visitor class.
I'm using Antlr4.Runtime.4.1.0-alpha003\lib\net40\Antlr4.Runtime.v4.0.dll

[System.CodeDom.Compiler.GeneratedCode("ANTLR", "4.1-SNAPSHOT")]
public interface IUfqVisitor<Result> : IParseTreeVisitor<Result> {
    /// <summary>
    /// Visit a parse tree produced by <see cref="UfqParser.setAtom"/>.
    /// </summary>
    /// <param name="context">The parse tree.</param>
    /// <return>The visitor result.</return>
Result VisitSetAtom([NotNull] UfqParser.SetAtomContext context);

The
<see cref="UfqParser.setAtom"/>.

is not accurate, there is not such thing like setAtorm on my generated UfqParser class

It's just a minor issue as it causes several xml warning when I build the project

ATNSimulator deserialization errors

Hi Sam,

I'm getting deserialization errors in the ATNSimulator. The first is saying that it is expecting version 3 but getting version 5

In commit f74aabf (update to match 4.1) you changed the SerializedVersion const from 5 to 3.

If I change it to 5 I then get an error with the expected GUID.

Was there a merge issue or perhaps a newer version of the JAR?

Thanks

Copyleft issue in Antlr 4 C# target

While going over the distribution license we noticed that a file in the source tree for the C# target has a viral license attached to it, as opposed to the BSD license for the overall ANTLR project. The file in question is:

https://github.com/tunnelvisionlabs/antlr4cs/blob/master/runtime/CSharp/Antlr4.Runtime/Sharpen/CRC32.cs

The license on this file is GPL v2 and it looks like it originated from the Oracle open jdk project. Is there any chance of this being addressed in the official distribution?

C# results don't match Java results for simple grammar

I have a simple grammar. The C# version prints the message:

line 1:6 mismatched input 'parr' expecting ID

Even though it should match 'parr' to an ID. I've create a gist with all the files.

https://gist.github.com/GaryPickrell/06f4ae5d6e4a5d7596c7

Resharper support

You have already answered to this:

If you have ReSharper installed, you should note that ReSharper does not support IntelliSense for 3rd-party code generation tools. You'll need to uninstall the extension or disable its IntelliSense customizations to restore the proper behavior, or bring the issue up with that company to find a solution. If you do not have ReSharper installed, please open a separate issue for this.

But there 2 issues:
a) be real. if I have to choose what saves more of my time antlr or resharper...
b) there's enormous amount of code-generating plug-ins. MS native: T4, XSD, setting files, winForms designer. And that is only what immediately comes to mind. And they all are supported by resharper.

Actually it's quite obvious where you'd gone wrong. You should generate code into files that you include into project. That is it.

How to use ANTLR4CS without Visual Studio?

I am a Java newbie and just want to use ANTLR4+CS for a small test project which might eventually grow into something big.

I don't use Visual Studio and have no plans to alter my habits in that regard. So how do I actually get a working ANTLR4+CS combo given the set of files available on both websites?

On the same note, why don't you simply include the most recent ANTLR4CS in the official distribution of ANTLR? The current method seems a bit complex.

The ANTLR4 download page states that it supports C# as a target, but I don't seem to be able to get that to work either; it says "error(31): ANTLR cannot generate 'org.antlr.v4.codegen.CSharp4Target' code as of version 4.2.2" no matter what C# version I try to target.

Type initializer exception when creating Lexer

Hi, I am using ANTLR4.0 with C# target for a simple calculator grammar, everything is OK for generating the C# code, e.g. Lexer, parser, listener, vistor... But when start paring in the following code I got exception when creating the CalculatorLexer:

    string s = "a = 3 + 2;";
    AntlrInputStream input = new AntlrInputStream(s);
    CalculatorLexer lexer = new CalculatorLexer(input);   <- Exception throw from here!!
    CommonTokenStream tokens = new CommonTokenStream(lexer);

The Exception information is as follows:

  An unhandled exception of type 'System.TypeInitializationException' occurred in MyApp.exe
  Additional information: The type initializer for 'Calculator.CalculatorLexer' threw an exception.

Anybody can help?
Thanks

Accessing Text property from predicates

The Text property ($a.getText() in Java) seems to be unusable with antlr4cs.
Having the following grammar:

grammar Test;

compileUnit : rule EOF;
rule : a=NAME b=NAME { $a.Text == $b.Text }?;

NAME : [a-z]+;
WS : ' ' -> skip;

When I build the project I have the following error:

error AC0066: attribute 'Text' isn't a valid property in '$a.Text'
error AC0066: attribute 'Text' isn't a valid property in '$b.Text'

This looks like a bug (unless I'm doing something wrong ofc)

toStringTree() and \n

I'm running the Definitive Antlr 4 Expr example, but get extra \n in toStringTree. I've worked around it by replacing \\n to \n, which appears to be the problem:

class Program
{
    static void Main(string[] args)
    {
        AntlrInputStream input = new AntlrInputStream(File.Open("t.expr", FileMode.Open));

        ExprLexer lexer = new ExprLexer(input);
        CommonTokenStream tokenStream = new CommonTokenStream(lexer);
        ExprParser parser = new ExprParser(tokenStream);

        IParseTree tree = parser.prog();

        string output = tree.ToStringTree(parser).Replace("\\n", "\n");//Environment.NewLine);
        Console.Write(output);
        Console.Read();
    }
}

Major performance issue

I have with channels but also applies to anywhere there are a large number of tokens.

I'm using C# and have made it a 64bit program. I have a 90k test text file and I try and push any text outside a tag or inside a "literal" tag into a text channel.

The test text file look roughly like:

 {literal}

90K worth of text

{/literal}

A summarised version of the grammar is:

LITERAL: {literal}                      -> pushMode( PASS_THROUGH );

mode PASS_THROUGH;
    END_LITERAL:    '{/literal}'        -> popMode;
    PASS_THROUGH_CHAR: .                -> channel( TEXT );

The problem is that on a fast I7 processor it takes around 25 seconds to process and uses about 7GB of memory. I would have expected to only use a few megabytes and taken not much more than a second.

If I do:

CommonTokenStream stream = new CommonTokenStream(new AntlrInputStream("text"));
stream.Fill();

All the processing time and memory are used up in the fill. It's almost like each token has it own copy of the input stream not a reference to the input stream: 90K * 90K = 8Gb.

Regards
Terry

Generated method in Lexer class is a Java method

I've successfully generated the parser/lexer classes by following the directions in the readme. The Lexer class fails to compile. For the most part, the class is C# code, except for the following functions:

      // is this character followed by an identifier or
      // a dot? this is used in parsing numbers, to distinguish
      // floating-point numbers from ranges and method calls.
      public boolean followed_by_ident_or_dot() {
        CharStream cs = getInputStream();
        int nextChar = cs.LA(1);
        // KNOWN POTENTIAL ISSUE : this fn needs to be
        // aligned with the list appearing in xidstart....
        return (java.lang.Character.isUnicodeIdentifierStart(nextChar)
                || nextChar == dotChar);
      }

      // are we at the beginning of the file? This is needed in
      // order to parse shebangs.
      public boolean at_beginning_of_file() {
        return (getInputStream().index() == 0);
      }

CLS Compliance

It would be nice if the generated code was fixed to be CLS compliant.

Some examples of areas that are not CLS compliant are:

• generated properties and their backing fields whose name differs only by case
• type names with non alpha-num characters (token type names end with parenthesis)
• public members with a _ as the initial identifier's character

Can ANTLR lexers/parsers be used for VS syntax highlighting and Error List?

Hi -- I'm using Antlr4cs and it's working beautifully; apologies if this is the wrong place to post!

I'm happily developing a language and I'd like to start integrating my language into an IDE. The natural place for me feels like integrating it into Visual Studio. It occurs to me that someone might have figured out a general way to plug an ANTLR lexer into the Visual Studio syntax highlighting system, or ANTLR parsers into an MSBuild task such that errors appear in the Visual Studio error list.

Is there any kind of starting point, base package, plugin system, or similar which lets you integrate an ANTLR4CS language with Visual Studio's language support mechanisms?

Alternatively, has anyone had any success building, say, a Sublime Text build system / language def automatically from their grammar?

Antlr4BuildTasks.dll errors detecting Java path install in Windows x64 Registry

Antlr4BuildTasks.dll was failing to load the Java install path from registry.

I've finally found what reason was and I've got a fix that you can include in your code.
The problem is that there are 2 ways of accesing the Registry, in 32bits or 64 bits mode. Some keys are stored in one and some in the other.

You were using RegistryView.Default as access mode and it couldn't find the full path (I've got Windows 7 64bits)
This code will try 32 bits first and then 64 if it couldn't find it. Finally it would throw an exception if Java is not installed.
With this fix my solution is already generating files.

    private string JavaHome
    {
        get
        {

            string javaKey = "SOFTWARE\\JavaSoft\\Java Runtime Environment";

    var value =  GetJavaHomeFromRegistry(javaKey, RegistryView.Registry32);

    if(string.IsNullOrEmpty(value) )
        value = GetJavaHomeFromRegistry(javaKey, RegistryView.Registry64);

    if (string.IsNullOrEmpty(value))
        throw new FileNotFoundException("I couldn't find java install");

         return value;
        }
    }

private static string GetJavaHomeFromRegistry(string javaKey, RegistryView registryView)
    {
    using (var baseKey = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, registryView).OpenSubKey(javaKey))
        {
            if (baseKey != null)
            {
                string currentVersion = baseKey.GetValue("CurrentVersion").ToString();
                using (var homeKey = baseKey.OpenSubKey(currentVersion))
                    return homeKey.GetValue("JavaHome").ToString();
            }
        }
    return null;
    }

'Text' isn't a valid property

I'm following along with Scott Stanchfield's tutorials to help me re-learn Antlr, and I'm having some issues almost immediately.

My entire grammar is (essentially) as follows:

rule : id=IDENT { System.Console.WriteLine( $id.Text ); } ;
IDENT : 'a'..'z'+ ;

I'm getting an error whenever I build that says 'Text' isn't a valid property in '$id.Text'.

The error can be avoided by changing the property to the method GetText() in the grammar file, building, then going into the generated parser class and switching back to the property.

Antlr4ClassGenerationTask fails when there are spaces in file paths

MSBuild task Antlr4ClassGenerationTask doesn't handle spaces in file paths.

When building my project at C:\Users\Admin\Documents\Visual Studio 2012\Projects\Compiler\CompilerV4\CompilerV4.csproj

I see the build error:

Error 1 Unknown build error: Could not find or load main class Studio

The project will only compile if relocated to a path with no spaces.

Including grammar with only fragments breaks generated lexer

A grammar that contains only fragments (e.g. a file with only Unicode character classes in it) breaks the generated lexer. Putting just one (useless or overridden) rule in it fixes this.

Grammars to demonstrate the problem:

grammar Test;
import Unicode;

program : 'test' 'test' ;

WS : (UNICODE_CLASS_Zs)+ -> skip ;

 

lexer grammar Unicode;

fragment
UNICODE_CLASS_Zs    : '\u0020' | '\u00A0' | '\u1680' | '\u180E'
                    | '\u2000'..'\u200A'
                    | '\u202F' | '\u205F' | '\u3000'
                    ;

feature request: usability improvements for antlr4 language support + antlr4 build task

While working on a project that includes a grammar, I've been struggling a bit with the way that the nuget package wires everything up out-of-the-box. Everything seems to work correctly on the antlr side and the final result compiles and works.

However, when I try to create partial classes to add some functionality to my parser and visitor, Visual Studio shows me a compile error for every single member that I reference in the generated code, and Intellisense does not work for these classes either. So, I am constantly having to refer to other API docs to lookup method names instead of having my editor list them as it does with everything else. It also wreaks havoc with refactoring tools.

I did a bit of digging, and the reason this happens is that the generated code is only generated on build, and there is no mention of it in the .csproj file. Antlr4.net40.targets on line 138 is passing in \obj\Debug for the output directory. Ideally, the code would be generated in the same directory as the grammar and included in the project like this:

<Compile Include="MyGrammarParser.cs">
  <DependentUpon>MyGrammar.g4</DependentUpon>
</Compile>

Doing so would allow the editor to have access to the same code that the compiler is looking at, and then all the issues I mentioned go away. I would be happy to make the necessary changes and send a pull request if that sounds like a worth-while feature, but I haven't been able to find the source for the visual studio extension. I'm assuming it is open source.

compile time error on Recognizer.AddErrorListener

i'm able to add a custom error listener (derived from BaseErrorListener ) to my parser and it functions as expected at run time.

but get I am getting a compile time error trying to add a listener to the lexer.

Given that the "AddErrorListener" method is defined on the base recognizer and thus identical for both the parser and the lexer I find this more than a little confusing.

"cannot convert from 'CustomErrorListener' to 'Antlr4.Runtime.IAntlrErrorListener'"

i must be missing something here...

-mark

Something isn't working

I did step 1 to 5:
https://github.com/tunnelvisionlabs/antlr4cs/wiki

All perfect. Successful. No errors. Like a kitten.

Then, I imported a set of *.g files that someone else made. I verified their properties:
Build Action = Antlr3
Copy to output = Do not copy
Custom Tool = MSBuild:Compile
Custom Tool Namespace = MyCoolTool.Grammar
Debug Grammar = False
Language Target = (not set)
Profile Grammar = False

This means two things in my mind:

1. The antlr extension works
2. It should compile

Now, putting a using MyCoolTool.Grammar; produces an error. The namespace isn't there. Without that, the lexer class that's supposed to be generated, isn't recognized as being there, either. All attempts to "guess" what class it has generated (if any) fail.

From the example, from the Run() method, only the first line works:

AntlrInputStream inputStream = new AntlrInputStream("hello world\n");

Absolutely fine, but that doesn't get me anywhere.

I seems as if the grammar files remain completely untouched. As if no attempt is made to compile them. Further suspicion was raised by the build output:

1>------ Rebuild All started: Project: MyCoolTool, Configuration: Debug Any CPU ------
1>  MyCoolTool -> N:\Documents\Development\MyCoolTool\MyCoolTool\bin\Debug\MyCoolTool.exe
========== Rebuild All: 1 succeeded, 0 failed, 0 skipped ==========

I don't see antlr compiling there, or anywhere. How do I get it to do that?

Erroneous extraneous input detected in C# (but not in Java)

Hello,

I'm afraid I found a bug in Antlr4 CS 4.3.0. Using the same grammar and the same input stream, the parsing succeeds in Java but fails in C# ("extraneous input"). The traces show that the C# version consumes all the left brackets in a row, and then complains about a right bracket. The Java version works as expected.

Here is my grammar:

grammar Expr;

root                        : assignment EOF
                            ;
assignment                  : LOCAL_VARIABLE '=' expression
                            ;
expression                  : logical_and_expression
                            ;
logical_and_expression      : relational_expression ('AND' relational_expression)*
                            ;
relational_expression       : primary_expression (('<'|'>') primary_expression)*
                            ;
primary_expression          : '(' + expression + ')'
                            | UNSIGNED_INT
                            | LOCAL_VARIABLE
                            ;

LOCAL_VARIABLE              : [_a-z][_a-zA-Z0-9]*
                            ;
UNSIGNED_INT                : ('0'|'1'..'9''0'..'9'*)
                            ;
WS                          : [ \t\r\n]+ -> skip
                            ; 

And my test program (C#), showing the input stream:

string expression = "b = (((a > 10)) AND ((a < 15)))";
AntlrInputStream input = new AntlrInputStream(expression);
ExprLexer lexer = new ExprLexer(input);
CommonTokenStream tokens = new CommonTokenStream(lexer);
ExprParser parser = new ExprParser(tokens);
parser.Trace = true;
parser.root();

The Java version:

String expression = "b = (((a > 10)) AND ((a < 15)))";
ANTLRInputStream input = new ANTLRInputStream(expression);
ExprLexer lexer = new ExprLexer(input);
CommonTokenStream tokens = new CommonTokenStream(lexer);
ExprParser parser = new ExprParser(tokens);
parser.setTrace(true);
parser.root();

Your help would be appreciated.
Best regards,

Cyril.

issue with lexer rule orderering

I have run into a problem where the order of mh lexer rules is not being followed, I have tried my grammer in both antlrworks test rig and a command line application, but the following produces different tokens between the two:

HEADER: '#'     ;       
LIST: [ ]* [0-9]+ '. '  ;           
ULIST: '* '     ;               
NL : ('\r'? '\n');
WORD: ~('\n'|'\r')+;
WS: [ \t]+

in the c# version, the HEADER rule never produces a token, but in antlrworks it does

Improve error message when Java is not installed

EOF token has a wrong value of the Column property

I use ANTLR 4.2.1-alpha001.
When the last token (which is not hidden) has a not zero Column, the EOF token has the same Column value of the last token.
For example, when we parse the following C code, ANTLR 4 generates an EOF token with 1 Column value, but, it should have 0 Column value.

void main() {
}

Semantic Predict in Ambiguous assignment statement

Hi, I am trying to implement a C-like language with some extensions for other data types.

   scheduleExpression
      : date
      | FROM date TO date EVERY INT DAYS
      | symbol EQUAL scheduleExpression                // assignment for schedule type
      ;

  expression
      :constant
      | symbol
      | expression (PLUS|MINUS) expression
      | symbol EQUAL expression                              // assignment
      ;

The issue is when parsing a simple assignment like the following:

g = 3.0;
b = g;  

Note that my language only have two data types: double and schedule and no variable declaration is required. From the code script above, variable (symbol) g is a double type when it is first assigned a double value. However, on the second row, the parser does not know g is a double type variable and will parse it based on the assignment of schedule as defined in scheduleExpression since the schedule assignment is defined earlier than the double type expression. The same issue happens when g is assigned a schedule value and in the grammar the expression is defined before scheduleExpression (this case b=g is parsed as expression assignment rather than scheduleExpression assignment).

Then I tried to add a semantic predicate as follows:

// IsSchedule() added in the assignment
scheduleExpression
    : date
    | FROM date TO date EVERY INT DAYS
    | symbol EQUAL {IsSchedule()}? scheduleExpression                // assignment for schedule type
    ;

// no change for expression
expression
    :constant
    | symbol
    | expression (PLUS|MINUS) expression
    | symbol EQUAL expression                                                  // assignment
    ;

and let IsSchedule() always return false. The parser generates exceptions for d=g. I thought it should fail matching scheduleExpression assignment and try to alternatively match expression assignment, however the result is not as expected. Please help. Thanks a lot!!!

Undocumented Build process

I'm not entirely clear how to build my own version of the NuGet packages to work around bugs. I think it involves running mvn but otherwise I'm unsure. Are there any sort of instructions?

Templates not available in VS 12, Code Generation defective, Examples out of date

1. the most bad one first: All c# examples of the final usage of the generated lexer, parser and listener/walker are out of date. They use properties and methods which are not present any more

2. VS 12 with Update 4: No ANTLR Templates available after installation (can also be reached by uninstall ANTLR and reinstall...)

3. Code Generation is not recognized by IntelliSense. Errors are displayed in ErrorList which are false negative because compilation is possible

Besides all effort to produce this VS/.NET target - it is totally useless in the current state...sorry

ANTLR 4 C# (4.1.0-alpha003) with java.g4 shows warnings when parsing hex numbers

When a parser generated by ANTLR 4 C# (4.1.0-alpha003) with java.g4 parses the following Java code, it shows the following warnings.
However, a parser generated by ANTLR 4 Java 4.1 with java.g4 doesn't show them.

• Java code

  class Hello {
      void main(String[] args) {
          System.out.println((String)args[0x00]);
      }
  }
  

• Warnings

  line 4:35 no viable alternative at input '[0x00'
  line 4:35 extraneous input 'x00' expecting {'instanceof', '(', '[', ']', '.', '=', '>', '<', '?', '==', '<=', '>=', '!=', '&&', '||', '++', '--', '+', '-', '*', '/', '&', '|', '^', '%', '+=', '-=', '*=', '/=', '&=', '|=', '^=', '%=', '<<=', '>>=', '>>>='}
  

how do you use this thing from the command line

Can't find classes in VS2013 Professional

Sorry to bother you all with such a noob question. I've installed Antlr4 in VS2013, created a simple grammar and built the solution. I've aloso verified that the support files are generated in .\obj\Debug folder. All looks well and I'm ready to use this code in my own test application. Anltr4.Runtime.net45 is in my references (no other references to Antlr content). However, when I declare...

 public class MyParser : MyParserVisitor ...

The MyParserVisitor class isn't there to inherit from in the IDE. Am I missing something very basic? I suspect I'm about to kick myself or beat myself with the Antlr book I just bought :) Thanks for your help!

Combined1.g4(1,1): error AC0050: '´' came as a complete surprise to me

When I compile a simple g4 file, I've this error on line 1 caracter 1 ???

Combined1.g4(1,1): error AC0050: '´' came as a complete surprise to me

Example :

grammar Combined1;

@parser::members
{
    protected const int EOF = Eof;
}

@lexer::members
{
    protected const int EOF = Eof;
    protected const int HIDDEN = Hidden;
}

/*
 * Parser Rules
 */

compileUnit
    :   EOF
    ;

/*
 * Lexer Rules
 */

WS
    :   ' ' -> channel(HIDDEN)
    ;

I don't know where is the problem ?

When parser rule name is C# keyword, generated parser breaks

When the parser rule name happens to be a C# keyword, the generated XParser.cs file breaks. It contains reserved keywords as method names, and won't compile.

Grammar that demonstrates the problem:

grammar MyCLikeLanguage;

program : namespace* ;
namespace : 'namespace' ID '{'  '}' ;
ID : [a-z]+ ;
WS : [ \t\r\n]+ -> skip ;

<assoc=right> not generating the correct precedence

I'm not an expert with Antlr so it's entirely possible I'm doing something wrong, but given the standard example:

grammar Combined1;

expr : expr '^'<assoc=right> expr
 | expr '*' expr 
 | expr '+' expr 
 | INT 
 ;

INT: [0-9]+;

2^2^2 is parsed as (2^2)^2 instead of 2^(2^2): ([] ([0] ([0 0] 2) ^ ([7 0] 2)) ^ ([7] 2))

iOS does not allow JIT compilation -- happens in the Sharpen utilities

I've been working to integrate antlr4cs into a Unity3D project to run on iOS. Several of the classes in the Sharpen namespace rely on Just-in-Time compilation of generic types. Apple forbids using JIT compilation as a safety concern, and more likely, because it would allow developers to go around the App Store, and cut Apple out of the revenue stream.

Since Unity uses .NET v3.5, I have to use the v3.5 target of ANTLR C# that relies on the Sharpen classes to reproduce functionality not available in that version of .NET.

I ran into 3 places where JIT occurs:

1. ConcurrentDictionary uses the SplitOrderedList generic class.
2. AtomicReference uses System.Threading.Interlocked, which according the the source code listed at www.dotnetframework.org uses JIT as a hack to get around a problem the developers faced.
3. Recognizer uses the ConsoleErrorListener generic class.

To fix (1), I found a class from Lucene.Net.Support.Compatibility that replaces ConcurrentDictionary without needing JIT support.

To fix (2), I commented out the Interlocked.CompareExchange () function call, and handled the variable update manually with a spin lock around it.

I didn't spend a lot of time fixing (3), just commented out that line for now.

Would you guys be interested in rolling changes 1 and 2 into the main branch? I do not feel comfortable doing so, as I am new to this code base, and there may be unintended side affects I'm not aware of.

Unable To Retrieve The Text Of The Current Token From The Context

I've been working on implementing a test grammar loosely based on SQL and i've been attempting to retrieve text of tokens - the documented way (according to the book) that I'm seeing in Java versions is within the Listener -

        public override void EnterBasicId(EtlSqlParser.BasicIdContext context)
        {
            base.EnterBasicId(context);
            Console.WriteLine(context.PARTIALID().GetText());
            Console.WriteLine(parser.CurrentToken.Text);
        }

The first output does not output the identifier but the second output using the parser does ... is there something I'm missing here?

basicId : PARTIALID ;

PARTIALID 
    : ESCAPEDID
    | IDUNESCAPED
    ;

fragment ESCAPEDID: '[' IDUNESCAPED ']' ;
fragment IDUNESCAPED : IDBEGIN (INT | IDBEGIN)+ ;

fragment IDBEGIN : [a-zA-Z_$];
fragment INT : [0-9]+ ; 

ANTLR4 not showing in Add New Items in Visual Studio 2013

I've just upgrade to Visual Studio 2013 Professional from 2010 and I find that ANTLR4 won't show in the Add New Items dialogue. I used NuGet to install ( and uninstall ) it many times it and I've tried both the stable version and pre-release version and both will not show. If I open and existing 2010 project that already uses ANTLR the properties of both the lexer and parser are the default Microsoft build options not the ANTLR4 build options and still no ANTLR4 in the Add New Items dialogue.

Any Suggestions as I really need to put the 2010 project together in .NET 4.5.

Regards
Terry

Question about redistribution of different packages

I'm writing an open source library using ANTLRcs as it's lexer/parser generator. This library is, in-turn, used by other libraries in the same solution (a remote debugger facility) and this might even expand more in the future. It supports .NET 3.5+ and will in the future probably be ported to cover .NET 4 portable class library.

At the moment, - it's very dirty, I know - I'm distributing the Antlr4.Runtime.v3.5.dll as a unique runtime for my library whatever the final target is, and not releasing separate versions for the various frameworks versions. This is bad, as it will cause conflicts if a user wants to use antlr for other reasons.

I, of course, want to clean this up, but I'm a bit reclutant to go the full depth - it would mean 2*N projects, nugets, etc where N is the number of different versions Antlr4.Runtime is packaged (at the time of writing, 9), when the scenarios I really want to cover are two at most (net35 and portable-net40).

Do you have any suggestion on how to avoid this proliferation or how to work around it ?

Is there a reason to have so many versions apart the obvious separation of standard and portable targets ?

Thanks in advance.

tokens file is searched for in wrong path with option tokenVocab=SBLexer;

To reproduce:
g4 files in project subfolder, e.g. < ProjectRoot >\Antlr

parser grammar SBParser;
options { tokenVocab=SBLexer; }

Error 2 cannot find tokens file 'obj\Debug\SBLexer.tokens'

The cs and tokens files are generated in folder obj\Debug\Antlr

Workaround: need to copy the file to that folder in order for the solution to build, need to remember this if one comes back to the projects 2 months later and modifies the lexer grammar, as the updated tokens file is still generated in the obj\Debug\Antlr folder, and the copy one made to the obj\Debug folder might be outdated.

ANTLR + Xamarin?+Windows Phone

I have been looking into using ANTLR in a Xamarin project, targeting Windows Phone, iOS, and Android. I found your ANTLR Runtime PCL here. But alas, it appears it uses classes that are not supported on Windows Phone (at a minimum, ConcurrentDictionary and SortedDictionary.) So it appears I can't use this to target Windows Phone, at least not without changes.

William Jockusch

ParseRuleContext start and stop value possibly swapped if rule empty

Hi,

in my production codebase I noticed a strange behavior which breaks my calculation of the text span length of a given ParseRuleContext. I isolated the problem in this grammer:

start:  START+ middle END+;
middle: BTOY*;

START: 'A';
BTOY:  [B-Y];
END:   'Z';

WS : (' ')+ -> skip ; // doesn't affect the problem

If I run it with a small Visitor like this:

    class Visitor : bugtestBaseVisitor<string>
    {
        public override string VisitStart(bugtestParser.StartContext context)
        {
            PrintInfo(context, "Start ");
            return base.VisitStart(context);
        }

        public override string VisitMiddle(bugtestParser.MiddleContext context)
        {
            PrintInfo(context, "Middle");
            return base.VisitMiddle(context);
        }

        private void PrintInfo(ParserRuleContext context, string name)
        {
            Console.WriteLine("{0} has span: {1} to {2} thats {3} chars",
                                name, 
                                context.start.StartIndex, 
                                context.stop.StopIndex,
                                SpanLength(context));
            Console.WriteLine("start: " + context.start.ToString());
            Console.WriteLine("stop : " + context.stop.ToString());
        }

        private int SpanLength(ParserRuleContext ctx)
        {
            return ((ctx.stop.StopIndex + 1) - ctx.start.StartIndex);
        }
    }

It produces the following output. Notice how the start and stop property are in reverse order if middle matches an empty string.
I would expect that the span of middle if empty would be something like: startIndex=42 stopIndex=41 thus the calculation of the length would result in 0. (Assuming (stopIndex + 1) - startIndex is the correct way)

I'm using the latest nuget package: 4.4.1-alpha001

I search ANTLR support for Mono

I want write parser in Mono. It is possible migrate from NET to Mono with this sources? Which source changes I must do?

@leftfactor creates tokens with dollar signs in the generated parser

I tried specifying @leftfactor{rulename} in a grammar and some invalid tokens are generated in the xxxParser.cs:

RULE_var$lf$rulename = 0, RULE_var$nolf$rulename = 1,

doing a simple text replace of var$lf$rule in var_lf_rule and var$nolf$rule in var_nolf_rule solves the issue until the next regeneration of the grammar [so, not a showstopper].

When parser rule collides with ParserRuleContext member compilation fails

I was playing around with the WebIDL grammer. After applying the patch prescribed by #13, I was still having problems:

1>------ Build started: Project: CoolProject, Configuration: Debug Any CPU ------
1>C:\dev\CoolTool\CoolProject\WebIDLParser.cs(228,27,228,36): warning CS0108: 'CoolProject.WebIDLParser.DefinitionContext.exception()' hides inherited member 'Antlr4.Runtime.ParserRuleContext.exception'. Use the new keyword if hiding was intended.
1>C:\dev\CoolTool\CoolProject\WebIDLParser.cs(318,4,318,23): error CS1656: Cannot assign to 'exception' because it is a 'method group'
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

Causing the first message (warning):

public ExceptionContext exception() {
    return GetRuleContext<ExceptionContext>(0);
}

To me, it seems this generated exception() method should be given a different name, since it collides with the exception member on Antlr4.Runtime.ParserRuleContext. I've noticed that another method generated from the WebIDL grammar is enum_(). This underscore suffix is a decent way of avoiding the name collision problem (as well as a collision with reserved C# symbols). It's not pretty, but at least it gets the code compiling right off the bat.

ANTLR 4 C# (4.1.0-alpha003) accepts some code differently from ANTLR 4.1 Java

When I wrote Lua grammar, I encountered strange behavior.
I created the minimum grammar where I can cause the behavior.

grammar Lua;
chunk : exp (',' exp)* EOF;
exp : NAME | exp callSuffix;
callSuffix : '(' ')';
NAME : [a-zA-Z_][a-zA-Z_0-9]*;
WS : [ \t\r\n\u000C]+ -> channel(HIDDEN);

Then, I wrote some tests as follows.

    class MyListener : IParseTreeListener {
        private readonly Parser _parser;
        public MyListener(Parser parser) { _parser = parser; }
        public void VisitTerminal(ITerminalNode node) { }
        public void VisitErrorNode(IErrorNode node) { }
        public void EnterEveryRule(ParserRuleContext ctx) {
            Console.WriteLine("Enter: " + _parser.RuleNames[ctx.GetRuleIndex()]);
        }
        public void ExitEveryRule(ParserRuleContext ctx) {
            Console.WriteLine("Exit: " + _parser.RuleNames[ctx.GetRuleIndex()]);
        }
    }

    [Test]
    public void ParseLua() {
        var stream = new AntlrInputStream("func()");
        var lexer = new LuaLexer(stream);
        var commonTokenStream = new CommonTokenStream(lexer);
        var parser = new LuaParser(commonTokenStream);
        parser.AddParseListener(new MyListener(parser));
        parser.ErrorHandler = new BailErrorStrategy();
        parser.chunk();
    }

The test shows the following debug message. exp wrongly occurs twice on the same depth without ,.

Enter: chunk
Enter: exp
Exit: exp
Enter: exp
Enter: callSuffix
Exit: callSuffix
Exit: exp
Exit: chunk

However, a parser with ANTLR 4.1 Java shows the following debug message.

Enter: chunk
Enter: exp
Enter: exp
Exit: exp
Enter: callSuffix
Exit: callSuffix
Exit: exp
Exit: chunk

Constants named differently than in Java target

The constants for the EOF token and the HIDDEN channel are named "Eof" and "Hidden" in C# as opposed to "EOF" and "HIDDEN" in Java.

I'm using antlr4-csharp-4.0.1-SNAPSHOT-complete.jar