peytoncasper / parser Goto Github PK

Full Regex decomposition will be supported, but as an initial step, the WHITESPACE rule requires the ability to transition to END when a character other than [\n\t\r ] is received while in the WHITESPACE recursive step.

As a result, the first Regex syntax parser will support is the ^.

The Lexer should effectively be a component of the overall Parser class. However, the Lexer should be responsible for receiving a String (Later on File Handler) and iterating through each character while passing each character to the active FSMs next() function.

Additionally, it should have a nextToken() or next() function that is responsible for returning the next complete token either recognized or unrecognized back to the calling Parser.

next() is currently a stub function. This needs to be implemented such that it accepts a given input Character and decides which state to progress to next. In addition, upon reaching a termination state, next() should return the unrecognized or recognized token to the Lexer.

Open Items

• Decide how to handle an unrecognized token. Error or Custom token?

We need to understand how existing Parsers handle an unrecognized Token that doesn't have a proper transition.

What gets added to the AST/Parse Tree?
How does the Parser handle this Token?

Ultimately this should be the top-level class containing a valid Lexer, FSM and String/File Handler. The initial implementation will make the Parser responsible for all of these items.

1. Accepting a FSM definition in JSON format
2. Creating the resulting FSM class
3. Passing the FSM through the NFA Validation step
4. Passing the NFA through the DFA Validation step
5. Creating the Lexer class
6. Translating the finalized list of tokens into a Parse Tree/Abstract Syntax Tree

The Lexer converts our File (String) into an array for efficient indexing. Until support for FileHandler has been added, we need correct length checking to prevent an error when iterating past the bounds of the "file"

Currently, when initializing states during the creation of a FiniteStateMachine, there is no error handling if a state does not exist. Ideally, this would be caught as a syntax error during editing, but it would be better to also flag this during compilation and flag this as an error with an indicator of which rule failed.

The Token map is currently added to States, but we still need to ensure the field is being parsed from JSON and added into each Token map during the Edge conversion to State Transitions.

If the Lexer never receives a Token value, it will throw a tokenizationError. This needs better logging to help with debugging and understand why a nil token was returned as opposed to EOF or UNRECOGNIZED