parser's Issues
Not (^) Operator as a Rule Abstraction
Full Regex decomposition will be supported, but as an initial step, the WHITESPACE rule requires the ability to transition to END when a character other than [\n\t\r ] is received while in the WHITESPACE recursive step.
As a result, the first Regex syntax parser will support is the ^.
Add Error Handling on FSM Creation to Indicate Missing State
Currently, when initializing states during the creation of a FiniteStateMachine, there is no error handling if a state does not exist. Ideally, this would be caught as a syntax error during editing, but it would be better to also flag this during compilation and flag this as an error with an indicator of which rule failed.
Add Logging to TokenizationError in Lexer Class
If the Lexer never receives a Token value, it will throw a tokenizationError
. This needs better logging to help with debugging and understand why a nil token was returned as opposed to EOF or UNRECOGNIZED
Implement Parser Class
Ultimately this should be the top-level class containing a valid Lexer, FSM and String/File Handler. The initial implementation will make the Parser responsible for all of these items.
- Accepting a FSM definition in JSON format
- Creating the resulting FSM class
- Passing the FSM through the NFA Validation step
- Passing the NFA through the DFA Validation step
- Creating the Lexer class
- Translating the finalized list of tokens into a Parse Tree/Abstract Syntax Tree
Implement Lexer Class
The Lexer should effectively be a component of the overall Parser class. However, the Lexer should be responsible for receiving a String (Later on File Handler) and iterating through each character while passing each character to the active FSMs next()
function.
Additionally, it should have a nextToken()
or next()
function that is responsible for returning the next complete token either recognized or unrecognized back to the calling Parser.
Implement Token Map on END State Transitions
The Token map is currently added to States, but we still need to ensure the field is being parsed from JSON and added into each Token map during the Edge conversion to State Transitions.
File Length Checking
The Lexer converts our File (String) into an array for efficient indexing. Until support for FileHandler has been added, we need correct length checking to prevent an error when iterating past the bounds of the "file"
Research UNRECOGNIZED Token
We need to understand how existing Parsers handle an unrecognized Token that doesn't have a proper transition.
What gets added to the AST/Parse Tree?
How does the Parser handle this Token?
Implement FSM next() Function
next()
is currently a stub function. This needs to be implemented such that it accepts a given input Character and decides which state to progress to next. In addition, upon reaching a termination state, next()
should return the unrecognized or recognized token to the Lexer.
Open Items
- Decide how to handle an unrecognized token. Error or Custom token?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.