lexer.input() does not reset the state stack about ply

[dabeaz]

Traditionally, specifying new input has never reset anything in the lexer. This is to support use cases where input might be assembled in chunks or from different sources, but made to appear as if it were a continuous source of tokens to the parser. Honestly, I'm not sure how well this is defined though. So, would need to look at it further.

[vpelletier]

This is to support use cases where input might be assembled in chunks or from different sources

How would this work at yacc.LRParser level ? As far as I can see, current API will loop until get_token returns a false value (ie, None), at which point it will insert a $end token.

Background: I tried to use ply to represent the grammar of a protocol, which my program analyses as the protocol is being used between two independent entities. So my lexer is very custom. As I want parser's output to be updated as soon as possible after link activity, I want to push tokens as soon as they are received - without checking if they make the grammar reach topmost rule (where no state is lost if $end is encountered) as this would defeat the purpose of using ply automated state machine generation. So far, the only solution I found was to extend yacc.LRParser so as to reuse its internals - so I would be very interested to know of a better way if it exists.

[bazsi]

Well, supporting the use case you are describing (supply input in multiple chunks) would be pretty useful, and is possible with traditional bison/flex.

However when I opened this ticket I came to the conclusion that this was not supported (I cant remember the exact details but for sure I read the source), thats why I asked.

Anyway, my code now reuses the parser instance and calls input, but I could instantiate a new parser instead.

So on my part, input() which allows multiple chunks would be the best option.

[dabeaz]

An incremental lexer would require passing a modified lexer and/or tokenfunc to the parser. For example:

def modified_get_token():
      # Return the next token
      ...
      return tok

yacc.parse(tokenfunc=modified_get_token)

Within the modified_get_token() function, you could drive the lexer, check for EOF, and feed input in a more customized manner.

Again, I don't know if this has ever been fully explored in PLY. Maybe needs some more documentation. I will look into this.

[dabeaz]

For now, I'm going to leave the parsing state alone on new input. The lexer() can have its state manually reset on new input if needed. There are more advanced uses where you would not want to reset the state on input.