massung / lexer Goto Github PK

I'm using the lexer package to create a lexer for my esolang, and I wanted to implement syntactic/substitution macros by parsing the macro definition into a new rule for a lex-state. Is there a way to accomplish what I want to do?

It looks like there is a naming conflict with lexer from the CL-LEXER package (check its .asd file here).

When asked to load "lexer" Quicklisp loads the one that is provided by the CL-LEXER, whereas this one is not loaded, and I am afraid that it will not since there is no RE package in the Quicklisp projects repository.

I suggest renaming the system name from LEXER to MASS-LEXER or any other that will not conflict with other systems in the Quicklisp repository.

But then RE package should be also added to the Quicklisp repository.

(deflexer string-lexer (:case-fold t :multi-line t)
("%s+" )
("//.$" :next-token (setq *comment $$))
("if" (VALUES :IF ))
(""(""|[^\"])+"" (values :string $$))
("%a%w*" (values :id $$ )))

(defun read-all-tokens (in-string)
(loop with l = (string-lexer in-string)
for x = (multiple-value-list (funcall (lex-next-token l)))
until (null (first x))
collect x))

in the REPL after
L-1C 4 > (read-all-tokens "if ifi")
((:IF #<LEXER::TOKEN "if" IF>) (:IF #<LEXER::TOKEN "if" IF>) (:ID #<LEXER::TOKEN "i" ID>))
but i expect
((:IF #<LEXER::TOKEN "if" IF>) (:ID #<LEXER::TOKEN "ifi" ID>))

where i have error?
And why error?

in documentation says - http://www.lispworks.com/documentation/HyperSpec/Body/13_adca.htm
what's why (char-equal (char-upcase #\я) #\Я) is NIL

Can you append unicode support to you lexer?

Hello
I have question about you deflexer. In my code i need call lexer from rule of deflexer (for example skip token of comment, read and return next token after comment) How i may write it in deflexer?

I am using lexer along with your other project parse.

I am having trouble figuring out how I can lookahead on my input without modifying the lexer state.

In this example I wish for check-bar to be able to lookahead to see if the next token is "bar" and if it is to just ignore it without consuming the token and fail otherwise.

(define-lexer my-lexer (st)
  ("%s+" (values :next-token $$))
  ("foo" (values :foo $$))
  ("bar" (values :bar $$)))

(define-parser check-bar
  "This parser will fail unless the next token is bar but will not consume the token."
  (.let (state0 (.get))
    (.do (.is :bar)
         (.put state0))))

(define-parser foo-then-bar
  (.let* ((foo (.is :foo))
          (_ 'check-bar)
          (bar (.is :bar)))
    (.ret (list foo bar))))

(defun run-parser (string)
  (with-lexer (lexer 'my-lexer string)
    (with-token-reader (next-token lexer)
      (parse 'foo-then-bar next-token))))

However the following repl output shows that check-bar consumes token from the lexer state:

PARSER> (run-parser "foo bar")
; Debugger entered on #<LEXER::LEX-ERROR {1004E95393}>
[1] PARSER> 
; Evaluation aborted on #<LEXER::LEX-ERROR {1004E95393}>
PARSER> (run-parser "foo bar bar")
("foo" "bar")
T

How can I lookahead at my input without consuming any tokens?