tree-sitter-grammars / tree-sitter-commonlisp

Common Lisp grammar for tree-sitter

tree-sitter-commonlisp

WIP. Goal is to have a better syntax highlighting for Neovim and some semantic refactoring https://github.com/nvim-treesitter/nvim-treesitter-refactor/.

All praise goes to https://github.com/sogaiu/tree-sitter-clojure which is extended by this grammar.

TODOs:

• support number literatls that are different from clojure (e.g. .9)

Macros with special respresentation in syntax tree (when written with lowercase letters):

• defun and friends (e.g. defmethod)
• loop macro

This grammar is used in https://github.com/Wilfred/difftastic to generate syntax-ware diffs for Common Lisp.

tree-sitter-commonlisp's Issues

Consider a more permissive license?

I'd love to include tree-sitter-commonlisp in difftastic, but this grammar is GPL unlike the other tree-sitter grammars available.

Would you be willing to relicense to something like MIT?

I understand this is a very personal thing, so no worries if you'd rather not.

Write access to the repo for me to work on tagging?

@theHamsta , I am discussing with GitHub developers to use tree-sitter-commonlisp tagging for GitHub code navigation (https://github.com/orgs/community/discussions/55704).

I pointed them to my fork of tree-sitter-commonlisp, because I want to improve the tagging queries little by little. And if I will need to wait for you to merge every change, and wait for GitHub developers who also can not respond immediately, it will be very difficult for me to coordinate this, fix problems, etc.

Maybe you can give me write access to your repo? I will then point GitHub developers to your repo.

I will only use write access for tagging work. If I ever want to touch anything other than tagging, which is unlikely, I will open a pull request for you to review and not commit it by myself.

defun including a "@" appear to break highlighting

Hi,

First off, thanks for making this available, it's very much appreciated.

As the title says, I think I found a minor bug where including a "@" in my function name breaks highlighting:

;; this highlights properly
(defun foo (bar)                                                                                                                                                                                         
  (baz))                                                                                                                                                                                                     

;; this doesn't and breaks all highlighting below                                                                                                                                                                                                        
(defun @foo (bar)                                                                                                                                                                                        
  (baz))

I'm on NVIM v0.5.1 with latest available treesitter:

    -- Post-install/update hook with neovim command                                                                                                                                                          
    use {                                                                                                                                                                                                    
        'nvim-treesitter/nvim-treesitter',                                                                                                                                                                   
        run = ':TSUpdate'                                                                                                                                                                                    
    }
[...]
require'nvim-treesitter.configs'.setup {                                                                                                                                                                     
    ensure_installed = 'maintained',                                                                                                                                                                         
    highlight = { enable = true },                                                                                                                                                                           
    indent = { enable = true  },                                                                                                                                                                             
    rainbow = {                                                                                                                                                                                              
        enable = true,                                                                                                                                                                                       
        extended_mode = false,                                                                                                                                                                               
        max_file_lines = nil,                                                                                                                                                                                
    }                                                                                                                                                                                                        
}       

Please let me know if this is not the right place to open an issue or if more info is needed.

Support parsing #\Replacement_Character

I've been testing this grammar against the .lisp files in Sly, and I noticed that the grammar treats the following character as two literals:

#\Replacement_Character

This produces:

(source (char_lit) (sym_lit))

Tagging support

The tree-sitter tags command can be used to implement "go to definition" functions. (In particular at github, I assume.)

It is documented here: https://tree-sitter.github.io/tree-sitter/code-navigation-systems

It does not work for tree-sitter-commonlisp. Right now (after fixing #12) if I run tree-sitter tags ~/prj/cl+ssl/cl-plus-ssl/src/package.lisp I get just

No tags config found for path "/home/anton/prj/cl+ssl/cl-plus-ssl/src/streams.lisp"

To support tagging, the languiage impl should provide a queries/tags.scm file.
An example: The tree-sitter-ruby

After adding #14, the same tags command as above prints function definition locations. The function names are not package qualified, but I think it may be even better that way for github code navigation currently, as symbol references produced by tree-sitter parse command are also not package qualified (unless in the source code they have a package prefix)

initialize-instance      | function     def (50, 11) - (50, 30) `(defmethod initialize-instance :after ((stream ssl-stream)`
print-object     | function     def (61, 11) - (61, 23) `(defmethod print-object ((object ssl-stream) stream)`
stream-element-type      | function     def (73, 11) - (73, 30) `(defmethod stream-element-type ((stream ssl-stream))`
close            | function     def (76, 11) - (76, 16) `(defmethod close ((stream ssl-stream) &key abort)`
open-stream-p    | function     def (95, 11) - (95, 24) `(defmethod open-stream-p ((stream ssl-stream))`
stream-listen    | function     def (98, 11) - (98, 24) `(defmethod stream-listen ((stream ssl-stream))`
stream-read-byte         | function     def (109, 11) - (109, 27) `(defmethod stream-read-byte ((stream ssl-stream))`
stream-read-sequence     | function     def (123, 11) - (123, 31) `(defmethod stream-read-sequence ((stream ssl-stream) seq start end &key)`
stream-write-byte        | function     def (146, 11) - (146, 28) `(defmethod stream-write-byte ((stream ssl-stream) b)`
while            | function     def (154, 10) - (154, 15) `(defmacro while (cond &body body)`
stream-write-sequence    | function     def (157, 11) - (157, 32) `(defmethod stream-write-sequence ((stream ssl-stream) seq start end &key)`
stream-finish-output     | function     def (175, 11) - (175, 31) `(defmethod stream-finish-output ((stream ssl-stream))`
stream-force-output      | function     def (178, 11) - (178, 30) `(defmethod stream-force-output ((stream ssl-stream))`
install-nonblock-flag    | function     def (190, 7) - (190, 28) `(defun install-nonblock-flag (fd)`
install-nonblock-flag    | function     def (201, 7) - (201, 28) `(defun install-nonblock-flag (fd)`
install-nonblock-flag    | function     def (208, 7) - (208, 28) `(defun install-nonblock-flag (fd)`
install-nonblock-flag    | function     def (212, 7) - (212, 28) `(defun install-nonblock-flag (fd)`
install-handle-and-bio   | function     def (227, 7) - (227, 29) `(defun install-handle-and-bio (stream handle socket unwrap-stream-p)`
install-key-and-cert     | function     def (263, 7) - (263, 27) `(defun install-key-and-cert (handle key certificate)`
x509-certificate-names   | function     def (276, 7) - (276, 29) `(defun x509-certificate-names (x509-certificate)`
ssl-stream-handle        | function     def (289, 11) - (289, 28) `(defmethod ssl-stream-handle ((stream flexi-streams:flexi-stream))`
ssl-stream-x509-certificate      | function     def (292, 7) - (292, 34) `(defun ssl-stream-x509-certificate (ssl-stream)`
ssl-load-global-verify-locations         | function     def (295, 7) - (295, 39) `(defun ssl-load-global-verify-locations (&rest pathnames)`
ssl-set-global-default-verify-paths      | function     def (309, 7) - (309, 42) `(defun ssl-set-global-default-verify-paths ()`
ssl-check-verify-p       | function     def (316, 7) - (316, 25) `(defun ssl-check-verify-p ()`
ssl-verify-init  | function     def (330, 7) - (330, 22) `(defun ssl-verify-init (&key`
maybe-verify-client-stream       | function     def (348, 7) - (348, 33) `(defun maybe-verify-client-stream (ssl-stream verify-mode hostname)`
handle-external-format   | function     def (373, 7) - (373, 29) `(defun handle-external-format (stream ef)`
with-new-ssl     | function     def (378, 10) - (378, 22) `(defmacro with-new-ssl ((var) &body body)`
make-alpn-proto-string   | function     def (403, 7) - (403, 29) `(defun make-alpn-proto-string (protocols)`
make-ssl-client-stream   | function     def (412, 7) - (412, 29) `(defun make-ssl-client-stream (socket`
make-ssl-server-stream   | function     def (554, 7) - (554, 29) `(defun make-ssl-server-stream (socket`
get-selected-alpn-protocol       | function     def (599, 7) - (599, 33) `(defun get-selected-alpn-protocol (ssl-stream)`
stream-fd        | function     def (609, 12) - (609, 21) `(defgeneric stream-fd (stream)`
stream-fd        | function     def (614, 11) - (614, 20) `(defmethod stream-fd (stream) stream)`
stream-fd        | function     def (617, 11) - (617, 20) `(defmethod stream-fd ((stream sb-sys:fd-stream))`
stream-fd        | function     def (621, 11) - (621, 20) `(defmethod stream-fd ((stream system:fd-stream))`
stream-fd        | function     def (625, 11) - (625, 20) `(defmethod stream-fd ((stream ccl::basic-stream))`
stream-fd        | function     def (629, 11) - (629, 20) `(defmethod stream-fd ((stream stream))`
stream-fd        | function     def (634, 11) - (634, 20) `(defmethod stream-fd ((stream two-way-stream))`
stream-fd        | function     def (638, 11) - (638, 20) `(defmethod stream-fd ((stream stream))`
stream-fd        | function     def (642, 11) - (642, 20) `(defmethod stream-fd ((stream comm::socket-stream))`
stream-fd        | function     def (657, 13) - (657, 22) `(defmethod stream-fd ((stream system::socket-stream))`

Upcoming change to tree-sitter-clojure

I'm opening this issue as a heads up that we're planning to merge a change to the tree-sitter-clojure master branch.

IIUC, tree-sitter-commonlisp depends on a fork of tree-sitter-clojure at this commit, so if that's right, perhaps our upcoming intended change won't have any affect. I didn't analyze in detail how tree-sitter-clojure is used by tree-sitter-commonlisp, but for reference our intended change is here: sogaiu/tree-sitter-clojure#31

A brief overview of the changes follows.

---

kwd_lit and sym_lit will now contain up to 3 new nodes:

(kwd_lit
  ns: (kwd_ns)
  delimiter: "/"
  name: (kwd_name))

(sym_lit
  ns: (sym_ns)
  delimiter: "/"
  name: (sym_name))

Only the (kwd_name) and (sym_name) are required.

sym_lit nodes can still contain metadata nodes (this is unchanged).

---

IIUC, tree-sitter-commonlisp redefines kwd_lit and sym_lit anyway so may be it wouldn't have had any effect on that account as well. We did, however, change some underlying bits that those two depended on though and didn't ascertain whether tree-sitter-commonlisp used any of them.

If we're mistaken that there would be no effect on tree-sitter-commonlisp from our upcoming changes, please let us know.

Thanks!

P.S. Our current plan is to merge around 2023-01-05 which is in about 2 weeks time.

---

On a more general note...

Though at this time we don't anticipate many changes going forward, we're planning to post certain change-related announcements to a pinned issue ahead of merging such changes.

If you'd like to keep track of what changes might be coming up, please have a look at the issue and consider subscribing :)

Parse error from `loop` with `maximizing`

For example:

(loop for s in symbols
      maximizing (length (symbol-name s)))

Produces:

{Node source (0, 0) - (4, 0)}
  {Node ERROR (1, 0) - (4, 0)}
    {Node ( (1, 0) - (1, 1)} "("
    {Node loop (1, 1) - (1, 5)} "loop"
    {Node for (1, 6) - (1, 9)} "for"
    {Node sym_lit (1, 10) - (1, 11)} "s"
    {Node for_clause_word (1, 12) - (1, 14)}
      {Node in (1, 12) - (1, 14)} "in"
    {Node sym_lit (1, 15) - (1, 22)} "symbols"

defconstant is not highlighted

With commonlisp added to be languages, I get "Parser not available for language commonlisp"

My config:

require'nvim-treesitter.configs'.setup {
  ensure_installed = {
    "javascript", "typescript", "glimmer", "tsx",
    "html", "jsdoc", "regex", "bash",
    "toml", "html", "json",
    "css", "lua", "commonlisp"
  },
  highlight = {
    enable = true,
  }
}

Support parsing defpackage

The grammar currently considers this to be an error:

(defpackage foo-bar
  (:use cl))

It's not clear to me which grammar rules are preventing this.

bug: Tree-sitter fails when encountering square brackets

Did you check existing issues?

• I have read all the tree-sitter docs if it relates to using the parser
• I have searched the existing issues

Tree-Sitter CLI Version, if relevant (output of tree-sitter --version)

tree-sitter 0.21.0 (1c55abb5308fe3891da545662e5df7ba28ade275)

Describe the bug

Just started learning commonlisp, so there may be some subtleties here that I'm simply overlooking. However: Tree-sitter fails when encountering the square brackets in this piece of code:

(ql:quickload :cl-collider)
(in-package :sc-user)
(named-readtables:in-readtable :sc)

(setf *s* (make-external-server "localhost" :port 48800))
(server-boot *s*)

(defvar *synth*)
(setf *synth* (play(sin-osc.ar [320 321] 0 0.2))) 
(free *synth*)

(server-quit *s*)
(server-query-all-nodes)

Steps To Reproduce/Bad Parse Tree

(source [0, 0] - [14, 0]
  (list_lit [0, 0] - [0, 27]
    value: (package_lit [0, 1] - [0, 13]
      package: (sym_lit [0, 1] - [0, 3])
      symbol: (sym_lit [0, 4] - [0, 13]))
    value: (kwd_lit [0, 14] - [0, 26]
      (kwd_symbol [0, 15] - [0, 26])))
  (list_lit [1, 0] - [1, 21]
    value: (sym_lit [1, 1] - [1, 11])
    value: (kwd_lit [1, 12] - [1, 20]
      (kwd_symbol [1, 13] - [1, 20])))
  (list_lit [2, 0] - [2, 35]
    value: (package_lit [2, 1] - [2, 30]
      package: (sym_lit [2, 1] - [2, 17])
      symbol: (sym_lit [2, 18] - [2, 30]))
    value: (kwd_lit [2, 31] - [2, 34]
      (kwd_symbol [2, 32] - [2, 34])))
  (list_lit [4, 0] - [4, 57]
    value: (sym_lit [4, 1] - [4, 5])
    value: (sym_lit [4, 6] - [4, 9])
    value: (list_lit [4, 10] - [4, 56]
      value: (sym_lit [4, 11] - [4, 31])
      value: (str_lit [4, 32] - [4, 43])
      value: (kwd_lit [4, 44] - [4, 49]
        (kwd_symbol [4, 45] - [4, 49]))
      value: (num_lit [4, 50] - [4, 55])))
  (list_lit [5, 0] - [5, 17]
    value: (sym_lit [5, 1] - [5, 12])
    value: (sym_lit [5, 13] - [5, 16]))
  (list_lit [7, 0] - [7, 16]
    value: (sym_lit [7, 1] - [7, 7])
    value: (sym_lit [7, 8] - [7, 15]))
  (ERROR [8, 0] - [14, 0]
    value: (sym_lit [8, 1] - [8, 5])
    value: (sym_lit [8, 6] - [8, 13])
    value: (sym_lit [8, 15] - [8, 19])
    value: (sym_lit [8, 20] - [8, 30])))
test.lisp	   0.10 ms	  2822 bytes/ms	(ERROR [8, 0] - [14, 0])

Expected Behavior/Parse Tree

I expect treesitter to be able to parse this normally, as the code compiles fine on sbcl/arch linux.

Repro

No response