nushell / tree-sitter-nu

A tree-sitter grammar for nu-lang, the language of nushell

Tree-sitter grammar for Nushell

[WIP] nushell grammar for tree-sitter

goals (brainstorming)

• Ability to serve as a parser for a repl
• Ability to work in editors that support tree-sitter (hx, nvim, others)
• Ability to work in rust
• Fast
• Ability to be utilized for syntax highlighting
• ...

installation guide

• for neovim users, please have a look at installation/neovim.md
• for helix users, please have a look at installation/helix.md

contributing

see instructions

tree-sitter-nu's Issues

Add support for new nushell feature - raw strings

https://www.nushell.sh/blog/2024-05-28-nushell_0_94_0.html#new-language-feature-raw-strings-toc

Parsing of range fails

This bug is introduced by this change #50.
It seems that the rule, which parses a number omitted after decimal point (e.g. 1.) as float, parses head of range value as float incorrectly.

Parsing of numbers using underscores fails.

Parsing of list destruction match pattern fails

1. fails to parse with guard
2. parsed as range value incorrectly without comma

helix integration with nushell syntax highlighting

we've broken the helix syntax highlighting somehow. would be nice to fix it. i've played with it and can't figure it out.

I change the rev to the latest commit in the repo and it breaks.

then hx --grammar fetch and hx --grammar build

❯ hx --grammar fetch
Fetching 169 grammars
168 up to date git grammars
1 updated grammars
        nu now on 2d0dd587dbfc3363d2af4e4141833e718647a67e
❯ hx --grammar build
Building 169 grammars
168 grammars already built
1 grammars built now
        ["nu"]

blocks cannot be used as values

Description

since we cannot backtrack, it is difficult to tell apart blocks and records. so currently, blocks are not parsed as values

# this will not parse
ls | each { $in + 4 }

# however this parses
ls | each {|it| $it + 4 }

Exports fail to highlight after first occurence

Zed

Helix

export const GAMMA = 0.5772156649015329
export const E = 2.718281828459045
export const PI = 3.141592653589793
export const TAU = 6.283185307179586
export const PHI = 1.618033988749895

https://github.com/nushell/nushell/blob/main/crates/nu-std/std/math.nu

Fails to parse a multi-line command enclosed in parentheses

`duration` is parsed as a `string` when used as an argument.

tree-sitter-nu version: 4e5567e

Support the spread operator

Currently, the ... is parsed as a string, whereas it's a special operator that should be assigned something special, and then highlighted as an operator. If it's parsed as a expr_unary, highlights.scm can then be changed as follows:

diff --git a/queries/nu/highlights.scm b/queries/nu/highlights.scm
index c73f8f3..ce0b839 100644
--- a/queries/nu/highlights.scm
+++ b/queries/nu/highlights.scm
@@ -161,7 +161,7 @@
     "++="
 ] @operator)
 
-(expr_unary ["not" "-"] @operator)
+(expr_unary ["not" "-" "..."] @operator)
 
 (val_range [
     ".."

Grammar railroad diagram

Using this page https://meimporta.eu/tree-sitter/json2ebnf.html we can get a navigable railroad diagram for tree-sitter grammars like this project.

See also tree-sitter/tree-sitter#2268 .

Cheers !

inconsistent behavior with unquoted strings

Description

# this should parse as a pipeline with two elements
'nushell' | str contains n

# but it is parsed as two pipelines, with one ending at `contains`
'nushell' | str contains
n

# quoting the string solves it though
'nushell' | str contains 'n' # <- okay

A newline between `}` and `else` cause parse error

A newline between } and else is allowed only if it is parenthesized.
This is originally reported in this comment.

script of the sort "^'bin' command" is not parsed

Just as the title says, having ^'bin' command in the nu file doesn't get parsed.

`inf` is parsed as a number unless quoted

tree-sitter-nu version: 2d0dd58

Wrong highlighting in if else statement.

Treesitter highlighting fails when else flag is missing a closure else $statement.

Look at the let keywords below.
Example without closure:

Example with closure:

Tested in neovim.

Parsing of re-export module fails

https://github.com/nushell/nushell/blob/b0b4c3dffd9f84ea18236e8f9055d642de029a1d/crates/nu-std/std/mod.nu#L1-L18

command identifier with `?` is not parsed correctly

use *file* identifier: produces a tree-sitter error

Using the pattern use file identifier produces a TS error. This happens when the identifier is not put in a list ex: use file [identifier - this produces no error. Also happens when * is used instead of an identifier.

Picture (look at the highlighting of the use - on the line before the incorrectly highlighted use there is an TS error, which can be seen via the TSPlayground):

The line after an empty comment is parsed as a comment.

#
echo hello

Some record keys are not parsed correctly

The following keys are not parsed correctly

• unquoted string contains a hyphen
• number
• variable

Parsing of pipeline fails when the first pipeline element is variable or control expression, and there is a newline before the next element

Example 1.

Example 2.

In Example 1, match expression must be parsed as a pipe_element but it is not.
And entire script must be parsed as a pipeline but it is not.

In Example 2, $xs and | each {|x| $x } must be parsed as pipe_element of same pipeline but they are not.

Incorrect parsing for long param flag with dashes

First example:

#!/usr/bin/env nu

def main [
  --long-param-with-dashes
  --long_param_with_underscores
] {
  if $long_param_with_dashes {
    print "dash param"
  } else {
    print "no dash param"
  }

  if $long_param_with_underscores {
    print "underscore param"
  } else {
    print "no underscore param"
  }
}

Syntax highlighting is incorrect and there are multiple "param_short_flag" for the first param flag (with dashes):

Second example (probably same cause but different effect):

#!/usr/bin/env nu

def main [
  --dry-run (-n)
] {
  if $dry_run {
    print "Dry run"
  } else {
    print "Real execution"
  }
}

There is now an error in the tree:

highlight fails with `let root = $env.FILE_PWD` without parens around the RHS

Example:

fails to highlight

let DIR = $env.FILE_PWD
let cache = "nixpkgs-wayalnd"

screenshot: https://user-images.githubusercontent.com/327028/271797080-9fdee9c2-ff53-4254-833c-e7ebf404044e.png

highlights okay

let DIR = ($env.FILE_PWD)
let cache = "nixpkgs-wayalnd"

screenshot: https://user-images.githubusercontent.com/327028/271797077-59cdae89-9400-40f5-bc3b-0f7f9d701c80.png

I tested this in Helix - I bumped the tree-sitter-nu to latest here and then rebuilt my system and tested again. results match the screenshots I shared.

commit 4209c5ce6c294eaf4ff6f415152c34380746dbdf (HEAD -> master)
Author: Cole Mickens <[email protected]>
Date:   Tue Oct 3 22:21:57 2023 +0200

    languages.toml: bump tree-sitter-nu

diff --git a/languages.toml b/languages.toml
index af1466f1..5cbeb0b7 100644
--- a/languages.toml
+++ b/languages.toml
@@ -1811,7 +1811,7 @@ indent = { tab-width = 2, unit = "  " }

 [[grammar]]
 name = "nu"
-source = { git = "https://github.com/nushell/tree-sitter-nu", rev = "98c11c491e3405c75affa1cf004097692da3dda2" }
+source = { git = "https://github.com/nushell/tree-sitter-nu", rev = "307559b6a3885ef0c55bcd5e23e9e164a7ce03bd" }

 [[language]]
 name = "vala"

Interpolated strings produce a lot of nodes

At Zed we have a tool built into our editor for examining the syntax tree for any buffer we have open. While using this to add nu syntax highlighting to Zed, I noticed that there is a new tree sitter node being added for every character in an interpolated string. I expect this isn't the intended behavior so I thought I'd let y'all know 🙂

Parsing of `def` with flags fails

Can we simplify this part of the grammar?

Apparently, this specific section of the grammar is difficult to parse in zed's CI. See this issue.

I'm wondering if it could be simplified somehow?

Parser breaks with this syntax

With the type@'type' on the command parameter, the syntax highlighting breaks.

Example of the code:

export def-env main [
	command?: string@'nu-complete rbenv',
	...args
] {
    let new_env = if $command in ["rehash", "shell"] {
        # implement each on indiviudaly e.g.
        if $command == "shell" {
            { RBENV_VERSION_OLD: $env.RBENV_VERSION RBENV_VERSION: $args.0 }
        } else {
            error make { msg: $"`($command)` command is not supported yet" }
        }
    } else {
    	if ($command | is-empty) {
    		^rbenv
    	} else {
        	^rbenv $command $args
    	}
        {}
    }
    load-env $new_env
}

def 'nu-complete rbenv' [] {
    ^rbenv help
    | lines
    | where ($it | str starts-with "   ")
    | each {|entry| $entry | split row ' ' | get 0 }
}

Screenshoot:

Parsing fails when the right-hand side of a binary expression is an unquoted string

Highlighting breaks on `alias` before `let`

The following breaks:

alias less = bat
let foo = $'($env.HOME)'

It's fine when alias is removed.

Fails to parse unquoted string containing symbols in a list

Closure parsed as a record

In the following example:

#!/usr/bin/env nu

let $list = [a b c d]

# Correctly parsed
$list | filter {|x| $x | str starts-with c }

# Still valid but not correctly parsed
$list | filter { str starts-with c }

The closure for filter command is incorrectly parsed in the last line. And this leads to an error in the tree.

syntax highlighting gets confused

Trying out Zed with nu highlighting I found this in default_env.nu. Notice that after the pipe, it stops highlighting.

Looking at the Syntax Tree windows in Zed, there are a few ERRORs.

It would be nice to get this fixed in order to have a more pleasant experience in Zed.

Syntax Highlighting

Do you have any plans for syntax highlighting in this repo? It appears common that treesitter repos (I sampled cpp, js and python) do have highlights.scm files. My general experience so far was that syntax highlighting starts working once I install the parser with TSInstall.

I started looking into it and if you don't mind, I'd like to give it a try :)

Help with finding ERRORs in tree-sitter-nu.

Related Problem

Because tree-sitter-nu is still WIP there are various errors still in it. The code could still be highlighted even when errors occur (except for the snippest of code affected by the error), so they might go unnoticed.

To deal with this it can be helpful for people to test their nushell scripts via tree-sitter-cli. This can be done manually (running tree-sitter parse on every file and checking for errors there) or through I've script (tree-sitter-get-errors.nu) that I've written that I would post below.

Steps for helping out:

1. Add the script below to your $env.NU_LIB_DIRS directory - by default it is a folder called scripts in your configuration directory.
2. Import the script via use tree-sitter-get-errors.nu
3. You can then call tree-sitter-get-errors to find TS errors in your nushell files. Example usage:
   • tree-sitter-get-errors file1.nu file2.nu Can take any number of files by name
   • tree-sitter-get-errors --glob *nu Can accept glob
   • command_that_returns_paths | tree-sitter-get-errors Can take stdin
   • Or any combination from the listed examples.
4. You can pipe the output into columns to get a list of all files names with an error in them. (Optional)
5. Make an issue providing either the full text, or preferably a minimal snippet of the code that still provides an error. You can use the table output from the tree-sitter-get-errors script, to view a list of all error line ranges (LINES START AT 0) in each file to find the code snippet that creates the TS error.

Related script

• tree-sitter-get-errors.nu:

# Test files for tree-sitter errors.
#
# Prints a table with the filename where an error has occured.
# The list is the line numbers of each error in the format of:
#       "$STARTING_ERROR_LINE $FINAL_ERROR_LINE"
#
# **Line numbers are offset by -1**: Treesitter parse lines starting from line 0.
#
# The stdin can also be used to accept paths (they are appended to the files).
export def main [
    ...files: path      # Files to parse for error.
    --glob (-g): string # Provide a glob to add to the files.
    --threads (-t): int # Number of threads to use. Defaults at all.
    --no-reduce (-R)    # Instead of printing in a table, provide the result in table of lists.
]: [list<path> -> table, list<path> -> list] {
    # Parameter assingment with globs & stdin.
    let stdin = ($in | default [])
    let files = ($files ++ $stdin) | path expand |
        append (if $glob != null {glob ($glob | default "")}) | uniq

    # Setting threads.
    let threads = $threads | default 0

    # Output
    if $files == [] {return ("Run command with --help to see usage.")}
    $files | par-each --threads $threads  {|it| _tree-sitter-get-errors-single $it} |
        where {|it| ($it | describe) != list<any>} |
        if $no_reduce {$in} else {$in | reduce {|it, acc| $acc | merge $it}}
}

# Parse file for TS errors.
#
# Returns table with one column being the file name.
# The list elements are the line number ranges of the error.
export def _tree-sitter-get-errors-single [file: path] {
    tree-sitter parse $file | split row "\n" |
        where {|it| $it =~ '^\s++\(ERROR'} |
        str replace -r '^\s++\(ERROR \[(\d++)[^\[]++\[(\d++).*+$' '$1 $2' |
        # Return a table with file name - relative if possible.
        wrap (try {$file | path relative-to (pwd)} catch {$file})
}

add queries for different tree-sitter-based editors

i think queries should be written for tree-sitter-based editors to enable syntax highlighting

resources:

• official tree-sitter query docs
• neovim - nvim-treesitter docs
• helix - helix query docs
• lapce - lapce docs