grncdr / js-shell-parse

parse bash, with javascript (UNMAINTAINED)

shell-parse

Parse bash scripts into AST's

Synopsis

var parse = require('shell-parse')
parse('echo $PATH') //=> an ugly AST object

(better synopsis will come after real tests)

Description

This thing parses strings containing bash scripts into an AST that you might execute using an interpreter or something. The AST structure is still in flux, so you probably don't want to build on this just yet!

If you want to help, there's a whole bunch of failing test fixtures (borrowed from the lovely shellcheck project). You can start with reading the grammar. The grammar defines how text will be matched, while the corresponding rule callbacks in overrides.js defines the way those matches will be processed into AST nodes. Be sure to read the comments as there's a tiny bit of magic going on.

If you get stuck trying to fix a test case, email me. GitHub issue notifications almost never get my attention.

Examples

See the tests for all the things that can be parsed.

License

MIT

js-shell-parse's Issues

Any plan to finish this?

Hey, this is an awesome project!

I'm considering the use of this in Vorpal / Cash, however this would involve fine-tuning and perfecting its use.

I noticed you haven't committed anything in over a year. Do you have any plans on picking this project back up, or should I consider it dropped?

Thanks.

variable asignations

Add support to detect asignations, currently ```a='b' is identified as a one command

Don't have a good solution for parsing nested backticks

You can nest backtick command substitution by escaping the inner backticks:

echo `echo \`echo ok\``

The amount of backslashes increases exponentially as you nest further, because you need to double up the backslashes at each level to compensate for the fact that they will be interpreted multiple times:

echo `echo \`echo \\\`echo \\\\\\\`echo ok\\\\\\\`\\\`\``

Currently I can't think of a way to express this in a PEG grammar without a bunch of duplicate rules for each level of nesting. Honestly, I'd like to punt on it and just say "sorry we won't parse that" but I'm not sure how long that would be an acceptable solution.

Create simple gh-pages repl

You can pull in : https://github.com/bmeck/js-shell-parse/tree/gh-pages

It is useful for showing to people / getting a quick answer about AST structures like google.github.io/traceur-compiler/demo/repl.html does

way to parse back the AST into code?

Acornjs (https://github.com/ternjs/acorn#main-parser has the ability to parse and generate given an AST.

Would it be possible given a shell ast to parse it back to the original output? or even better, add the original string to the output object?

Intentional inconsistency?

I wasn't sure if this was intentional; however, I did notice some inconsistency when I have a variable assignment that includes an in-line comment:

actual

parse('DOMAIN="example.com" # with a comment')[0].type
// => 'command'

expected

parse('DOMAIN="example.com" # with a comment')[0].type
// => 'variableAssignment'

On the other hand, I get exactly what I'm expecting if I remove the comment:

parse('DOMAIN="example.com")[0].type
// => 'variableAssignment'

Not a huge deal as I can sort of work around this; however, I thought I'd mention it in case I was missing something.

Add support for `case` statements

case statements are not handled at all. I'm building a POSIX shell in Javascript for NodeOS and would need to available all the shell syntaxis... Could you help us? :-)

Publish on NPM

Package is not currently available on NPM, so it should... :-)

Detect valid commands

Allow to detect if a command line is valid or malformed

`>|` redirection fails

The >| redirection is not correctly parsed, js-shell-parse believes that's a pipe and that the > is part of the first command and raising a SyntaxisError, since it thinks that's a redirection of stdout to some not defined place.

Shell syntax and terminology (substitution, subshells, etc.)

Couple of things:

• <() and >() is process substitution, not command substitution.
• $() and backticks are both command substitution; the first being the preferred syntax because nesting backticks is heinous. But they are 100% equivalent.

I bring this up because:
[Edit: note the discussion of the use of the term subshell in the comments that follow.]

• Command substitution does not necessarily invoke a subshell, and in most cases won't. The only thing it guarantees is out-of-order evaluation: it simply evaluates the expressions inside the substitution first, then evaluates the surrounding statement using the results of the substitution.
• There aren't any builtins that explicitly mean 'subshell'; to do exactly that (without a compound action or backgrounding the task), its just sh -c ....
• Process substitution always invokes a subshell, since it is equivalent to invoking the expression inside the substitution and redirecting its output to an anonymous file descriptor in a background task, then invoking the outer expression with its input being the contents of that file descriptor. The <() syntax is a pretty notorious bashism, although its also present in ksh93 and zsh.
  • zsh (...and very recent bash?) also has the related =(), meaning file substitution, taking the same form as process substitution but using a regular file rather that a file descriptor or a FIFO, and can be used when data produced with <() exceeds the pipe buffer. It is also exactly equivalent to using a tempfile.

I think it might be worth asking (given the name and the tagline you've given the project), it is your intention to parse out shell or bash? I'm not super hardline anti-bash, but there are some very challenging extensions in bash (things like [[ $* =~ (.*) ]] && BASH_REMATCH[1] which you might be able to avoid, at least for the time being, if you were to limit yourself to POSIX shell rather than bash, per se.

Use this on the web?

Hey @grncdr, this is a very cool project! I'd like to run this in the browser, inside a Chrome extension. Is there a version that's compiled for the browser, e.g., via Browserify or similar?

Escaped newlines not parsed properly

mentioned in #14, will be reviewing shellcheck tests to highlight important tests cases

Please release 0.0.3. >> redirection is not handled in npm version 0.0.2.

example:

> parse('echo "problem" >> file')
SyntaxError: Expected " ", "\n", "\t", a list of statements or end of input but ">" found.

Unify variable and variableSubstitution

According to the spec both variable and variableSubstitution (with brackets) are the same thing, and in fact the first one is just sugar syntaxis for the brackets-one, so they should resolve to the same token.

Another alternative would be that variableSubstitution expresion would be calculated and in fact it makes sense, but this deviates from the spec and from how bash works, so I'm not fully sure if this "extension" should be done... :-/