Comments (8)
After peeking at this and realizing how many stars align to create this problem I'm not sure if there'll be an immediate/obvious fix (but I think it'll be feasible to raise a better error). For now I'll just break this down in public:
-
In the parsing routine, I've been tentatively trying out a step that knocks ~varlikes out of most invocations. I don't remember the exact circumstances that encouraged me to add it and my comment wasn't useful enough, but (projecting/guessing) maybe I couldn't get
argparse
to handle them, and they're reasonably common in invocations ofenv
andsudo
.Lines 3938 to 3939 in b743d2e
-
The assumption that the varlikes will be stripped out leaks into the parser for awk. I assumed the space-separated form, so I set that argument not to actually expect an argument. (It would be stripped in the space-separated case.)
Lines 1908 to 1910 in b743d2e
-
I think argparse is parsing this as roughly equivalent to
-v -c ategory="$1"
(and failing because I've set this option not to expect an argument). resholve surfaces these as warnings instead of errors here:Lines 1367 to 1368 in b743d2e
It's doing this for two reasons, one of which is thornier than the other:
-
In a few cases, resholve is coping with mutually-exclusive syntaxes from different command variants by just trying N parsers in turn, surfacing parse errors as mere warnings, and issuing the "I don't have any command-specific rules" error if none of them hit without error (i.e., there is no command parser for this command that can cope with the syntax in this invocation). This occurs at
Lines 3941 to 3978 in b743d2e
-
I didn't have a great sense of how good the parsers would be and what all kinds of things would cause parse errors here, so I didn't want to break a bunch of existing uses by making parse errors blocking.
(I think I'm on board with raising a distinct error if there are parsers and they all fail--you're right that the error is misleading.)
-
from resholve.
I have some uncommited WIP to add a new error, at least. Here's what the new error looks like when I use your invocation as a test case:
not ok 65 exercise built-in syntax parsers in 343ms
# (from function `parsers' in file tests/helpers.bash, line 174,
# in test file tests/parsers.bats, line 10)
# `parsers' failed with status 15
# WARNING:__main__:CommandParser CommandParser(prog='awk (generic)', usage=None, description=None, version=None, formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=False) passing instead of "argument -c/--traditional: ignored explicit argument 'ategory=$1'"
# awk -F'\t' -vcategory="$1" '{ split($9, cs, "|");for (i in cs) if (cs[i] == category) { print; break; }; }'
# ^~~
# [ stdinNone ]:1: 'awk' is able to execute its arguments, and none of my command-specific rules for figuring out if this specific invocation does or not succeeded. Look above for warning messages with more context on what went wrong.
WDYT?
from resholve.
Oof, removing var=val
arguments before the parser sounds like the kind of thing that was bound to come back to bite you eventually. Long term, you may not be able to manage without writing a custom argument parser. The ones that already exist generally assume you're defining the interface being parsed and can just... not do things it doesn't support.
As for the proposed error message: not bad, but maybe do s/is able to execute its arguments, and/executes its arguments in some circumstances, and/
. It sounds slightly clearer, at least to me.
from resholve.
Oof, removing
var=val
arguments before the parser sounds like the kind of thing that was bound to come back to bite you eventually.
Hehe :)
I suspected it might.
Long term, you may not be able to manage without writing a custom argument parser. The ones that already exist generally assume you're defining the interface being parsed and can just... not do things it doesn't support.
Yes, I've suspected as much for a bit.
There was actually an early period (at least, for this functionality) where I was trying to just build up a stable of parsing techniques, but it was a bit of a mess and I suspected there'd be literally 0 chance of outsider contributions on these while doing it that way.
I hope to end up, whether it's on a common parser or home grown, at something that is config driven, so I've got some incentive to stretch argparse as far as it'll go to understand the requirements better before tilting at the windmill.
I might have mentioned this in the issue about making custom parsers easier to bolt on, but I also a vague sense that it'd be nice if the format could graduate into something flexible enough to be re-usable in other tooling cases? (A probably-quixotic dream about actual upstreams or at least other parties helping with some of the lifting of creating and managing them...)
As for the proposed error message: not bad, but maybe do
s/is able to execute its arguments, and/executes its arguments in some circumstances, and/
. It sounds slightly clearer, at least to me.
👍
from resholve.
I hope to end up, whether it's on a common parser or home grown, at something that is config driven, so I've got some incentive to stretch argparse as far as it'll go to understand the requirements better before tilting at the windmill.
Whatever you do, I think you at least need a code-based escape hatch. There's no way a config-driven system can really handle everything. The world is too weird. Given that, you might as well make the system modular at the code level, then make some generic handlers that cover a lot of cases and can be invoked from data read from a config file.
I've been mulling over how that kind of system should be structured, off and on, since your reply on #81. Being a functional programmer at heart, I come at it from the perspective of composability. It's clear you need some kind of system that takes a list of strings representing a command and returns several lists of strings representing the commands it would invoke, and also produces some (sometimes failing) method to turn desired modifications of the invoked commands back into a modification of the original command. Where it gets messy is that you can't always statically determine the subcommands. They often depend partially on runtime data, and you need some kind of sufficiently flexible structure to represent most cases of that, while also still being workable for the individual command's parsers.
The more I think about this problem as a whole, the more I think it basically all comes down to a composable element like that. Parsing shell and modifying it, in itself, comes down to almost the same operation. And the hard part will always come down to defining the boundary of the statically analyzable portion of the system in a useful way.
from resholve.
Woke up a little smarter today and realized that we can brute force this for now by adding a second parser that recognizes the closed form (well, it recognizes the actual argument).
I've cut a v0.8.1 release and opened a nixpkgs PR.
In case speaking it out loud helps me remember or search for it later, I separately committed a failing syntax/parser test so that there's a good example to point at.
from resholve.
Ah of course, the good old getopt
approach. normalize then use a simpler parser.
EDIT: Oh, I see. You're doing either/or. It should cover more cases at least, so long as people don't mix both forms on the same command line.
from resholve.
so long as people don't mix both forms on the same command line.
Nightmare fuel :)
from resholve.
Related Issues (20)
- `AttributeError` when resolving `install --strip-program` HOT 2
- `AttributeError` when resolving `sort --compress-program=`
- exercer rule not being applied HOT 3
- support mkDerivation `finalAttrs` call pattern? HOT 1
- Function argument substitution failed HOT 2
- Build error on package using resholve HOT 1
- failure to correctly resolve absolute paths shadowed by builtins HOT 2
- Port to python3? HOT 4
- Parsing error on &! HOT 3
- resholve Nix API mangling package meta HOT 1
- Issue parsing zsh nested expansion. HOT 1
- Port to Python 3 HOT 8
- when invoked with multiple scripts, `--prologue` only applies to the first HOT 2
- handling commands that run other commands in a different path-space (different system, chroot, vm, etc.) HOT 2
- position indicators may be wrong in tab-indented scripts
- look for "unresholved" paths
- doublecheck coproc and execer check
- Missing subsequent `! type <command>` replacements HOT 4
- `make ci` fails to `touch` files HOT 4
- `make update` fails to find `../../nixpkgs` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from resholve.