Use with npx about nodenv

[jasonkarns]

Don't forget to nodenv rehash after installing global modules, otherwise you won't have the shim in your PATH.

[tylercollier]

Thanks for that awesome explanation!

I think the reason I haven't had "problems" with this issue in a while is that when I see the "command not found, it exists in these node versions", I just go ahead and install it globally in the version of node that I'm in, which I realize is kind of against the point of npx.

I love nodenv and am extremely grateful for it so I'd be happy to help test the npx.bash script in any way you'd like. I'm guessing though that you're confident of the academic cases and want more "in the wild" tests, so here's what I'll do. I'll put a reminder on my calendar for a month from now and check in then at the latest, but if I encounter a scenario earlier I'll report. I'm currently a coding bootcamp instructor so I am often working with multiple versions of node so I bet I run into something before long :-)

[jasonkarns]

Yeah, to my knowledge most other version managers manipulate the PATH (and other env vars), whereas nodenv has static shims that live within an unmodified PATH, that do the bin lookup at invocation time.

Reopening to track conversation and for any further thoughts as people experiment.

[tylercollier]

I don't really know why it might be a good idea not to have the exec script included by default. But I'd vote to put it in the new version. That way if I want that behavior on all my machines, I simply update to the newest nodenv (a little easier than manually creating the npx.bash).

[jasonkarns]

Throwing in my initial first guesses without much research so... grain of salt. :)

When a module is installed globally, nodenv makes a shim for it. This shim is in your PATH. So according to a system which foo lookup, it will find the executable in nodenv's shims and report "Hey, we successfully found an executable matching foo in PATH. Cool." So my suspicion is that npx simply checks to see whether the desired binary exists in PATH; assuming that if it does, it can just invoke it and skip the "temporary installation". (I'm still not clear on whether npx's installation of modules is temporary or permanent.)

However, as far as nodenv is concerned, the existence of the shim doesn't guarantee that the executable is installed for the given version. When the shim is actually executed, nodenv checks if the matching executable is installed for the currently active version. If it is, it executes it; hunky dory. If not, it does a search through all the installed nodes to see which versions do have the desired executable; and then prints the helpful message that you saw.

foo not found; foo exists in these nodes:

At present I don't have any more feedback for you, or comments for npx. I need to investigate further to confirm my suspicions before forming any opinions about next steps. But this is what I suspect is the problem. I'll keep this issue open as you/others dig deeper and hopefully shed more light on npx's behavior.

[tylercollier]

I'll close this for now. I suspect it's something npx would have to deal with, so I'll move it there. FWIW the creator of npx said here that

npx has no explicit support for any version managers. It does some installation and $PATH-manipulation work

I'm guessing it's a difference between nodenv using shims where the rest of the node managers work more globally.

[ELLIOTTCABLE]

Okay, I can't even get npx to show up, with Nodenv:

❯ ~ nodenv global 8

❯ ~ nodenv version 
8 (set by /Users/ec/.nodenv/version)

❯ ~ npx                     
zsh: command not found: npx

❯ ~ npm i -g npm 
/Users/ec/.nodenv/versions/8.1.4/bin/npm -> /Users/ec/.nodenv/versions/8.1.4/lib/node_modules/npm/bin/npm-cli.js
/Users/ec/.nodenv/versions/8.1.4/bin/npx -> /Users/ec/.nodenv/versions/8.1.4/lib/node_modules/npm/bin/npx-cli.js
+ [email protected]
updated 1 package in 13.804s

❯ ~ [15s] which npm
/Users/ec/.nodenv/shims/npm

❯ ~ which npx
npx not found

Although, based on this Issue, it looks like it wouldn't work, even if I got it to install? :x

[jbcpollak]

I'm running into this problem with npm-merge-driver, which is a bit of a problem because merging package-lock.json files would be a huge win.

You can get npx to "work" by using the --ignore-existing flag, but that's a ton slower than just letting npx call the locally installed version.

I'm not sure how this would work technically, but ideally it would be great if nodenv could be npx aware (which seems more reasonable than the other way around) so that nodenv could allow npx to execute the locally installed version from another node version, or perhaps just ignore the local version only if its the shim that's going to error out.

[ELLIOTTCABLE]

Might I suggest nodexenv or something, that we could then alias to npx locally? (=

[jasonkarns]

So to re-summarize the issue here:

1. nodenv creates shims for all executables that exist in any managed node version
2. npx checks for the existence of an executable before installing+running the requested executable
3. npx will find the shim for an executable in PATH even if it's only installed in non-active nodes

So the solution, in short, is to ensure that nodenv's shims directory is not in PATH when npx is running. This can be accomplished with a simple exec hook which removes the shims dir from PATH, but only for the npx command.

The following hook should be placed at $(nodenv root)/nodenv.d/exec/ with a .bash extension. Filename doesn't matter, but I suggest: $(nodenv root)/nodenv.d/exec/npx.bash

https://github.com/nodenv/nodenv/blob/npx/nodenv.d/exec/npx.bash

I've test this locally, but haven't been running it for long. I would be grateful if you all (@tylercollier @ELLIOTTCABLE @jbcpollak ) could try this hook for a bit and see if you run into any problems.

If it works without any hitches, I'll consider either shipping this hook by default, or baking the behavior directly into the shim/exec logic.

edit: just pushed an experimental branch with this hook bundled https://github.com/nodenv/nodenv/tree/npx

[tylercollier]

I didn't use npx for a long time. Recently, to launch local executables within projects, I've been typing node_modules/.bin/EXECUTABLE_NAME, and then a friend mentioned npx, and I had never realized that npx worked that way (check for a file in node_modules/.bin for you first before installing a temporary copy). Since the time he mentioned it, I've been using npx without issue for a while, and hadn't even realized that perhaps I'm using a version from a different version of node. As in, perhaps I am using node 10 for some project, then I type npx SOME_COMMAND, and perhaps it's executing that with e.g. node 8's npx? How can I check? If it seems to be working for me, should I not worry? Because I'm guessing even though it may seem to work, it might hose me someday due to version mismatches. I'm happy to help but I'm not sure how to know if the exec hook made a difference or not.

[jasonkarns]

So you definitely don't have to worry about any executables running on other node versions. The nodenv shims will only invoke executables for the currently active node.

Say you invoke npx standard. The steps that follow are:

1. you start with nodenv's shims in your PATH (thanks to your .bashrc, et. al.)
2. when you invoke a shim (npx in this case, or node or eslint, etc) the shim invokes nodenv exec npx standard
3. nodenv-exec resolves the active node version (based on .node-version file et. al.), and prepends the selected node version's bin directory onto PATH and then executes the command (npx standard)
4. npx itself prepends $(npm bin) (local node_modules/.bin) onto PATH and then runs the subcommand (standard in this case)

Thus, by the time standard is invoked, your PATH has the local node_modules/.bin first, then the active node's bin directory second (and all global modules installed in that version), and finally nodenv's shims.

Scenario 1: You're in a project where the npx subcommand (standard in this case) is installed locally.
It'll run the local executable no problem, because the local node modules .bin/ is "first" in PATH.

Scenario 2: You're in a project where the subcommand is not installed locally; or you're not in a node project at all. But the subcommand is installed globally in the currently active node version. Say you want to run npx express-generator. In this case, express-generator wouldn't be found in local node modules, but it would be found in the currently active node version's bin directory (which is "second" in PATH). So it would run the globally installed executable no problem.

Scenario 3: You're in a project where the subcommand is not installed locally; nor is it installed globally in any of your managed node versions. This means nodenv does not have a shim for the subcommand at all. Say you want to run npx eslint. In this case, eslint wouldn't be found in the local node modules .bin, nor in the active node's bin, nor in nodenv's shims directory at all. Ergo, npx will install eslint on-demand and run it.

All the above scenarios give the desired behavior as a user of npx; and work already without this exec hook. Indeed, the observable behavior of npx is the same for a nodenv user as a non nodenv user. (perhaps with nvm, or just a homebrew-installed node) The only unique scenario that causes problems with nodenv and npx is the following:

Scenario 4: You're in a project where the subcommand is not installed locally; it is installed in node version 8, but not version 10; and say v10 is currently active. Say we want to run npx teenytest. Since it's installed in at least one of nodenv's managed versions, that means there exists a teenytest shim in $(nodenv root)/shims/. Thus: npx teenytest will not find teenytest in local node modules (first in PATH), will not find teenytest in node v10's bin (second in PATH), but will find the teenytest shim in nodenv's shims dir (third in PATH). Since npx has "found" the teenytest executable, it dutifully executes it. Of course, if you invoke an executable that is installed in another node version but not the currently active one, you get the following output:

$ teenytest
nodenv: teenytest: command not found

The `teenytest' command exists in these Node versions:
  8.9.1

Which is precisely the same output as if you had invoked npx teenytest (because eventually npx just invokes teenytest directly)

So if you haven't had any issues, then that means you're only invoking executables that are already installed either a) locally or b) globally in the active node version. And that's fine – it's the most common use-case after all. (primarily why this issue has languished for so long). As more and more node devs move away from globally installed modules, the chances of Scenario 4 occurring get slimmer and slimmer.

But regardless, the way this problem is solved is via the exec hook – which basically runs immediately before Step 3. Once the nodenv shim for npx has been invoked, there's no longer a reason for the shims to be in PATH at all (they've already done their duty at this point). So if the exec hook removes the shims dir from PATH before proxying the npx call, then the only possible executables to be found by npx are either a) installed locally or b) installed globally in the active node version. (Any shims created from global modules in other node versions will not be in PATH.) Thus npx will consider any executables it can't find as "not installed" and will install them on-demand as expected.

[tylercollier]

I posted this a few minutes ago with a different github account. Reposting it with my original account.

Update as promised. I tried it just now and it worked. I am on a new machine and I installed nodenv via homebrew. I have node versions 10.15.0 (set as global) and 11.6.0, and I installed create-react-app globally via npm in version 11.6.0. I then ran nodenv rehash. When I typed npx create-react-app blah, it said (as expected):

nodenv: create-react-app: command not found

The `create-react-app' command exists in these Node versions:
  11.6.0

I then put the npx.bash script where you said. I ran the same CLI line as above, and it worked! It ran create-react-app with node 10.15.0. Great!

I told you I am a bootcamp instructor and hope to report on "in the wild" conditions. I switched companies and my first class with this new company starts tomorrow, so I'll put another reminder on my calendar for a month from now and report back.

[tylercollier]

I haven't had the opportunity to show many others to use nodenv as I intended. However, I have multiple machines and today used a different machine than previous. I tried to use npx and got the error as originally described. I switched my nodenv to the npx branch and ran the npx command again and it worked perfectly. I'd say LGTM. Thanks for the great work!