Support npm-shrinkwrap? about ied

[STRML]

After today's left-pad fiasco, I think inclusion of a proper shrinkwrap file with dependencies identified by checksum is even more important. While unpublish is unfortunately not going away, at least identifying packages by hash could prevent the very real danger of someone (even someone at npm) republishing a package with the same version number.

According to npm, it's not possible to republish a module with the same version, even if the repo has been deleted. That's good, but it would be great if an ied shrinkwrap could go a step further and pin by author and hash.

Hopefully someday soon npm will support signing packages as well.

👍 for breaking compat with the existing shrinkwrap format for the purpose of adding functionality and preserving repeatable builds. Shrinkwrap is broken anyway (especially in npm@2) and has multiple redundant fields. There's very little lost there anyway.

Is anyone working on this?

[alexanderGugel]

@hone Awesome! Thanks for this explanation!
Some really good thoughts in there... didn't decide on a final design yet... thanks for your input! I appreciate it!

Will look more into this over the eastern holidays... hopefully we're going to have something useable by then... Bundler is definitely a good resource and learning resource in regards of shrinkwrapping...

[STRML]

@danielb2 Try https://github.com/skybet/shonkwrap or the like; I copied the file into our own project's scripts/shrinkwrap.sh and associated helper scripts. I share your disappointment with npm shrinkwrap as a whole.

[alexanderGugel]

Yes, some sort of npm shrinkwrap is planned. I'm not sure if we're gonna come up with an equivalent or stick with the way npm handles it.

[glenjamin]

I would like to 👍 this - I'm really impressed with the CAS approach for the node_modules folder, but without a shrinkwrap option I'm loathe to adopt this for production use yet.

It would be great to use an npm compatible shrinkwrap format, to allow members of the dev team to still use npm if desired.

Is there anything I can do to help out?

[hunterloftis]

npm's shrinkwrap is riddled with issues. Different parts of npm support shrinkwrap to different levels - for instance, npm-prune is shrinkwrap unaware.

Given the issues with shrinkwrap's implementation (both in npm 2 and 3), many users avoid it in favor of just pinned dependencies in package.json. This is not a perfect solution, but it is pragmatic, and I recommend it to many of our users at Heroku when they run into edge-cases in the current shrinkwrap implementation.

Additionally, the workflow required when using shrinkwrap can be onerous.

If ied can implement shrinkwrap in an npm-compatible way, that would be great, but the node.js community needs an effective, fast, and simple dependency-locking solution. If this can best be accomplished outside of npm-shrinkwrap, then I'd say go for it.

[Rush]

Locking dependencies on package.json is no solution as it locks only direct dependencies, all indirect ones will happily install at whatever latest version matches their requirements, it's a recipe for breakage.

I'd love to use ied but it's a no go for without full dependency locking.

[hunterloftis]

Locking dependencies on package.json is no solution as it locks only direct dependencies, all indirect ones will happily install at whatever latest version matches their requirements, it's a recipe for breakage.

The vast majority of dependency-related app breakage happens from floating top-level dependencies (the default ^ and ~). Bumps in nested dependencies are a much rarer source of problems.

[Rush]

The vast majority of dependency-related app breakage happens from floating top-level dependencies (the default ^ and ~). Bumps in nested dependencies are a much rarer source of problems.

Maybe but if it happens you're screwed and debugging it is such a pain. It happened to me. A good solution is a systematic one ...

[glenjamin]

The vast majority of dependency-related app breakage happens from floating top-level dependencies

It seems to me that it would logically follow that my top-level dependencies break because of their floating top-level dependencies?

Either way, I'm not especially concerned with shrinkwrap as a way to avoid breakage - I'm more interested in it as a tool to produce consistently reproducible builds across multiple machines. When I check out a particular revision and run npm install (in an app), I want the same result every time.

[hunterloftis]

Of course - we need a system to lock / shrinkwrap dependencies. My point is that the need for that, the end result, is far more important than compatibility with today's shrinkwrap... especially given that today's solution is a partial one.

[alexanderGugel]

Agreed.

[Rush]

Indeed, no compatibility needed.

[alexanderGugel]

I was even thinking of actually producing a script that when executed installs all dependencies = instead of generating a declarative shrinkwrap file. That way ied wouldn't have to be installed on all machines.

[glenjamin]

That would be awesome, especially if it handled checksums and idempotency!

[hunterloftis]

The script idea is interesting; a couple questions:

• How would a script be able to deal with different environments? For instance, npm takes things like proxy values into consideration, can be run with different flags for dev and prod, etc. All these options seem like they could lead to really complex installation scripts.
• If a bug exists in the ied-locking implementation, that bug may exist in all scripts made with that version of ied, since dependencies won't be declarative, correct?

I'm curious to see more of the benefits of a script-based, vs declarative solution. Is the primary goal just to make it so users don't all have to install ied?

[alexanderGugel]

How would a script be able to deal with different environments? For instance, npm takes things like proxy values into consideration, can be run with different flags for dev and prod, etc. All these options seem like they could lead to really complex installation scripts.

Most likely through some sort of global config (very simple .npmrc-like file) or environment variables.

If a bug exists in the ied-locking implementation, that bug may exist in all scripts made with that version of ied, since dependencies won't be declarative, correct?

Yup. But I think it would less error-prone to generate a fairly simple, imperative script than a more complicated declarative manifest via a ied.

I'm curious to see more of the benefits of a script-based, vs declarative solution. Is the primary goal just to make it so users don't all have to install ied?

Primarily, yes. Forcing users to use npm to install ied to install their dependencies might be too complicated / annoying for a lot of users. Just running node shrinkwrapped.js might be the easiest solution.

[hunterloftis]

Hey @hone, talk to us about your Bundler 1.0 experience. Esp. --standalone.

[hone]

Hi! I work with @hunterloftis at heroku and I worked on Bundler, the Ruby dependency manager, for a few years. One of the features that was added in 1.1, was this flag called --standalone. The premise behind it was to allow a prebundled app to be run without needing Bundler installed for packaging purposes. I think a feature like this would solve #23 (comment). As an example of how this works:

In Bundler, we have a Gemfile which is equivalent to a package.json. In this case I have one dep.

source "https://rubygems.org"

gem 'rspec'

A Gemfile.lock gets produced by running bundle install which is a resolution graph of all the dependencies and their dependencies. In node land, this would be more complicated b/c of the forest of dependencies. The benefit of something like this is when you want to remove/update a particular dependency, you get conservative updating. What I mean by this is when you want to update a dependency A, the existing resolver can base the new solution upon an existing solution and only introduce the minimum number of dependency changes needed to update A. So, if it could get away with just updating A, it would do that. Also, with removing a dependency, in Bundler you can simply remove that dependency from the Gemfile and re-run bundle install and the Gemfile.lock will have the new solution. Contrast with shrinkwrap where removing a node module either involves reshrinkwrapping after you removed a single line from your package.json. You can also manually copy the diffs / changes you want, but that can get nasty.

GEM
  remote: https://rubygems.org/
  specs:
    diff-lcs (1.2.5)
    rspec (3.4.0)
      rspec-core (~> 3.4.0)
      rspec-expectations (~> 3.4.0)
      rspec-mocks (~> 3.4.0)
    rspec-core (3.4.4)
      rspec-support (~> 3.4.0)
    rspec-expectations (3.4.0)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.4.0)
    rspec-mocks (3.4.1)
      diff-lcs (>= 1.2.0, < 2.0)
      rspec-support (~> 3.4.0)
    rspec-support (3.4.1)

PLATFORMS
  ruby

DEPENDENCIES
  rspec

--standalone produces a setup.rb file that looks something like this:

require 'rbconfig'
# ruby 1.8.7 doesn't define RUBY_ENGINE
ruby_engine = defined?(RUBY_ENGINE) ? RUBY_ENGINE : 'ruby'
ruby_version = RbConfig::CONFIG["ruby_version"]
path = File.expand_path('..', __FILE__)
$:.unshift "#{path}/"
$:.unshift "#{path}/../#{ruby_engine}/#{ruby_version}/gems/diff-lcs-1.2.5/lib"
$:.unshift "#{path}/../#{ruby_engine}/#{ruby_version}/gems/rspec-support-3.4.1/lib"
$:.unshift "#{path}/../#{ruby_engine}/#{ruby_version}/gems/rspec-core-3.4.4/lib"
$:.unshift "#{path}/../#{ruby_engine}/#{ruby_version}/gems/rspec-expectations-3.4.0/lib"
$:.unshift "#{path}/../#{ruby_engine}/#{ruby_version}/gems/rspec-mocks-3.4.1/lib"
$:.unshift "#{path}/../#{ruby_engine}/#{ruby_version}/gems/rspec-3.4.0/lib"

If you don't know Ruby, $: represents the load path of where Ruby looks for libraries, so require can find the files. This file manually sets up the load path so you can pull all the deps specified by your Gemfile.lock. Now, if this is distributed out/packaged you don't need the bundler runtime at all.

I hope that helps. I'd be happy to discuss things more if people are interested. I also understand node has a unique / different problem space with dependencies, so maybe not everything applies.

[hone]

@alexanderGugel :) glad it was informative. Feel free to ping if you have any questions I can help with as your exploring this.

[oheck]

@alexanderGugel any estimate on when there will some kind of shrinkwrap support?

[danielb2]

npm is a constant let down, and npm shrinkwrap isn't working according to docs. I was hoping ied would have something that allowed me to update things

[milesj]

Just my 2 cents but NPM shrinkwrap is just terrible all around -- I literally have issues with shrinkwrap on a daily basis.

Ruby's gems or PHP's composer lock files work wonderfully, and if IED could function more like those, it would be much preferred.

[STRML]

It is terrible, but not because of any flaw in the format. Generating that file has been buggy for years and required all manner of hacks, but that doesn't discount it when it is generated correctly.

[jeffgran]

Thumbs way up for taking a cue from the rubygems/bundler system. Bundler's way of handling dependencies and the Gemfile.lock file work really well for production deploys. I would LOVE it if there was an alternative to npm that worked like bundler for locking down versions.