I'm Kat!
I'm a Rust, C#, and JavaScript developer working at Microsoft and I do a bunch of open source stuff.
You can also find me on Mastodon as @[email protected] and Matrix as @kat:zkat.tech, or on Discord as kat#8645
.
standalone ci-oriented package installer for npm projects (moved)
Home Page: https://github.com/npm/libcipm
License: MIT License
I'm Kat!
I'm a Rust, C#, and JavaScript developer working at Microsoft and I do a bunch of open source stuff.
You can also find me on Mastodon as @[email protected] and Matrix as @kat:zkat.tech, or on Discord as kat#8645
.
This is a continuation of #45, a side-effect of the provided fix for it in #46 and was introduced in [email protected]
.
For the case where a default install
script is used and a binding.gyp
file is present in the package root, npm ci
will fail with packages that target a different platform that the one currently running.
To reproduce on Windows, use the following package.json
, run npm install
to create the package-lock.json
, then npm ci
.
Since [email protected]
is the current latest, this is happening right now in [email protected]
grunt-ts
has a dependency on fsevents
which is only supported on Mac OS (darwin
), so it fails on Windows
{
"dependencies": {
"grunt-ts": "^5.0.0-beta.4"
}
}
$ npm ci
npm WARN prepare removing existing node_modules/ before installation
> [email protected] install d:\repos\z\node_modules\fsevents
> node-gyp rebuild
d:\repos\z\node_modules\fsevents>if not defined npm_config_node_gyp (node "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" rebuild )
Traceback (most recent call last):
File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\gyp_main.py", line 16, in <module>
sys.exit(gyp.script_main())
File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 545, in script_main
return main(sys.argv[1:])
File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 538, in main
return gyp_main(args)
File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 523, in gyp_main
generator.GenerateOutput(flat_list, targets, data, params)
File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 2019, in GenerateOutput
sln_projects, project_objects, flat=msvs_version.FlatSolution())
File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1778, in _GatherSolutionFolders
return _DictsToFolders('', root, flat)
File "C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1731, in _DictsToFolders
for folder, contents in bucket.iteritems():
AttributeError: 'MSVSProject' object has no attribute 'iteritems'
gyp ERR! configure error
gyp ERR! stack Error: `gyp` failed with exit code: 1
gyp ERR! stack at ChildProcess.onCpExit (C:\Users\rrey\AppData\Roaming\npm\node_modules\npm\node_modules\node-gyp\lib\configure.js:336:16)
gyp ERR! stack at ChildProcess.emit (events.js:127:13)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:209:12)
gyp ERR! System Windows_NT 10.0.17134
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\rrey\\AppData\\Roaming\\npm\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd d:\repos\z\node_modules\fsevents
gyp ERR! node -v v9.7.1
gyp ERR! node-gyp -v v3.6.2
gyp ERR! not ok
added 77 packages in 5.201sactTree: info lifecycle [email protected]~install: [email protected]
Perhaps enable this as part of --low-mem
. So it will also reduce memoization in pacote, etc.
git deps in npm have their deps installed and prepare
scripts executed. We'll need to do the same for any git dependencies that go through cipm.
The best way to do this is probably just to do npm pack <git url>
and let npm take care of it.
Currently, npm ci
runs the postinstall script if it is present, which works well enough, but I would like a little more control after installing during npm ci
by running a postci script instead.
My use case is with CI services in a lerna maintained Git repo, where I would like to run lerna bootstrap
in a postinstall script, while running lerna bootstrap --ci
in a postci script.
(I found a similar request on npm.community.)
I was asked to open a corresponding issue for this problem in this repository. I won't copy the full body of that issue to this one, though. Here's the minimum repro though:
$ mkdir test-npm-ci-only-dev
$ cd test-npm-ci-only-dev
$ npm install -D typescript
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN [email protected] No description
npm WARN [email protected] No repository field.
+ [email protected]
added 1 package from 1 contributor in 1.246s
$ rm -r node_modules/
$ npm ci --only=dev
added 0 packages in 0.019s
It would be useful to allow installing without a package.json
file present, using only package-lock.json
. This would be particularly useful in Docker CI environments, where bumping the version
field in package.json
otherwise causes the Docker cache to be invalidated even if no new dependencies were added.
Running cipm
(or through npm ci
) in a directory with only package-lock.json
results in an ENOENT
error for package.json
:
ENOENT: no such file or directory, open '/Users/victor/c/p/test-ci/package.json'
Error: ENOENT: no such file or directory, open '/Users/victor/c/p/test-ci/package.json'
Is package.json
actually used for anything? The npm docs for package-lock.json
do read:
version
The version of the package this is a package-lock for. This must match what's in package.json.
So maybe doing this would be a bad idea for some reason I don't understand?
Why does cipm
need to be a separate tool?
Couldn't it be a command line flag for the existing npm
tool?
npm 6.x ci doesn't seem to respect the --no-optional
flag or optional=false
setting in .npmrc
.
I'm trying to switch our CI build to using npm ci
to speed up time, but it ends up taking more time because one of the optional dependency reaches out to the internet, causing the build to time out (our CI server is in an air-gapped environment).
I think I'd like to block initial release on having at least some baseline i18n support. My preference would be to make it similar to the way npx
does it, by using y18n
.
Had a user with a messed-up system try it out and it turns out that if node-gyp
isn't available globally, cipm can't run gyp. npm works around this by bundling its own node-gyp so you know it'll always be available.
npm itself now uses pacote@8, which has this fix (zkat/pacote#151), but I still sometimes encounter zkat/pacote#142 using [email protected] with Node 6.13.1 on macOS. The reason appears to be that cipm still has the old pacote dependency.
On virtual machines no limited(limited by os.cpus) workerFarm use many thread and fail with coredump.
Maybe we can add some environment variable for redefine default maxConcurrentWorkers
in:
Lines 13 to 16 in 89afe14
More details in: https://npm.community/t/npm-ci-failed-child-send-is-not-a-function/726/8
install
and postinstall
should not be run until the package's dependencies have finished installing and running their own install
and postinstall
.
Getting this working will involve parsing the requires:[]
fields on each dependency and building an ordered tree. Preferably, this tree will not just be flattened to a single list, so we have the ability to parallelize scripts at some point.
In order to support cache configuration, offline mode, private packages, alternative registries, etc, etc, we need to pass a blob of various pacote-related configs through. See lib/config/pacote.js
in npm for an idea of what's needed.
These should also have similar or identical semantics to how they work in npm, and the options should be visible with cipm -h
.
Hi! Thanks for awesome project, hope to see it along with npx as part of node install!
I'm trying to suppress all logs (except final message added X packages in Ys
) by calling cipm --loglevel error
, but it seems like does nothing - anyway I see info about each package and fetch info if cache is clear.
NodeJS 8.9.4, cipm 0.9.1
From #34
I was reading about the whole package-lock.json fiasco where the lock file wasn't really locking dependencies which is how I found a reference to cipm.
I was also looking for how to use the package-lock.json
file instead of package.json
to install the packages. NPM docs about package-lock.json nor package-locks tell you how you use it "only" what they are for:
Describe a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies.
So I searched and found cipm
. This worked fine at my local machine, but at Gitlab CI it didn't work. After some debugging I found out that the command didn't log errors but installed only the half of the dependencies right e.g. node-sass was not build as well as dependencies pulled in via git instead of npm registry. I searched again and found in an archived NPM issue that npm ci
is out. This works as expected.
And now I'm confused ๐
Does npm ci
use libcipm from this project or only cipm
? In my opinion it would make sense to send people to npm ci
instead of cipm
in the Readme.
Looks like we're not gonna be able to get away with just invoking npm itself after all. We're gonna have to write config stuff ourselves or patch npm to allow outputting these vars
cipm doesn't link binaries for deps yet, so lifecycle scripts that use them... can't.
This was first opened upstream to npm
at npm/npm#20275
It appears that the root cause of the upstream issue is that cipm
is not considering the default value of the install script:
"install": "node-gyp rebuild":
If there is a binding.gyp file in the root of your package and you haven't defined your own install or preinstall scripts, npm will default the install command to compile using node-gyp.
Detection for the binding.gyp
file should be run and pkg.scripts.install
should be added to the pkg
object before the install
hook is called here:
Line 285 in 08ed1cc
It looks like there's a difference in behavior between npm and cipm here, where bins aren't linked correctly by cipm in some cases. There's a pretty good repro in a gist by @mollerse, using the following package.json
:
{
"name": "npm-ci-repro",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"test": "rm -rf node_modules package-lock.json && npm i && ls node_modules/.bin && npm ci && ls node_modules/.bin"
},
"keywords": [],
"license": "ISC",
"devDependencies": {
"json2csv": "3.11.5",
"npm-run-all": "4.1.2"
},
"dependencies": {
"d3": "4.11.0"
}
}
I haven't looked yet to see if this bit of functionality has already been extracted from npm core or whether it is planned or not, but it sure would be nice!
It looks like when #58 was resolved by merging #59 and libcipm
was published to npm as v3.0.3
, it was actually tagged and published as v3.0.3
without the merged fixes from #59.
The latest
branch does indeed include the fixes from #59, however you can also see that the package.json file was not updated properly to v3.0.3
.
Would it be possible to re-tag latest
as v3.0.4
and republish? At least that way this fix could be included whenever the next NPM
release is made available.
Thanks!
Passing --ignore-scripts
seems to have no effect, or at least doesn't do what I would expect it to do: ignore preinstall
/install
/postinstall
scripts of dependencies during installation.
I noticed there is at least code for accepting the parameter, but haven't looked further yet to see where it might be getting dropped.
I also haven't looked at #6 yet to see if the refactor inadvertently fixes this.
Hi, for latest libcipm 3.0.3 a vulnerable version of dependency npm-lifecycle was reported in npm audit. Could you please npm audit fix
your package and publish to npm?
npm audit output follows:
High Arbitrary File Overwrite
Package tar
Patched in >=4.4.2
Dependency of npm
Path npm > libcipm > npm-lifecycle > node-gyp > tar
More info https://nodesecurity.io/advisories/803
It just crashes right now.
Keep in mind that this isn't as simple as ignoring failures: Any dependencies of the optionalDep which are not used by anything else need to be removed.
So I got here because I was searching for a reliable way to install javascript dependencies based on a package-lock.json file. Several stack-overflow and git-hub issues linked to this repo.
Would my typical use case be to install libcpm globally or in my project as a dev-dependency?
How do I use the API for libcipm? Is it identical to the API for cipm?
What is the difference between libcipm and cipm and do I need both?
I would really appreciate an example of how to use this library to install javascript dependencies based on the versions that are "locked" in an already generated package-lock.json file (if that is indeed what this library intends to do).
There are several places in cipm when an object is being passed to .then()
in bluebird promise chain.
It works but it causes warning message as described here: https://github.com/petkaantonov/bluebird/blob/master/docs/docs/warning-explanations.md
So using npm ci creates a long warning stack with messages like
(node:21662) Warning: .then() only accepts functions but was passed: [object Object]
Everything still works but that's very frightening and I think that it is important to fix this warning and ask npm to bump cipm version. I will add pull request here in several minutes.
It appears that currently the --no-bin-links
option - intended to avoid creating symlinks when doing npm install
- is not supported in this API. Is this an intentional design decision, or something that just hasn't been implemented yet?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.