canva-public / js2nix

Node.js modules installation using Nix

js2nix

A tool that makes use of the Nix package manager to install Node.js dependencies declared in package.json and yarn.lock files. It is an experimental project to discover opportunities to use Nix for Node.js dependencies. Read more about the background problems here.

Goals of the project

• Provide incremental node_modules installations (don't build what have already been built) by turning every individual npm package into a self-containing Nix derivation
• Handle dependency cycles
• Allow overriding a package
• Allow package life-cycle scripts and their overrides
• Make the dependency graph explicit (expressed in Nix) and well-controlled with all the above
• Remove the need to be check generated files into a code-base, provides IFD if required
• Make the generation of the Nix expression pure, so no assumptions are made around missing SHAs, local packages locations, etc

Details

It is implemented as a CLI tool written in JavaScript and as a Nix library that picks up that tool and executes it internally to generate a Nix expression out of the given tuple of package.json & yarn.lock files in a pure manner as a separate Nix derivation, that then can be imported into the Nix runtime and the generated Nix derivations will be built via the provided Nix library to install Node.js dependencies.

Then the artifact can be symlinked to some local location as a node_modules folder or can be placed, or picked up by Nix as a part of NODE_PATH to make it available for the Node.js resolution mechanism. Also, every derivation that represents an npm package is a first-class citizen in Nix and can be used independently, which is a convenient way to provide Node.js based CLIs in Nix. That is, if the npm package exposes a binary, it will be picked up by Nix and being made available in PATH, with no additional effort.

A quick example

Let's create a file package.nix of a Nix expression with the following content:

with import <nixpkgs> { };

let
  js2nix = callPackage (builtins.fetchGit {
    url = "ssh://[email protected]/canva-public/js2nix.git";
    ref = "main";
  }) { };
  env = js2nix {
    package-json = ./package.json;
    yarn-lock = ./yarn.lock;
  };
in env.nodeModules

And then a node_modules folder can be created via:

 nix-build --max-jobs auto --out-link ./node_modules ./package.nix

The nodeModules is a Nix derivation that contains a compatible with Node.js module resolution algorithm layout. Note that the layout of the resulting node_modules is similar to what PNPM package manager is providing, that is not a flat layout but rather the canonical layout with symlinked (from the Nix store) npm packages into it.

To find out more about the project, its background, implementation details, how to use it please go to the documentation space.

js2nix's Issues

Issue building [email protected]

Hi, I have an issue when running the generated install script when this package is in the dependency closure. As you can see from the log output, it tries to symlink string-width into isaacs-clui but can't because it's already there.

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/1sz22m4gsixc21cblc4151nhqgr7wjdl-cliui-8.0.2.tgz
source root is package
setting SOURCE_DATE_EPOCH to timestamp 499162500 of file package/package.json
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "installPhase" }
installing
Executing Node.js install script for @isaacs/cliui ...
Error: EEXIST: file already exists, symlink '/nix/store/p9cxyx378j2s8y79980d0vy4jldprlc6-string-width-5.1.2/pkgs/[email protected]/node_modules/string-width' -> '/nix/store/mqs7wj5iybcv3g37sdp8w557k7c8vxp6-isaacs-cliui-8.0.2/pkgs/[email protected]/node_modules/string-width'

Pure nix yarn.lock parsing

Hi there,

I've implemented a pure nix yarn.lock parser: https://gist.github.com/bouk/95f565b744168a3a0fcf396c1ebb805e

I was thinking this can be used in js2nix so it can work without having to generate and commit a yarn.lock.nix file. This would make the user experience a lot nicer since you'd only have to run yarn update to upgrade a version for example, without having to then run js2nix as well. Let me know if this is useful and if you need any help integrating it!

Packaging a node package using js2nix

I've been building my node packages with buildNpmPackage. However, I'd like to switch to js2nix. buildNpmPackage relies upon a package-lock.json so seems inappropriate. How are others building packages with js2nix dependencies?

Happy path to run

• I've set up packages.nix
• I've even set up the override (though the help message was impressive, it wasn't immediately clear where to put it [lib.load])
• I've done the prefetching, as well
• I've run nix-build --max-jobs auto --out-link ./node_modules ./package.nix
• Now, how do I run the CLI in the repo:

# where to put overrides
{
    tree = js2nix.load ./yarn.lock {
      overlays = [
       (self: super: {
         "[email protected]" = super."[email protected]".override (x: {
           src = x.src.override {
             # Let Nix to recognise the type of archive so it unpacks it appropriately
             name = "5dd91d76ba088119836f60f514dc0a0b8b30e78d.tgz";
             sha256 ="1g7davl86xvbbfwirjj66xfbavv9ld9250bbni84wlcj0jmhy7xm";
           };
         });
       })
      ];
    };
}

Checks don't work for modules

Some of the NPM modules utilise new feature of Node.js that is called modules, that is declared as:

{
  ...
  "type": "module,
  ...
}

in package.json file. And the current implementation of checks, like:

node -e 'require("/nix/store/4dj4m2yanj5z1182ji9y47vir6w723bk-formdata-polyfill-4.0.10/lib")

doesn't work with that new module type.

I found that for the packages of the module type, this check would work:

node --input-type=module -e 'import  "/nix/store/4dj4m2yanj5z1182ji9y47vir6w723bk-formdata-polyfill-4.0.10/lib/formdata.min.js"'

that is, the check needs to add --input-type=module option and specify the file explicitly.

fysa: `placebo-npm`

https://github.com/svanderburg/placebo-npm had been brought to my attention and I don't want to hold private the knowledge of its existence.

It seems to be adding ecosystem capability in the same space than js2nix.

Wire things up in a flake

I've read the TL;DR.

But I'm looking at a wall to climb, now.

As others have pointed out interop is cheap with: https://github.com/edolstra/flake-compat

No `meta.description`

e.g. using numtide/devshell results in a not-so-descriptive listing:

[general commands]

  canva-js2nix
  menu         - prints this menu
  treefmt      - one CLI to format the code tree

Syntax error on `yarn.lock`

On this PR's HEAD inside the direnv-loaded devshell:

❯ js2nix -l ./yarn.lock -o ./yarn.lock.nix
SyntaxError: Unknown token: { line: 3, col: 2, type: 'INVALID', value: undefined } 3:2 in lockfile%  

Could you help solve this? I'd be delighted to boost adoption across our teams.

Alternatively, reproduce with:

nix eval "github:input-output-hk/cardano-js-sdk?ref=refs/pull/721/head#env"

Some sort of peer dependency automation

I'm trying to adopt js2nix in a react-native app but having to manually resolve each peer dependency in this environment has been... painful.

I very much like the fact js2nix is not opinionated and so guarantees package lookup reproducibility but I wish there was some way to automate the addDependency dance.