opensuse / obs-service-node_modules

Build RPM packages using node modules offline

By default, npm download dependencies from registry.npmjs.org and hides the details in the node_modules subdirectory. Its job is to resolve version dependencies and provide it to Node application in such a way that it satisfied the dependencies and does not conflict with other dependencies. To be able to build and rebuild a package from sources, we will need to be able to install and possibly update these dependencies in a networkless environment like OBS.

When npm installs dependencies, it will create a package-lock.json that will contain the entire list of packages that can possible exist in the node_modules directory structure.

The purpose of this tool is to parse package-lock.json and prepare all externally download sources for use by npm during rpmbuild.

runtime requirements

npm 7+ is required to produce package-lock.json with lockfileVersion:2

As OBS service

• Get package-lock.json with localfileVersion: 2. For example,
  • npm install --package-lock-only --legacy-peer-deps --ignore-scripts with npm 7+
  • --legacy-peer-deps is required to fetch peer dependencies from remote locally so they are available during peer resolution in the VM. Without this you may get additional warnings during install.
• Make sure to put the package-lock.json next to the spec file and remove it from the sources. Sources should only have package.json, even if they ship a compatible package-lock.json
• Add the following line to the spec file:

  %include  %{_sourcedir}/node_modules.spec.inc
  

• Create file _service with the following content:

  <services>
    <service name="node_modules" mode="manual">
      <param name="cpio">node_modules.obscpio</param>
      <param name="output">node_modules.spec.inc</param>
      <param name="source-offset">10000</param>
    </service>
  </services>
  

• osc service manualrun
  • this generates the NPM dependency archive along with its source URLs
• osc add node_modules.obscpio
• osc add node_modules.spec.inc
• osc commit

Example

Source10:       package-lock.json
Source11:       node_modules.spec.inc
%include        %{_sourcedir}/node_modules.spec.inc
BuildRequires:  local-npm-registry

[...]

%prep
%setup
local-npm-registry %{_sourcedir} install --also=dev

[...]

%build
npm run build

In Practice

https://build.opensuse.org/package/show/openSUSE:Factory/cockpit-podman

External Resources

https://github.com/openSUSE/npm-localhost-proxy

obs-service-node_modules's Issues

Error if resolved dependency lacks Integrity

Traceback (most recent call last):
  File "/usr/lib/obs/service/node_modules", line 591, in <module>
    sys.exit(main(args))
  File "/usr/lib/obs/service/node_modules", line 343, in main
    process_packagelock_file(js)
  File "/usr/lib/obs/service/node_modules", line 309, in process_packagelock_file
    collect_deps_recursive("", js["dependencies"])
  File "/usr/lib/obs/service/node_modules", line 294, in collect_deps_recursive
    collect_deps_recursive(path, entry["dependencies"])
  File "/usr/lib/obs/service/node_modules", line 290, in collect_deps_recursive
    integrity = entry["integrity"]
KeyError: 'integrity'
Aborting: service call failed:  /usr/lib/obs/service/node_modules --cpio node_modules.obscpio --output node_modules.spec.inc --source-offset 10000 --outdir /tmp/home:DocB:branches:devel:languages:python/tryton-sao/tmpxfxuqph6.node_modules.service

This is caused by resolved entries lacking integrity entries,

"dependencies": {
        "align-text": {
          "version": "0.1.4",
          "resolved": "https://registry.npmjs.org/align-text/-/align-text-0.1.4.tgz",
          "bundled": true,
          "dev": true,
          "optional": true,
          "requires": {
            "kind-of": "^3.0.2",
            "longest": "^1.0.1",
            "repeat-string": "^1.5.2"
          }
        },

Is it possible to use modules located for example on github?

Is it possible to use modules located for example on github?
The module is loaded and packaged in node_modules, but when it is built, it still tries to load over the Internet.
For example, the package-lock.json file contains a module

"eslint-plugin-mattermost": "github:mattermost/eslint-plugin-mattermost#46ad99355644a719bf32082f472048f526605181",
...
"node_modules/eslint-plugin-mattermost": {
      "resolved": "git+ssh://[email protected]/mattermost/eslint-plugin-mattermost.git#46ad99355644a719bf32082f472048f526605181",
      "dev": true
    },
....
"eslint-plugin-mattermost": {
      "version": "git+ssh://[email protected]/mattermost/eslint-plugin-mattermost.git#46ad99355644a719bf32082f472048f526605181",
      "dev": true,
      "from": "eslint-plugin-mattermost@github:mattermost/eslint-plugin-mattermost#46ad99355644a719bf32082f472048f526605181"
    },

it is loaded and packaged with all modules
but when building it throws an error

npm ERR! request to https://codeload.github.com/mattermost/eslint-plugin-mattermost/tar.gz/46ad99355644a719bf32082f472048f526605181 failed, reason: getaddrinfo EAI_AGAIN codeload.github.com

Errors when dependencies specify multiple integrity values

We are seeing an issue when a package-lock.json specifies dependencies with multiple integrity values:

Files could not be expanded: service error: ValueError: too many values to unpack (expected 2)

service node_modules failed:
time="2023-09-18T16:21:19+02:00" level=warning msg="Path \"/etc/SUSEConnect\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping"
time="2023-09-18T16:21:19+02:00" level=warning msg="Path \"/etc/zypp/credentials.d/SCCcredentials\" from \"/etc/containers/mounts.conf\" doesn't exist, skipping"
Running /usr/lib/obs/service//node_modules --cpio node_modules.obscpio --output node_modules.spec.inc --source-offset 10000 --outdir /var/cache/obs/VJvo7VsTt4d9/out
Traceback (most recent call last):
File "/usr/lib/obs/service//node_modules", line 624, in
sys.exit(main(args))
File "/usr/lib/obs/service//node_modules", line 376, in main
process_packagelock_file(js)
File "/usr/lib/obs/service//node_modules", line 341, in process_packagelock_file
collect_v3_deps(js["packages"])
File "/usr/lib/obs/service//node_modules", line 326, in collect_v3_deps
add_standard_dependency(parse_supported_fetch_url(url), integrity, module, path)
File "/usr/lib/obs/service//node_modules", line 230, in add_standard_dependency
algo, chksum = integrity.split("-", 2)
ValueError: too many values to unpack (expected 2)

Here you have an example of a problematic case:

    "node_modules/array-unique": {
      "version": "0.3.2",
      "resolved": "https://registry.npmjs.org/array-unique/-/array-unique-0.3.2.tgz",
      "integrity": "sha1-qJS3XUvE9s1nnvMkSp/Y9Gri1Cg= sha512-SleRWjh9JUud2wH1hPs9rZBZ33H6T9HOiL0uwGnGx9FpE6wKGyfWugmbkEOIs6qWrZhg0LWeLziLrEwQJhs5mQ==",
      "dev": true,
      "engines": {
        "node": ">=0.10.0"
      }
    },

As you can see, the integrity field contains the following values, which is causing the OBS service to fail:

• sha1-qJS3XUvE9s1nnvMkSp/Y9Gri1Cg=
• sha512-SleRWjh9JUud2wH1hPs9rZBZ33H6T9HOiL0uwGnGx9FpE6wKGyfWugmbkEOIs6qWrZhg0LWeLziLrEwQJhs5mQ==

Add support for github: schema

dependencies: {
"sanitize-html": "github:apostrophecms/sanitize-html#3c7f93f2058f696f5359e3e58d464161647226db"
}

current git+https support could be extended here

resolve arch specific binaries

in cockpit-tukit we depend on esbuild, esbuild has arch specific binaries as optional deps, eg @esbuild/linux-x64 we currently have hacked around this but it would be nice if obs-service-node_modules could add the required arch specific packages without intervention

Support multiple versions of the same package

obs-service-node_modules does not support multiple versions of the same package, it appears multiple versions overwrite each other and as often javascript projects have multiple versions of the same package through peer deps. This causes build issues and in the worst case could cause silent miscompliation issues

Service stopped working as integrity is a list not a string

osc service manualrun
Traceback (most recent call last):
  File "/usr/lib/obs/service/node_modules", line 595, in <module>
    sys.exit(main(args))
  File "/usr/lib/obs/service/node_modules", line 347, in main
    process_packagelock_file(js)
  File "/usr/lib/obs/service/node_modules", line 313, in process_packagelock_file
    collect_deps_recursive("", js["dependencies"])
  File "/usr/lib/obs/service/node_modules", line 295, in collect_deps_recursive
    add_standard_dependency(url, integrity, module, path)
  File "/usr/lib/obs/service/node_modules", line 217, in add_standard_dependency
    algo, chksum = integrity.split("-", 2)
ValueError: too many values to unpack (expected 2)
Aborting: service call failed:  /usr/lib/obs/service/node_modules --cpio node_modules.obscpio --output node_modules.spec.inc --source-offset 1000 --outdir /home/asn/workspace/package/obs/home:gladiac:ig-klettern/routenbuch/tmpyawja9vu.node_modules.service

There are entries with two checksums:

"integrity": "sha1-Sb1jMdfQLQwJvJEKEHW6gWW1bfk= sha512-k92I/b08q4wvFscXCLvqfsHCrjrF7yiXsQuIVvVE7N82W3+aqpzuUdBbfhWcy/FZR3/4IgflMgKLOsvPDrGCJA==",

package-lock.json.txt

Please add a license

https://build.opensuse.org/package/show/openSUSE:Tools/obs-service-node_modules claims that the package is licensed under the MIT license, but this repository has no license. If the MIT license is correct, then I can submit a PR.

update to support package-lock.json with lockfileVersion": 3

npm perior to 9 now gives package-lock.json with lockfileVersion": 3
please update sevice to support this file format

Question: How to handle projects that require Yarn.

I am currently trying to package
https://github.com/marktext/marktext

They require yarn, which is hardcoded in their scripts, so npm install --package-lock-only --legacy-peer-deps fails.
Is it possible to create a yarn.lock and rename it or is npm required for the service to work?

Doesn't download packages not in npm registry

if package.json has explicit URL for package, like,

"olm": "https://packages.matrix.org/npm/olm/olm-3.2.1.tgz",

then the package-lock.json will contain this URL in version: as version: is never resolved. Service then fails with

WARNING:entry olm has no download

package-lock:

    "node_modules/olm": {
      "version": "3.2.1",
      "resolved": "https://packages.matrix.org/npm/olm/olm-3.2.1.tgz",
      "integrity": "sha512-B87bTpGIGieuV2FNauChjjQtVltwTGagQFoHm+3Dcse4amKAAGJB/I54dnP/JtbHZ+RYVoApM2OQ46Z4VH6eNg==",
      "license": "Apache-2.0"
    },
...
   "olm": {
      "version": "https://packages.matrix.org/npm/olm/olm-3.2.1.tgz",
      "integrity": "sha512-B87bTpGIGieuV2FNauChjjQtVltwTGagQFoHm+3Dcse4amKAAGJB/I54dnP/JtbHZ+RYVoApM2OQ46Z4VH6eNg=="
    },

fail on unsafe integrity

Make sure the integrity is maintained with hashes with safe algorithms provided by trusted user input and otherwise fail hard.

When I just skimmed the source to check how it deals with file hashes, it seems the default is currently unsafe. E.g. it sometimes generates hashes. Probably partly after #16 . But current node .lock files have good hashes. (These hashes are presently untrusted, but one thing after another.)

While a default off option to get the present behavior would be ok, maybe there is a better way? E.g. it might be preferable to have a file from the developer with hashes as a substitute for such cases, so that they can check these out of band. But I currently don't know why a .lock file would have no hashes.

make archive reproducible

It is probably possible to make the produced archive reproducible. Currently even if the content is the same, the archive is not:

> diffoscope systemsmanagement\:cockpit/cockpit/node_modules.obscpio ~/systemsmanagement\:cockpit/cockpit/node_modules.obscpio
[snip]
│ --rw-r--r--   1     1006      100     2765 2022-07-19 10:27:27.000000 yocto-queue-0.1.0.tgz
[snip]
│ +-rw-r--r--   1     1000      100     2765 2022-01-28 14:23:18.000000 yocto-queue-0.1.0.tgz

To solve this probably, after writing the file in:
https://github.com/openSUSE/obs-service-node_modules/blob/master/node_modules.py#L451
Set owner and group to nobody and mtime to 0.

obs-service-node_modules git project doesn't have tag

It will be nice to have at least one tag in the git project.
Useful to generate an archive/package

Thanks