ironcorelabs / recrypt-node-binding Goto Github PK
View Code? Open in Web Editor NEWBindings from recrypt-rs to work with NodeJS
Home Page: https://docs.ironcorelabs.com
License: GNU Affero General Public License v3.0
Bindings from recrypt-rs to work with NodeJS
Home Page: https://docs.ironcorelabs.com
License: GNU Affero General Public License v3.0
Publish version to match recrypt to designate "production ready" usage.
When we compile the Neon node-binding source, the Node addon binary that it generates is not portable between architectures. This prevents us from being able to compile the source and publish it to NPM as we don't know the platform the user is installing from. Having users compile the addon at install time isn't very feasible since the chance that they have the various build dependencies (Rust, Neon, etc) is very minimal. Instead, we need to generate compiled binaries for various platforms and have those be retrieved at install time depending on the environment we're present in.
This is a known issue within Neon, but not something that appears to be actively developed or discussed even.
A rough parallel to what we're doing within Neon is node-gyp which is a tool for building Node addons from C++ code for Node. Where Neon helps us build Rust code into a native Node module, node-gyp
does the same thing from C++ to native Node modules. As node-gyp
is a more stable build tool, there's a larger ecosystem of tooling around dealing with this issue.
What appears to be the most popular solution for handling the pre-compiled binary problem is the node-pre-gyp toolkit. This project aims to make it easy for distributing Node addons via pre-compiling binaries for various platforms. The quick and dirty workflow is
node-pre-gyp
for the location of your binaries as well as where they are hostedpostinstall
script to your package.json so that when users run NPM install on it, once it's been installed it discovers the machines platform and requests the appropriate binary from the AWS bucket, extracts it and configures all paths.node-gyp
if a binary cannot be found for the users machine.This rough workflow is about the same as we'd ideally do with our node-binding. We likely won't have support for falling back to compiling locally given the lifting required there, but everything else should work roughly the same.
Other various resources/thoughts I've found so far after looking into this
By default node-pre-gyp
defaults to uploading/downloading to AWS buckets. There are tools such as node-pre-gyp-github that upload to GitHub releases. It seems like using GitHub releases for our binaries might be a good idea. Alternatively we would probably use a GCP bucket.
Building the binaries for various platforms should be able to easily be done via Travis and there are a number of examples about how this is done for node-pre-gyp
projects. Travis can build for both linux/osx as well as various NodeJS versions. There isn't Windows support or 32-bit support, so we'd have to decide how much we care about those platforms.
This is a very detailed article which goes through the lengthy process that is involved to setup node-pre-gyp
.
I'm pretty skeptical that we'd be able to use node-pre-gyp
with our stuff since we're in Rust. Maybe there's a way to use it for only the uploading/post-install downloading, but we might end up doing the same amount of work if we just wrote a tool ourselves. More investigation would be needed before we can decide on a route there.
I've proven that the Neon module that is generated for these bindings works on both Node 8 and Node 10. However, there are no guarantees that future versions will work. So compiled versions would need to be build per Node version. This means that the cross-product of binaries would be [version of binding] x [version of node] x [platform (linux,osx,win)] x [supported architecture (x32/x64)]. Yeesh.
A good project to look at for using node-pre-gyp
is the Node Sqlite3 project.
Even if we end up getting this all working above, there's still an issue that the whole process only works if consumers of the binding build their code on the same platform where it runs. If we have the above setup correctly but a developer is building on their local Mac before pushing the results to some location where it'll be run in Linux, the whole system breaks down. We might end up needing a way for developers to specify their platform via some environment variable so that I could build on one architecture but have the results be built for a different architecture.
It should also be noted that Node 8/10 support WASM modules and I've proven that they work as expected. The WASM modules are also platform agnostic so they'd work to publish them to NPM. However the performance isn't great compared to these bindings so it depends on how much performance we need to wring out of this library.
This CI run failed. It only failed for our Mac builds.
dlopen(/Users/ironcore/actions-runner/_work/recrypt-node-binding/recrypt-node-binding/bin-package/index.node, 0x0001): tried: '/Users/ironcore/actions-runner/_work/recrypt-node-binding/recrypt-node-binding/bin-package/index.node' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/index.node' (no such file), '/usr/lib/index.node' (no such file)
In support of the common alpine-linux
deployment environment, build and upload to releases a musl linked version of these bindings.
Creating this should "just work" per neon if the build is run in a musl environment, causing recrypt-rs and its gridiron dependency to be built linked against musl.
Expose the new hash_256
method that was introduced here.
As a (Travis) build process
I want to have a script I can execute to build, package, publish binaries, and publish to npm
Probably will need to be worked on in conjunction with #9
Needs further research on how the Travis multi-arch stuff works. Does it kick off a job for each arch? How do we tie them back together? I started thinking about this when trying to write the "top level build/publish script" ticket. We need to publish a binary for each arch/node combo, and then publish to npm.
$ yarn add @ironcorelabs/recrypt-node-binding
yarn add v1.17.3
warning package.json: No license field
warning No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /home/foobaruser/project/pondjs/node_modules/@ironcorelabs/recrypt-node-binding: Command failed.
Exit code: 1
Command: node-pre-gyp install
Arguments:
Directory: /home/foobaruser/project/pondjs/node_modules/@ironcorelabs/recrypt-node-binding
Output:
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using [email protected]
node-pre-gyp info using [email protected] | linux | x64
node-pre-gyp WARN Using request for node-pre-gyp https download
node-pre-gyp info check checked for "/home/foobaruser/project/pondjs/node_modules/@ironcorelabs/recrypt-node-binding/bin-package/index.node" (not found)
node-pre-gyp http GET https://github.com/IronCoreLabs/recrypt-node-binding/releases/download/0.5.1/index-v0.5.1-node-v67-linux-x64.tar.gz
node-pre-gyp http 404 https://github.com/IronCoreLabs/recrypt-node-binding/releases/download/0.5.1/index-v0.5.1-node-v67-linux-x64.tar.gz
node-pre-gyp ERR! install error
node-pre-gyp ERR! stack Error: 404 status code downloading tarball https://github.com/IronCoreLabs/recrypt-node-binding/releases/download/0.5.1/index-v0.5.1-node-v67-linux-x64.tar.gz
node-pre-gyp ERR! stack at Request.<anonymous> (/home/foobaruser/project/pondjs/node_modules/node-pre-gyp/lib/install.js:149:27)
node-pre-gyp ERR! stack at Request.emit (events.js:198:15)
node-pre-gyp ERR! stack at Request.onRequestResponse (/home/foobaruser/project/pondjs/node_modules/request/request.js:1066:10)
node-pre-gyp ERR! stack at ClientRequest.emit (events.js:193:13)
node-pre-gyp ERR! stack at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:560:23)
node-pre-gyp ERR! stack at HTTPParser.parserOnHeadersComplete (_http_common.js:113:17)
node-pre-gyp ERR! stack at TLSSocket.socketOnData (_http_client.js:447:22)
node-pre-gyp ERR! stack at TLSSocket.emit (events.js:193:13)
node-pre-gyp ERR! stack at addChunk (_stream_readable.js:295:12)
node-pre-gyp ERR! stack at readableAddChunk (_stream_readable.js:276:11)
node-pre-gyp ERR! System Linux 5.0.0-23-generic
node-pre-gyp ERR! command "/usr/bin/node" "/home/foobaruser/project/pondjs/node_modules/@ironcorelabs/recrypt-node-binding/node_modules/.bin/node-pre-gyp" "install"
node-pre-gyp ERR! cwd /home/foobaruser/project/pondjs/node_modules/@ironcorelabs/recrypt-node-binding
node-pre-gyp ERR! node -v v11.15.0
node-pre-gyp ERR! node-pre-gyp -v v0.13.0
node-pre-gyp ERR! not ok
404 status code downloading tarball https://github.com/IronCoreLabs/recrypt-node-binding/releases/download/0.5.1/index-v0.5.1-node-v67-linux-x64.tar.gz
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.
neon
is coming up with a new build/distribution process, the experimental example project is here. Check it out and see if it will work to replace node-pre-gyp for us.
https://github.com/IronCoreLabs/recrypt-node-binding/runs/6751924442?check_suite_focus=true seems to indicate that we need to support arm64 on Mac:
dlopen(/Users/ironcore/actions-runner/_work/recrypt-node-binding/recrypt-node-binding/bin-package/index.node, 0x0001): tried: '/Users/ironcore/actions-runner/_work/recrypt-node-binding/recrypt-node-binding/bin-package/index.node' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/index.node' (no such file), '/usr/lib/index.node' (no such file)
HI, i'm trying to import this node-binding into my NextJs project to run, but it keeps tlling me this error: ./node_modules/@ironcorelabs/recrypt-node-binding/bin-package/index.node
Module parse failed: Unexpected character '�' (1:0)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
(Source code omitted for this binary file)
Please show me the proper way to import into framework like NextJs, thanks
As a release manager
I want to be able to have Travis build and release recrypt-node-binding for multiple architecture and multiple Node.js versions when a new release is created.
AC
yarn.lock update suggested: handlebars ~> 4.0.14.
yarn.lock update suggested: js-yaml ~> 3.13.1.
We should also update Recrypt to the latest and consume all breaking changes there as well. Also add Node12 support which will require us to rely on a fork of Neon.
Apparently neon binaries are supposed to be portable across Node versions. Investigate how that would affect our build and release process, we may be able to cut down on the number of artifacts and builds significantly.
As a (Travis) build process
I want a yarn target to call
so that a .tar.gz file is produced for the target architecture.
AC
node-pre-gyp
is used to produce this tar.gz so we can rely on it for naming and downloading at install timeyarn run package
unless there's a reason for it to be separate.index.node
file to minimize sizeCan be based off of the current node-pre-gyp
Apparently some people run non-LTS versions of NodeJS. We should support those versions so we don't get ugly error messages for them and now make us iterate our version support above version 8 in the consuming projects package.json
files.
Add a compile/publish step to build a binary for windows. We might need to move this repo over to GitHub Actions as part of this in order to add windows builds.
Figure out if we can use node-pre-gyp for the postinstall NPM step of pulling down the expected architecture binary from the expected location. We don't want node-pre-gyp to do anything with node-gyp, just the configuration/simplification of determining an architecture and pulling down the proper binary.
Now that there is a manually written index.js
as the entrypoint, we're duplicating a lot of work between index.js
and index.d.ts
that is error prone. Write an index.ts
that replaces both in the repo, and a build process that produces both of the original files for NPM packaging.
Now that recrypt-rs has Schnorr signing implemented, we need to consume those changes in the Node binding and expose a method to be able to do it. Also consume the security changes PR as well.
Similar to what we did for the wasm-binding code, we should implement changes to the node-bindings to map errors to extract out the internal error message from recrypt-rs and expose that to the caller. Currently the caller just gets a fairly unhelpful "neon error" style message.
Contains updates to new rand 0.6 and other dep updates. Consuming this will also fix the next release of the bindings to not be pulled from recrypt-rs master. (https://github.com/IronCoreLabs/recrypt-node-binding/blob/master/native/Cargo.toml#L13)
GitHub is reporting three vulnerabilities:
lodash critical CVE-2019-10744
mixin-deep high sev CVE-2019-10746
set-value high sev CVE-2019-10747
As a (Travis) build process
I want a way to upload a binary tar.gz to github releases
so that it can be downloaded at NPM install time.
AC:
./node_modules/.bin/node-pre-gyp install
A good starting point might be
node-pre-gyp-github. During inital assessment this package did not upload the full archive and only created an 100 byte (correctly named) file in github.
Also note that file can only be downloaded once the release is not marked DRAFT.
Hey, I'd love to start using this and the associated wasm libraries but we have recently upgraded to Node 21, which you don't support yet.
Are there any plans to support Node 21? If so, when?
Thanks in advance.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.