License: Apache License 2.0
See examples and typedoc generated README
For more information on how Balena Etcher forks a priveleged process to write SD cards, see balena-io-modules/mountutils#78
![balena-renovate[bot] avatar](https://avatars.githubusercontent.com/u/56742327?v=4 "balena-renovate[bot]")
Hello.
I'm trying to install etcher-sdk into my Vue app bundled inside Electron.
Installation of package goes smoothly, but once electron-rebuild kicks in with installing prebuilt binaries and native dependencies, it comes with following errors:
• electron-builder version=21.2.0
• rebuilding native dependencies [email protected]/[email protected], @ronomon/[email protected], [email protected], [email protected], [email protected], [email protected], [email protected] platform=linux arch=x64
• install prebuilt binary name=ext2fs version=1.0.31 platform=linux arch=x64
• install prebuilt binary name=drivelist version=8.0.10 platform=linux arch=x64
• build native dependency from sources name=drivelist
version=8.0.10
platform=linux
arch=x64
reason=prebuild-install failed with error (run with env DEBUG=electron-builder to get more information)
error=prebuild-install info begin Prebuild-install version 5.3.3
prebuild-install WARN install prebuilt binaries enforced with --force!
prebuild-install WARN install prebuilt binaries may be out of date!
prebuild-install info looking for cached prebuild @ /home/unm4sk/.npm/_prebuilds/49bf21-drivelist-v8.0.10-electron-v76-linux-x64.tar.gz
prebuild-install info found cached prebuild
prebuild-install info unpacking @ /home/unm4sk/.npm/_prebuilds/49bf21-drivelist-v8.0.10-electron-v76-linux-x64.tar.gz
prebuild-install WARN install incorrect header check
• install prebuilt binary name=mountutils version=1.3.19 platform=linux arch=x64
• build native dependency from sources name=ext2fs
version=1.0.31
platform=linux
arch=x64
reason=prebuild-install failed with error (run with env DEBUG=electron-builder to get more information)
error=prebuild-install info begin Prebuild-install version 5.3.3
prebuild-install WARN install prebuilt binaries enforced with --force!
prebuild-install WARN install prebuilt binaries may be out of date!
prebuild-install info looking for cached prebuild @ /home/unm4sk/.npm/_prebuilds/dd792d-ext2fs-v1.0.31-electron-v76-linux-x64.tar.gz
prebuild-install http request GET https://github.com/balena-io/node-ext2fs/releases/download/v1.0.31/ext2fs-v1.0.31-electron-v76-linux-x64.tar.gz
prebuild-install http 404 https://github.com/balena-io/node-ext2fs/releases/download/v1.0.31/ext2fs-v1.0.31-electron-v76-linux-x64.tar.gz
prebuild-install WARN install No prebuilt binaries found (target=8.2.4 runtime=electron arch=x64 libc= platform=linux)
• build native dependency from sources name=mountutils
version=1.3.19
platform=linux
arch=x64
reason=prebuild-install failed with error (run with env DEBUG=electron-builder to get more information)
error=prebuild-install info begin Prebuild-install version 5.3.3
prebuild-install WARN install prebuilt binaries enforced with --force!
prebuild-install WARN install prebuilt binaries may be out of date!
prebuild-install info looking for cached prebuild @ /home/unm4sk/.npm/_prebuilds/a92ee8-mountutils-v1.3.19-electron-v76-linux-x64.tar.gz
prebuild-install http request GET https://github.com/resin-io-modules/mountutils/releases/download/v1.3.19/mountutils-v1.3.19-electron-v76-linux-x64.tar.gz
prebuild-install http 404 https://github.com/resin-io-modules/mountutils/releases/download/v1.3.19/mountutils-v1.3.19-electron-v76-linux-x64.tar.gz
prebuild-install WARN install No prebuilt binaries found (target=8.2.4 runtime=electron arch=x64 libc= platform=linux)
This results in missing dependencies on project.
Compiling of etcher itself works fine, but inside another app using etcher-sdk fails with installation.
How can I fix this?
Calculate progress & ETA for compressed images where we can't determine the uncompressed size, by basing them upon the source's size & how much of the source has be read.
See also: #9
Connects to: balena-io/etcher#2403
There are a bunch of comments around why things are done in a specific way that have been omitted from the Etcher SDK so far. We should "backport" them
npm install fails on with v8.4.0 (b5c8e98). Fails both with node v16.18 and v18.15. Definitely looks related to the unbzip2 update.
kbee@system76:~/dev/etcher-sdk/tmp-repo$ npm install
npm ERR! code 2
npm ERR! git dep preparation failed
npm ERR! command /home/kbee/.nvm/versions/node/v18.15.0/bin/node /home/kbee/.nvm/versions/node/v18.15.0/lib/node_modules/npm/bin/npm-cli.js install --force --cache=/home/kbee/.npm --prefer-offline=false --prefer-online=false --offline=false --no-progress --no-save --no-audit --include=dev --include=peer --include=optional --no-package-lock-only --no-dry-run
npm ERR! > [email protected] prepare
npm ERR! > browserify -s unbzip2Stream index.js | uglifyjs >> dist/unbzip2-stream.min.js
npm ERR! npm WARN using --force Recommended protections disabled.
npm ERR! npm WARN deprecated [email protected]: Use the built-in String.padStart function instead
npm ERR! npm WARN deprecated [email protected]: See https://github.com/lydell/source-map-url#deprecated
npm ERR! npm WARN deprecated [email protected]: Please see https://github.com/lydell/urix#deprecated
npm ERR! npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm ERR! npm WARN deprecated [email protected]: https://github.com/lydell/resolve-url#deprecated
npm ERR! npm WARN deprecated [email protected]: See https://github.com/lydell/source-map-resolve#deprecated
npm ERR! npm WARN deprecated [email protected]: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
npm ERR! npm WARN deprecated [email protected]: Deprecated due to CVE-2021-21366 resolved in 0.5.0
npm ERR! npm WARN deprecated [email protected]: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm ERR! npm WARN deprecated [email protected]: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm ERR! npm WARN deprecated [email protected]: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm ERR! sh: 1: cannot create dist/unbzip2-stream.min.js: Directory nonexistent
npm ERR! npm ERR! code 2
npm ERR! npm ERR! path /home/kbee/.npm/_cacache/tmp/git-cloneXUt5ok
npm ERR! npm ERR! command failed
npm ERR! npm ERR! command sh -c browserify -s unbzip2Stream index.js | uglifyjs >> dist/unbzip2-stream.min.js
npm ERR!
npm ERR! npm ERR! A complete log of this run can be found in:
npm ERR! npm ERR! /home/kbee/.npm/_logs/2023-03-09T12_43_46_526Z-debug-0.log
When regenerating the auto-generated docs, I see a lot of changes for the SDK, and some are valid, some are spurious but still an issue.
resin referred currently: example, https://github.com/resin-io-modules/etcher-sdk/blob/... When I regenerated it I got balena-io-modules, so maybe the git remote by which the project is checked out is relevant? (I have it checked out by balena-io-modules/etcher-sdk. Here the regeneration actually fixes stuffDefined in /home/alexis/dev/resin.io/etcher-sdk-2/node_modules/@types/node/index.d.ts:3800, Here on regeneration it will be still wrong, just differently wrongThis updated start.elf file has been provided by upstream raspberrypi/usbboot and fixes flashing custom raspberrypis that use an eMMC: balena-io-modules/node-raspberrypi-usbboot#41
Including this updated firmware will allow testbot, witch uses etcher-sdk, to provision this type of raspberrypis for automated testing.
Currently when writing chunks that are non-divisible by the block size, the chunk will be zero-padded to a multiple of the block size – this could potentially wipe data unintentionally when writing smaller parts on disk individually.
While the SDK currently does not facilitate doing this, it should still be implemented properly, by reading the block in question, copy the to-be-written data over and write it back, especially for future use.
We could use an aligned buffer pool here instead of allocating a new aligned buffer every time.
What do you think @mcraa @builder555 ?
lzma-native attempts to load wrong binding under Electron when npm-linked
Error: Cannot find module 'etcher-sdk/node_modules/lzma-native/binding-v3.0.8-node-v54-darwin-x64/lzma_native.node'
After trying to update to v7, I am getting the following error:
LINK : fatal error LNK1104: cannot open file 'C:\Users\Me\AppData\Roaming\npm-cache_cacache\tmp\git-clone-a3b4f5f6\deps\doc\liblzma.def'
Am I missing something?
See block-stream in Etcher/sdk/writer
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
debug, @types/debug)mocha, @types/mocha)yargs, @types/yargs)These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
.github/workflows/flowzone.yml
package.json
@balena/node-beaglebone-usbboot ^3.0.0@balena/udif ^1.1.2@ronomon/direct-io ^3.0.1aws4-axios ^2.4.9axios ^0.27.0balena-image-fs ^7.2.0blockmap ^4.0.3check-disk-space ^2.1.0cyclic-32 ^1.1.0debug ^3.1.0drivelist ^11.1.0file-disk ^8.0.1file-type ^16.5.4gzip-stream ^2.0.0lzma-native ^8.0.6mountutils ^1.3.20node-raspberrypi-usbboot 1.0.6outdent ^0.8.0partitioninfo ^6.0.2rwmutex ^1.0.0tslib ^2.0.0unbzip2-stream 4a54f56a25b58950f9e4277c56db2912d62242e7unzip-stream ^0.3.0xxhash-addon ^1.4.0yauzl ^2.9.2zip-part-stream ^2.0.0@balena/lint ^5.4.0@types/bluebird ^3.5.23@types/chai ^4.1.4@types/cli-spinner ^0.2.0@types/crc ^3.4.0@types/debug 0.0.31@types/file-type ^5.2.1@types/lodash ^4.14.108@types/mocha ^5.2.4@types/node ^18.11.18@types/progress ^2.0.1@types/sinon ^5.0.1@types/yargs ^11.0.0@types/yauzl ^2.9.0chai ^4.1.2cli-spinner ^0.2.8mocha ^8.0.1progress ^2.0.0rimraf ^2.6.2sinon ^6.1.3ts-node ^9.0.0typedoc ^0.17.8typedoc-plugin-markdown 2.4.2typescript ^4.6.3yargs ^16.2.0winusb-driver-generator ^2.0.0node >=16
.nvmrc
node 18
Are the scanner.exe and file-to-file.exe examples meant to be usable on windows? Scanner seems to report removable device information, but all drives are missing a devicePath element in the output. There's a device and raw element both of which are something like '\\.\PhysicalDevice2' or similar, but if file-to-file.exe image.iso \\.\PhysicalDevice2 is used a new file is created on c:\ instead of the iso being written to the removable media.
It seems that it should be possible to use some device path such as those listed here: https://www.silabs.com/community/interface/knowledge-base.entry.html/2013/11/21/windows_usb_devicep-aGxD to refer to the usb removable media, but that also seems to fail (and would require some way of looking up the device vid, pid, serial number, and guid which aren't in the scanner output either). Is it possible to get scanner to output a devicePath that can be used to write to with file-to-file example?
This happens when flashing a .zip with the add-block-stream branch
As we may want to remove node10 from CI pipeline as it's EOL version since 2021-04-30 according to: https://github.com/nodejs/Release I'd like to raise the question if the node 10 target is still needed.
etcher-sdk's build / install step doesn't clean build directory, potentially causing stale, incorrect requires
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
These updates are currently rate-limited. Click on a checkbox below to force their creation now.
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.
.github/workflows/flowzone.yml
package.json
@balena/node-beaglebone-usbboot ^3.0.0@balena/udif ^1.1.2@ronomon/direct-io ^3.0.1aws4-axios ^3.3.0axios ^1.6.0balena-image-fs ^7.2.0blockmap ^4.0.3check-disk-space ^3.4.0cyclic-32 ^1.1.0debug ^4.3.4drivelist ^11.2.0file-disk ^8.0.1file-type ^16.0.0glob ^10.3.10gzip-stream ^2.0.0lzma-native ^8.0.6minimatch ^9.0.3mountutils ^1.3.20node-raspberrypi-usbboot 1.0.7outdent ^0.8.0partitioninfo ^6.0.2rwmutex ^1.0.0tslib ^2.0.0unbzip2-stream 4a54f56a25b58950f9e4277c56db2912d62242e7unzip-stream ^0.3.1xxhash-addon ^2.0.1yauzl ^2.9.2zip-part-stream ^2.0.0@balena/lint ^7.2.1@types/bluebird ^3.5.23@types/chai ^4.1.4@types/cli-spinner ^0.2.0@types/crc ^3.4.0@types/debug 4.1.12@types/file-type ^10.9.1@types/lodash ^4.14.108@types/mocha ^10.0.3@types/node ^20.0.0@types/progress ^2.0.1@types/sinon ^17.0.0@types/yargs ^17.0.29@types/yauzl ^2.9.0chai ^4.1.2cli-spinner ^0.2.8mocha ^10.2.0progress ^2.0.0rimraf ^5.0.5sinon ^17.0.0ts-node ^10.9.1typedoc ^0.25.3typedoc-plugin-markdown ^3.16.0typescript ^5.2.2yargs ^17.7.2winusb-driver-generator ^2.0.0node >=18
.nvmrc
node 18
If libusb is not available on the operating system, etcher doesn't show any warning regarding this issue.
Etcher: 1.4.4
OS: 4.4.0-127-generic Ubuntu 16.04
Through the lack of emission of an open event (or another mechanism), through which a stream's _read() could hook to (or check), file system reads can occur before a file descriptor is obtained, causing an EBADF error.
Connects to: #9
In order for the balena CLI to support Node 14, etcher-sdk needs to support it too. I understand Node 14 support is blocked by the following ext2fs issue: balena-io-modules/node-ext2fs#76
I am creating this issue for greater visibility and understanding of the problem.
$ npm test
> [email protected] test /Users/Jonas/Work/etcher-sdk
> mocha -r ts-node/register tests/**/*.spec.ts
block-write-stream
win32
✓ should write the correct bytes (318ms)
✓ should write the correct bytes sparse (884ms)
linux
1) should write the correct bytes
2) should write the correct bytes sparse
bzip2 support
✓ bzip2 level 9 File (1149ms)
3) bzip2 level 9 BlockDevice
✓ bzip2 level 1 File (946ms)
4) bzip2 level 1 BlockDevice
directory
✓ should be rejected with an error
dmg support
✓ raw File (218ms)
5) raw BlockDevice
✓ adc File (208ms)
6) adc BlockDevice
✓ zlib File (194ms)
7) zlib BlockDevice
✓ bz2 File (238ms)
8) bz2 BlockDevice
✓ invalid dmg file
Scanner
detected devices should be a Set
✓ should emit an empty Set
given only system available drives
✓ should emit an empty Set
given linux
given available drives
✓ should emit the non removable drives
given windows
given available drives
✓ should emit the non removable drives
given a drive with a single drive letters
✓ should use the drive letter as the name
given a drive with multiple drive letters
✓ should join all the mountpoints in `name`
given an error when listing the drives
✓ should emit the error
gzip support
✓ gzip support File (919ms)
9) gzip support BlockDevice
img
✓ mbr File (832ms)
10) mbr BlockDevice
11) gpt
SourceDestination.imageExtensions
✓ should be an array
✓ should not be empty
✓ should contain only strings
✓ should not contain empty strings
✓ should not contain a leading period in any file type extension
iso support
✓ iso File (811ms)
12) iso BlockDevice
metadata zip
✓ given an archive with an invalid `manifest.json` (116ms)
✓ given an archive with a `manifest.json` File (1079ms)
13) given an archive with a `manifest.json` BlockDevice
✓ given an archive with a `logo.svg` File (874ms)
14) given an archive with a `logo.svg` BlockDevice
✓ given an archive with a bmap file File (1774ms)
15) given an archive with a bmap file BlockDevice
✓ given an archive with instructions File (953ms)
16) given an archive with instructions BlockDevice
utils
streamToBuffer
given a stream that emits data
✓ should yield the stream data
given a stream that throws an error
✓ should be rejected with the error
xz support
✓ xz support File (898ms)
17) xz support BlockDevice
zip support
✓ given an empty zip directory
✓ given a zip directory containing only misc files
✓ given a zip with an unsupported compression method
✓ given a zip directory containing only an image File (896ms)
18) given a zip directory containing only an image BlockDevice
✓ given a zip directory containing an image and other misc files File (858ms)
19) given a zip directory containing an image and other misc files BlockDevice
compression method
✓ deflate File (874ms)
20) deflate BlockDevice
39 passing (49s)
20 failing
1) block-write-stream
linux
should write the correct bytes:
Error: Unmount failed, invalid drive
at Error (native)
2) block-write-stream
linux
should write the correct bytes sparse:
Error: Unmount failed, invalid drive
at Error (native)
3) bzip2 support
bzip2 level 9 BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
4) bzip2 support
bzip2 level 1 BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
5) dmg support
raw BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
6) dmg support
adc BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
7) dmg support
zlib BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
8) dmg support
bz2 BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
9) gzip support
gzip support BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
10) img
mbr BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
11) img
gpt:
Error: Timeout of 20000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/Jonas/Work/etcher-sdk/tests/img.spec.ts)
12) iso support
iso BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
13) metadata zip
given an archive with a `manifest.json` BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
14) metadata zip
given an archive with a `logo.svg` BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
15) metadata zip
given an archive with a bmap file BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
16) metadata zip
given an archive with instructions BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
17) xz support
xz support BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
18) zip support
given a zip directory containing only an image BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
19) zip support
given a zip directory containing an image and other misc files BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
20) zip support
compression method
deflate BlockDevice:
Error: Unmount failed, invalid drive
at Error (native)
npm ERR! Test failed. See above for more details.
A lot of users are getting errors that the unmount operation failed:
balena-io/etcher#3542
balena-io/etcher#3730
balena-io/etcher#1370
...etc.
Even though the verification makes it to 100%.
We can see that even if a user has a task manager open, the device can be marked as busy:
Even though nothing has changed, the task manager just has the device open, so it is marked as "busy".
If we make it to 100% verification, the flashing has succeeded, and so there is no need to propagate a failure error to the user, because the user thinks that something has gone wrong with the flashing process.
We should only show the error if the verification has failed.
Ultimately we should try to open the device exclusively so that no other process can have the device open to cause this error, but this is a more difficult problem.
Error: Could not initialize libusb. Check that your system has a usb controller.
Hello!
I hope you are doing well!
We are a security research team. Our tool automatically detected a vulnerability in this repository. We want to disclose it responsibly. GitHub has a feature called Private vulnerability reporting, which enables security research to privately disclose a vulnerability. Unfortunately, it is not enabled for this repository.
Can you enable it, so that we can report it?
Thanks in advance!
PS: you can read about how to enable private vulnerability reporting here: https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository
Gzip's original size is stored as a uint32 – it can't encode sizes larger than 2^32 - 1, as elaborated on in balena-io/etcher#638
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.