qgustavor / mega Goto Github PK

View Code? Open in Web Editor NEW

148.0 7.0 38.0 1.62 MB

Unofficial JavaScript SDK for MEGA

Home Page: https://mega.js.org/

License: MIT License

JavaScript 100.00%

javascript typescript

mega's Introduction

MEGAJS

Unofficial JavaScript SDK for MEGA

API documentation and examples are available in the website: https://mega.js.org/

This is based on tonistiigi's mega library.
This is all unofficial, based on developer guide and site source.
Make sure you agree with MEGA's Terms of Service before using it.

Contributing

Whenever you can, try contributing using pull requests!

IMPORTANT! Before opening issues or discussions, look at the documentation, look at the examples, look for existent issues, look for existent discussions, then, if you still think that's a issue, follow the issue template!

How to contribute using pull requests:

Fork the project
Clone it
Run npm install
Switch to a feature branch
Change the library
- Fix issues
- Add features
- Refactor things
npm run lint-fix to fix common issues
Build the bundled versions using npm run build
Run at least Node tests using npm test node to test Node
Optionally run npm test deno to test Deno if you have it installed (CI will test Deno anyway)
Create a pull-request against next (not main)

Before creating a pull request, please, run tests. It will avoid a lot of common issues! You can run npm run lint-fix to fix common issues.

If you implement new features them implement tests for it too if possible. The hash at the end of test/helpers/test-runner.mjs may be updated if tests are updated in a way it change server state (like adding new files to tests).

Project history

This package started as a fork, with the following objectives:

Make the original package work in browsers again: even following the instructions from the original library it stopped working because some dependencies used __proto__, which is non-standard and isn't supported in many browsers. Also the updated versions of those libraries broke backyards compatibility;
Reduce dependencies and replace big dependencies with smaller ones, like crypto libraries, which usually are huge;
Rewrite code using the new JavaScript syntax, allowing to use Rollup, which can generate smaller bundles;
Make tests work again after the changes above;
Continue the original library development implementing new features and improving performance.

Request package was replaced with a shim based in browser-request and xhr-stream, which additional changes in order to make it work inside Service Workers. Crypto was replaced with secure-random.

As there were many changes there isn't any plan to merge those changes into the original library, unless the original author accept those massive changes. That's why I put "js" in the name, which is silly because both libraries use JavaScript. At least it's better than other ideas I had, like "mega2", "mega-es" and "modern-mega".

In 1.0 release request was replaced with fetch, Deno support was added (including tests), TypeScript types were added, Rollup was replaced with esbuild, streaming libraries were updated or replaced and some issues were fixed.

mega's People

Contributors

Stargazers

Watchers

mega's Issues

Implement folder sharing

Improve error handling

Currently megajs handles errors on uploading and download by killing the entire stream and emitting an error event. Restarting a download is simple because, beside requiring a new download URL (which could be fixed by #30), it's possible to download a file midstream. Restarting a upload, in the other hand, it's not possible.

As #13 got fixed it's now easier to handle those kind of errors by simply retrying to upload the failed chunk, instead of requiring the entire stream to be killed. Retries could use exponential backoff (although would be better checking MEGA source code to check what it uses). The same can be applied to the download code making error handling simpler for the library users.

But doing that raise some issues:

How to handle unrecoverable errors which may require the entire upload or download to be restarted?
Would be useful emitting warnings so library users can handle those?
How many times should the library retry by default?

Edit: I researched how other NPM libraries handle that: I could only find S3 upload libraries as S3 support chunked upload. s3-upload-stream allow resuming incomplete uploads using sessions. intimidate uses exponential back-off and retries up to three times by default.

It's possible to allowing getting upload interval variables as a "token" (uploadStream.getToken() returns {uploadURL, size, position, remainingBuffer}). If upload fails, those can be reused to resume the upload (.upload({resumeToken, ...}). It allows a more detailed error handling as who's using the library can handle specific errors instead of just letting it blindly retry until the upload succeed.

Implement thumbnail uploads

Thumbnails work similar to file uploads: request an upload URL then upload to it.

Consider also uploading a default thumbnail for all files uploaded, even non-media files, to reduce the amount of metadata MEGA servers receive, in other words, if all files receive thumbnails isn't possible to distinguish media files from non-media files. This thumbnail needs to be similar in size to other thumbnails, so it can't be discerned.

Attributes could not be decrypted with provided key.

Can not login to my account.
The account has a lot of contacts and shared folders.

    at Function.unpackAttributes (C:\Users\wiedy\Desktop\mega\node_modules\megajs\dist\main.node-cjs.js:1420:13)
    at MutableFile.decryptAttributes (C:\Users\wiedy\Desktop\mega\node_modules\megajs\dist\main.node-cjs.js:1097:15)
    at MutableFile.loadMetadata (C:\Users\wiedy\Desktop\mega\node_modules\megajs\dist\main.node-cjs.js:1089:12)
    at new MutableFile (C:\Users\wiedy\Desktop\mega\node_modules\megajs\dist\main.node-cjs.js:1487:12)
    at Storage._importFile (C:\Users\wiedy\Desktop\mega\node_modules\megajs\dist\main.node-cjs.js:2387:38)
    at C:\Users\wiedy\Desktop\mega\node_modules\megajs\dist\main.node-cjs.js:2326:39
    at Array.forEach (<anonymous>)
    at C:\Users\wiedy\Desktop\mega\node_modules\megajs\dist\main.node-cjs.js:2326:18
    at Request._callback (C:\Users\wiedy\Desktop\mega\node_modules\megajs\dist\main.node-cjs.js:988:7)
    at Request.self.callback (C:\Users\wiedy\Desktop\mega\node_modules\request\request.js:185:22)
    at Request.emit (events.js:311:20)
    at Request.EventEmitter.emit (domain.js:482:12)
    at Request.<anonymous> (C:\Users\wiedy\Desktop\mega\node_modules\request\request.js:1154:10)
    at Request.emit (events.js:311:20)
    at Request.EventEmitter.emit (domain.js:482:12)
    at Gunzip.<anonymous> (C:\Users\wiedy\Desktop\mega\node_modules\request\request.js:1076:12)```

Upload Error: socket hang up

Uploading a file using storage.upload throws an Error: socket hang up.

fs.createReadStream('./myFile').pipe(s.upload('myFile'));

Error: socket hang up
    at createHangUpError (_http_client.js:330:15)
    at Socket.socketOnEnd (_http_client.js:423:23)
    at Socket.emit (events.js:164:20)
    at endReadableNT (_stream_readable.js:1054:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

I've got the same issue on the original mega repo. I cannot figure it out.

Support thumbnail and preview upload for already uploaded files

Add .uploadAttribute(type, [buffer], [callback]) to file.

Like .upload support Buffers or Streams. type accepts strings or integers, where thumbnail is 0 and preview is 1. No other options are given, even size, as attributes are always buffered in order to be encrypted using CBC.

Adding attributes to already uploaded files use the following command (example for thumbnail, replace 0 with 1 to upload preview image):

[{"a":"pfa","n":"file handle","fa":"0*uploaded attribute identifier"}]

Make API module configurable

Allow setting proxy
Allow changing gateway
Allow passing custom request object (which can make testing easier)

Support attribute downloading

Currently only uploading attributes (thumbnail and preview images) are supported.

Fix Node build

Only browser version is building, Node versions aren't.

Babel classes prevent tree shaking/minification

Babel transpiles ES6 classes in a way Rollup (and probably other tools) can't do tree shaking. Trying to include only File from the ES version will also include Storage and MutableFile classes along with their dependencies (like the big RSA script).

Possible solutions:

Creating File-only browser builds;
Creating non-transpiled builds;
Finding a way to make Rollup ignore Babel classes;
Finding a way to make Babel compilation be compatible with tree shacking;
Rewrite code to not use classes;
Replace Babel/Rollup with Closure Compiler;

Note that Closure Compiler minification remove the unused classes, but Uglify2 doesn't.

Support more attributes

It's a dead simple new feature, so why not?

const LABELS = ['not labelled', 'red', 'orange', 'yellow', 'green', 'blue', 'purple', 'grey']
...
this.label = LABELS[at.lbl || 0]
this.favorited = !!at.fav

Replace request library

It is under consideration since last year but I don't have too much time to look into it.

Seems the best idea is switching request with fetch, using something like cross-fetch to handle platform differences instead of using rollup-plugin-replace for that.

This plugin will still be used to handle differences in crypto implementations until someone finds a better way to managing it. When working in some other issue seemed that the pure JS crypto implementation was faster than Node crypto, but maybe it was just a bug that happened while testing. Maybe its time to implement some benchmarks on crypto code... although that's for other issue.

Switching to fetch makes streaming harder, but, as the library already supports chunked downloading and uploading, it makes more sense dropping support for streaming. Also MEGA stopped supporting single stream uploading anyway.

It would be a breaking change as the currently supported way of changing user-agent, proxy and other connection related settings is replacing the request instance. How to fix it?

Allowing setting agent and user-agent in API:
- storage.api.agent = new https.Agent(...)
- file.api.userAgent = 'MyApplication/1.0 (+https://example.com)'
Directly in File and Storage constructors:
- new Storage({..., userAgent: 'MyApplication/1.0 (+https://example.com)'})
- new File({downloadId: '...', agent: new https.Agent(...)})
- File.fromURL('https://...', {agent: new https.Agent(...)})
Both?

Edit: seems setting directly in the constructors is easier because it removes the need to set autologin: false to change connection settings used by Storage.

Invalid or expired user session, please relogin

When looping over multiple files I am getting "Invalid or expired user session, please relogin" after about 6 files.

Implement file management

Rename files
Move files between folders
Delete files in folders (move to rubbish bin)
Delete files in rubbish bin (permanently)

Uploads not completing

Both this and the CLI version don't complete uploads for most files. No error, the process seems to be waiting for a complete event that never fires.
The only file I could upload was a small .js file.
I suppose MEGA did some changes to their API - does anyone here have an idea how to solve this?

Thanks in advance!

Fork roadmap

Support new MEGA accounts

Hi,
when I use my old Mega Accounts and login with this API, it works fine.

When I create a new Mega Accounts and try to login with this API, error coming:
EEXPIRED (-8): The upload target URL you are trying to access has expired. Please request a fresh one.

It seems that new accounts using another type of login?

Support chuncked upload

It's similar to chunked downloading, but only the first byte is specified in the URL (${uploadURL}/${startByte} instead of ${uploadURL}/${startByte}-${lastByte}). Content-length should be the chunk length, not the whole file length.

Error in folders

For some reason some folders are failing to load. Error stacks:

TypeError: Cannot read property 'children' of undefined
---------------------------------
    at https://directme.ga/sw.js:4:8949
          var folder = nodes.find(function (node) {
                            ^
TypeError: Cannot read property 'find' of undefined
    at ...\main.node-cjs.js:1230:29

More information at: https://rollbar.com/qgustavor/Direct-MEGA/items/20/

Edit: there are two issues:

MEGA is sending ~~corrupted data~~ gzipped data to the Node client (probably because the response is too large)
~~There are orphan files in the requested folder: filesMap[file.p] === undefined~~
Sorting folders before files don't work when there are nested folders.

Library not validating download URL

Sometimes MEGA API don't return a download URL, mostly when download limits are reached. The download function don't check this, causing the library download "undefined", then fail.

0.16.2 storage.upload(filename).on will throw an error!

0.16.2:

storage.upload(filename).on(...)

will throw an error: storage.upload(filename) will return undefined instead of a steamable object.

Support non-encrypted uploading

It will allow users to choose their own encryption schemes: as AES is quite slower in systems which don't support it natively other ciphers would have a better performance.

Also it will allow users to upload files without any encryption, which is good considering the current encryption scheme is pointless and a waste of energy resources for files meant to be public or shared with a large number of users.

The only issue is that only this library and clients based on it can download the uploaded files: other clients, including the official clients, will download a corrupted file. Note that we can make they don't show any errors by calculating the MAC code of the corrupted file, which can be considered an extra security feature.

Replace crypto libraries with native node crypto

It would improve performance as it will use internally OpenSSL, which is hardware accelerated (when processor supports that). This StackOverflow question showed it's possible to do that.

Would be good if there was a require('crypto') browser replacement which uses crypto.subtle (or vice-versa).

2016-12-06: Seems most of MEGA uses AES. We can use 'crypto' on node and replace it with crypto-browserify/browserify-aes in browser, or write a asynchronous wrapper for crypto operations, using WebCrypto in browser.

2016-12-07: the library uses AES for all functions, except login related operations, when RSA is used. The slow pure JavaScript RSA can stay, but AES can be replaced with node Crypto ~~and WebCrypto~~, which have better performance.

Because WebCrypto don't implement AES-ECB it needs to be implemented using JavaScript, so it's better to just use the current libraries for browser build and write a replacement for Node, so it will have better performance. Also it will allow code keep synchronous.

Uploading crashes the application

buffer.js:627
    return _copy(this, target, targetStart, sourceStart, sourceEnd);
           ^

TypeError: argument should be a Buffer
    at Buffer.copy (buffer.js:627:12)
    at Stream.source.on.data (/var/www/removed/node_modules/megajs/dist/main.node-cjs.js:1837:12)
    at Stream.emit (events.js:185:15)
    at Stream.reemit (/var/www/removed/node_modules/duplexer/index.js:70:25)
    at Stream.emit (events.js:180:13)
    at Stream.write (/var/www/removed/node_modules/megajs/dist/main.node-cjs.js:346:10)
    at Stream.stream.write (/var/www/removed/node_modules/through/index.js:26:11)
    at Stream.ondata (internal/streams/legacy.js:15:31)
    at Stream.emit (events.js:180:13)
    at Stream.<anonymous> (/var/www/removed/node_modules/megajs/dist/main.node-cjs.js:53:20)

It don't stops downloading after emitting an error

If a download fails for some reason many 'error' events get emitted without stopping the download.
As a workaround is possible to create variable which stops getChunk from running if some error happens.

File selection in folder via URL isn't working

Tested via megajs-cli when trying to download a sub-folder. Note that it can be a bug in megajs-cli, but I don't think it is because it uses the node returned by loadAttributes when downloading.

Validate key lengths

Users using arbitrary keys make the code try to read a key which is smaller (or larger) than the expected one, causing "random" bugs or RangeError: Trying to access beyond buffer length errors.

To avoid that key lengths need to be checked. File keys are 256 bit (192 bits for the encryption key and 64 bits for the MAC), 43 bytes when encoded with Base64-URL. Folder sharing keys are 128 bit, 22 bytes when encoded with Base64-URL. There are other keys too.

Seems the best place to put this validation is on the AES constructor, so it would check every AES key used. The downside of this is that if there people using shorter keys intentionally because those may be zero padded then their code will stop working.

So what to do? Validate key lengths? Pad keys? Validate string keys but pad buffer keys?

Support login when downloading

Do you have plans to support 'premium' accounts when downloading?

Mega's daily transfer limits in some regions is quite low, making using it for any kind of retrievable storage very unappealing without a premium account, or completely impossible if you need to download a file larger than a few GB.

For uploads I'm using your library, however for downloads I'm having to call megatools which supports logged in downloads, but unfortunately doesn't support parallel chunk downloading like your library, making it very slow when you get an overloaded file server.

Cheers.

Allow start and end options in .download()

Similar to Node's fs.createReadStream and WebTorrent's file.createReadStream.

Of course, using a start value different from 0 will disable MAC verification.

Implement tests for parts which do API calls

Currently parts of the library that do API calls aren't being tested. Those parts need to be tested in order to avoid bugs like #21 and #22. But in there's a problem: tests shouldn't do API calls, otherwise we will have issues with network connectivity and rate limiting.

The basis of Storage testing are implemented using nock here but those tests are being skipped as they're failing. Those need to be fixed.

Note that using nock recording options should be used with care to avoid contributors credentials and metadata to being unintentionally leaked (even if no passwords are sent to servers).

If nock isn't the right tool for the job, as using it would be too complicated or it doesn't all provide features needed to simulate well the MEGA API servers, someone can suggest other tool?

Validate arguments passed to File constructor

From MEGA's source code downloadId, key and loadedFile arguments need to match the following regular expression: /^[\w-]+$/. Also when a folder is requested the key must be a 22 character string (when base64 encoded).

Doing this validation will fix issues caused by users passing wrong URLs to the library and tools based on it, like links without https://mega.nz/#!!some-random-key and https://mega.nz/#!some-id!%20some-random-key.

Implement shared folders

megadf?

Is there any way to check free space of mega account?

Shared files cannot be acessed without a key

Currently folders can be accessed without keys, allowing library users to access folder metadata without the decryption key. But shared files cannot be acessed because a bug which makes the library try to decrypt attributes without a key. In order to fix this bug it just needs to check if a key is defined before decrypting attributes.

Cache download URL

Calling .loadAttributes() on files loads the download URL from the API but it's not stored so a second call get's required then .download() is called. Caching would help in case .download() is called many times, like to download multiple parts from a file (but not the entire file).

The only catch I see on doing that is handling when the download URL expires. From my experience seems it expires automatically when the end of the file is downloaded and, probably, when it's not acessed after some time.

We need to handle expiration (checking the HTTP response codes and then requesting a new download URL if needed) and test how MEGA behaves to avoid an additional request in situations where we know it already expired.

Fail to upload to MEGA

When I try to upload files to MEGA, the error is indicated as below

CODE:

private uploadFile(filePath: string, parentFolder: MutableFile): Promise<Boolean> {
    const fileName = path.basename(filePath);
    return new Promise((resolve, reject) => {
      const readStream = fs.createReadStream(filePath);
      readStream.on('error', error => reject(error));
      readStream.on('end', () => {
        resolve(true);
      });
      readStream.pipe(parentFolder.upload({
        name: fileName,
        size: fs.statSync(filePath).size
      }));
    })
  }

ERROR:
c:\Users\zx261\Documents\projects\project\node_modules\megajs\dist\main.node-cjs.js:384

key.copy(newKey);

TypeError: Cannot read property 'copy' of undefined

at unmergeKeyMac (c:\Users\zx261\Documents\projects\project\node_modules\megajs\dist\main.node-cjs.js:384:7)

at getCipher (c:\Users\zx261\Documents\projects\project\node_modules\megajs\dist\main.node-cjs.js:308:18)

at checkCallbacks (c:\Users\zx261\Documents\projects\project\node_modules\megajs\dist\main.node-cjs.js:1582:7)

at streamToCb (c:\Users\zx261\Documents\projects\project\node_modules\megajs\dist\main.node-cjs.js:1789:11)

at Request.<anonymous> (c:\Users\zx261\Documents\projects\project\node_modules\megajs\dist\main.node-cjs.js:26:7)

at Request.emit (events.js:187:15)

at IncomingMessage.<anonymous> (c:\Users\zx261\Documents\projects\project\node_modules\request\request.js:1083:12)

at Object.onceWrapper (events.js:273:13)

at IncomingMessage.emit (events.js:187:15)

at endReadableNT (_stream_readable.js:1094:12)

Add option to retry on errors

The download function from file.js don't handle errors. Failed requests should be retried, with the number of retries and time between retries configurable.

This module seems to be a drop-in solution: retry-request

Disable chuncking when maxConnections is 1

Using a single connection will avoid loading extra HTTP headers and can reduce memory usage.

Direct link

Can you add a way to get the direct url for the files when creating a link for a MutableFile?

Error: MAC verification failed

I believe there is something wrong here. Probably 25% of the time I get this error when downloading. The original fork doesn't have this issue, I believe because it uses it's own cipher implementation for MAC instead of cryptos aes-128-ecb.

Error: MAC verification failed
at Stream.end (../node_modules/megajs/dist/main.node-cjs.js:504:33)

Names of directories are null

Hello
Each directory (true) name is null, except Cloud Drive, Rubbish bin and Inbox;
This feature were supported by tonistiigi's mega

Add support to file importing

Separate decryption from authentication

Currently the decryption function also do authentication. It's good because it makes using the library simpler. In the other hand it's not possible to do authentication checks on partial downloads, only in fully downloaded files.

So the proposal is separating internally decryption from authentication, keep the current functions doing that they already do and export an authentication function. Clients using partial downloads as a continue download mechanism (like megajs-cli) can verify the download after it ends using that function. They also can offer the option to check the MAC of some already downloaded file, allowing users to check if a file was already downloaded.

Allow Storage saving and loading

It will allow caching Storage for better performance.

Fix uploads on shared folders

As reported here uploading files to shared folders is bugged.

I found this issue while fixing #44 and tried to fix it by using the folder handler as share handler when it returns the user id instead of the share handler here. It was a quick fix and I think that's not what MEGA webclient does. After doing that tests that uploaded files to shared folders passed, but uploading files to folders on shared folders don't because when doing that it will get the wrong share handler.

There should be documentation on MutableFile upload events and callback

    fetch({
      url,
      gzip: true
    }).on('error', e => {
      ...
    }).on('response', res => {
      ...
    }).pipe(
      megaStream
        .upload(filename)
        .on('complete', () => {
          ...
        })
        .on('error', e => {
          ...
        })
    );

From the current documentation there is only 'complete' event https://github.com/qgustavor/mega/wiki/MutableFile available (and I assume error is also available) which doesn't seem to be the case - some files are not uploaded successfully but they do not trigger either complete or error function...

I cant import module

Module not found: Error: Can't resolve 'megajs'
`
import { File } from 'megajs';
ngOnInit() {

    const file = File.fromURL('https://mega.nz/#!wdFElA4D!5dDMSOtGHWKVzTY6Eh2WO6SKk4oheLBpX6kntqJZ7Dc');

    file.loadAttributes((error, file) => {
        console.log(file.name) // file name
        console.log(file.size) // file size in bytes

        file.download((err, data) => {
            if (err) throw err
            console.log(data.toString()) // file contents
        })
    })

    
}

Cannot login with certain email/password

Whenever I try to login I get the following error:
ENOENT (-9): Object (typically, node or user) not found. Wrong password?

It only occurs when I want to login with a specific account. I have no such issue with other accounts.

I checked the uh code and it's completely different then what the main website uses.
If you give me an email or something I can send you the details.

Problem to upload File to specific Folder

First I create folder (mkdir) in return I gate MutableFile = NEWFOLDER

fs.createReadStream(filePath + '\' + fileName).pipe(NEWFOLDER.upload(fileName));

file was uploaded to root not to NEWFOLDER

How close session ?

How can I logout and close session. I tried function storage.close() and storage.api.close() but it isn't working. In mega panel -> settings -> history session I see still active session.