nersent / qusly

⚡ An FTP/FTPS/SFTP client with modern API.

Qusly-Core

Qusly-Core is a powerful multi-protocol library for file transfer. Created for Qusly.

Features

• Supports FTP, FTPS, SFTP
• API that allows to call methods asynchronously
• Informative transfer progress info - e.g. eta, speed in B/s
• Connection pool
• Ability to register custom protocols

Example

import { Client } from 'qusly-core';

async function main() {
  const client = new Client({ pool: 2 });

  await client.connect({
    protocol: 'ftp',
    host: 'www.example.com',
    user: 'root',
    password: 'password',
  });

  // It will handle all methods at once.
  const [documents, videos] = await Promise.all([
    client.list('/documents'),
    client.list('/videos'),
  ]);

  console.log(document, videos);

  await client.disconnect();
}

main();

Installation

$ npm install qusly-core

Documentation

Roadmap

Components

We use ssh2 and basic-ftp under the hood.

Task queue

Certain protocols such as ftp don't support handling many request at the same time. When app calls API many times from many places, handling manually these cases is very hard - you can't use await in the most cases.

What if you want to use many connections to speed up transfering files?

That's where we come in. This library supports it all thanks to the powerful task manager.

Connection pool

Allows you to:

• Use all connections for every method
• Use one connection to generic methods e.g. listing files and the rest for transfering files

qusly's Issues

Cannot read property 'map' of undefined calling the ls function using SFTP Protocol

when I call ls function using the SFTP protocol there is no check when the call get an error.

line 301 in file Client.js
if (this._isSFTP) { return new Promise((resolve) => { this._sftpClient.readdir(path, (err, files) => { resolve(utils_1.getResponseData(err, { files: files.map(file => file_1.formatFile(basic_ftp_1.parseList(file.longname)[0])) })); }); }); }

fix version
if (this._isSFTP) { return new Promise((resolve) => { this._sftpClient.readdir(path, (err, files) => { if(err){ resolve(utils_1.getResponseData(err)); }else{ resolve(utils_1.getResponseData(err, { files: files.map(file => file_1.formatFile(basic_ftp_1.parseList(file.longname)[0])) })); } }); }); }

otherwise, we have this error:

Uncaught TypeError: Cannot read property 'map' of undefined at _sftpClient.readdir (api/node_modules/qusly-core/build/models/client.js:306:42) at reread (api/node_modules/ssh2-streams/lib/sftp.js:1608:16) at SFTPStream._transform (api/node_modules/ssh2-streams/lib/sftp.js:417:17) at SFTPStream.Transform._read (_stream_transform.js:186:10) at SFTPStream._read (api/node_modules/ssh2-streams/lib/sftp.js:183:15) at SFTPStream.Transform._write (_stream_transform.js:174:12) at doWrite (_stream_writable.js:397:12) at writeOrBuffer (_stream_writable.js:383:5) at SFTPStream.Writable.write (_stream_writable.js:290:11) at Channel.ondata (_stream_readable.js:639:20) at addChunk (_stream_readable.js:263:12) at readableAddChunk (_stream_readable.js:250:11) at Channel.Readable.push (_stream_readable.js:208:10) at SSH2Stream.<anonymous> (api/node_modules/ssh2/lib/Channel.js:167:15) at parsePacket (api/node_modules/ssh2-streams/lib/ssh.js:3248:10) at SSH2Stream._transform (api/node_modules/ssh2-streams/lib/ssh.js:694:13) at SSH2Stream.Transform._read (_stream_transform.js:186:10) at SSH2Stream._read (api/node_modules/ssh2-streams/lib/ssh.js:253:15) at SSH2Stream.Transform._write (_stream_transform.js:174:12) at doWrite (_stream_writable.js:397:12) at writeOrBuffer (_stream_writable.js:383:5) at SSH2Stream.Writable.write (_stream_writable.js:290:11) at Socket.ondata (_stream_readable.js:639:20) at addChunk (_stream_readable.js:263:12) at readableAddChunk (_stream_readable.js:250:11) at Socket.Readable.push (_stream_readable.js:208:10) at TCP.onread (net.js:597:20)

Move @types/ssh2 to (regular) dependencies

The @types/ssh2 devDependency should be moved to the dependencies section. I'm using the module in a Typescript project and one shouldn't need to worry about installing the dependencies of a module, see https://stackoverflow.com/a/46011417.

Invalid date on ms-dos style listening

FTP server returns for example 08-11-19 02:46PM, which can't be parsed by JS Date. I plan to implement custom date parser for this case in the future.

2.0.0

v2.0.0 will be the next major update. Class Client will have changed functions response to pure value instead of object { success: boolean, error: Error, ... } . I'll remove TreeTraversal class.

readDir does not work for the specific folder (adding different path than ./ ) with FTP Protocol

I upgrade the library from 1.0.5 to 1.1.1, I discover the ls function not exist anymore and I have to use the readDir, it works perfectly with the SFTP but when I tried with the FTP, it doesn't work.

TypeError: Cannot read property 'end' of null

Triggered on client.connect when using incorrect credentials.

exists function error in windows

Hi ,
I tried uploading some files using qusly, but it failed, here is the error message,

I'm sure this folder exists locally, so this error confuses me.

Here is my SFTP configuration

Environment：

Windows10
Node 16.13.1

Tree traversal

Modules with "~/..." missing in Typescript

When building a project with Typescript I get the following errors:

node_modules/qusly-core/build/strategies/ftp.d.ts:5:81 - error TS2307: Cannot find module '~/interfaces' or its corresponding type declarations.

5 import { IFile, IFtpConfig, ITransferOptions, IFtpOptions, ITransferInfo } from '~/interfaces';
                                                                                  ~~~~~~~~~~~~~~

node_modules/qusly-core/build/strategies/sftp.d.ts:8:82 - error TS2307: Cannot find module '~/interfaces' or its corresponding type declarations.

8 import { IFile, ITransferOptions, IFtpConfig, ISFtpOptions, ITransferInfo } from '~/interfaces';
                                                                                   ~~~~~~~~~~~~~~

node_modules/qusly-core/build/strategies/strategy.d.ts:4:83 - error TS2307: Cannot find module '~/interfaces' or its corresponding type declarations.

4 import { IFile, ITransferOptions, ITransferProgressListener, ITransferInfo } from '~/interfaces';
                                                                                    ~~~~~~~~~~~~~~

node_modules/qusly-core/build/utils/function.d.ts:1:31 - error TS2307: Cannot find module '~/interfaces' or its corresponding type declarations.

1 import { ITaskResponse } from '~/interfaces';

I noticed that the ~ is mapped in the qusly-core tsconfig.json and it apparently works when running tsc on the qusly-core project to generate the npm package. However, the type definitions in the npm package still contains imports from ~/interfaces.

I guess that simply replacing ~ with relative imports (most of the imports are relative imports anyways) would solve the issue?

TransferClient error: you must provide config

Triggered when calling constructor.

Migration to 4.0.0

This is a guide for migrating to v4.0.0. This version introduces a lot of breaking changes since v2.6.0.

Documentation:

• v4.x.y
• v2.6.0

Breaking changes

Client

Client.abort()

• Now aborts all tasks.
• To abort a transfer use Client.abortTransfer().

Client.connect(config, options)

• Changed options - IOptions => IOptions | FtpOptions | ISftpOptions.

Client.connected

• Removed.

Client.createBlank()

• Removed.

Client.delete()

• Removed, because in some protocols, there is no way to check if path is a folder or a file.

`Client.download(remotePath, dest, options)

• Switched the order of dest and remotePath arguments.
• Changed options to startAt (number).

Client.isSftp

• Removed.

Client.mkdir()

• Changed to Client.createFolder.

Client.protocol

• Removed. You can access this by Client.config.protocol.

Client.readDir()

• Changed to Client.list.

Client.rimraf()

• Changed to Client.removeFolder.

Client.stat()

• Removed.

Client.touch()

• Changed to Client.createEmptyFile.

Client.unlink()

• Changed to Client.removeFile.

`Client.upload(remotePath, source, options)

• Switched the order of source and remotePath arguments.
• Removed options argument.

Event connected

• Changed to connect.

Event disconnected

• Changed to disconnect. Emitted everytime a connection is closed.

Event abort

• Changed to transfer-abort. Emitted everytime some transfers are aborted.

Event progress

• Changed to transfer-progress. See listener options.

ConcurrentClient

• Removed.
• You can set a pool of connections by providing options to Client.constructor.

Interface IConfig

• Because of the ability to register custom protocols, this interface consists only of generic options. If you want to use interface
  for ftp protocols, check this.

Logger

We plan to make an API for logging every request and response happening inside Client and ConcurrentClient. This would be very helpful at tracking down potential issues.

ConcurrentClient uploads twice

When using upload method with ConcurrentClient, it is uploaded twice and the result file is 2x bigger than the local one.

No matters if you use 1 or 3 parallel clients.

Sample test

const qusly = require('qusly-core');
async function test(){
    var client = new qusly.ConcurrentClient(1);

    await client.connect({
        host: "localhost",
        user: "me",
        // ...
        protocol: "ftp"
    });

    const localPath = "myLocalFile.txt";
    const remotePath = "myRemoteFile.txt";
    client.on("progress", (e,info) => {
        console.log("Percent:",e.percent);
    })
    await client.upload(
        localPath,
        remotePath
    )
    await client.disconnect();

}
test().then(r => console.log(r))

qusly-core version: 2.5.1

Inconsistency between clients upload method

Hi, and thanks for your work!
I was looking at the difference between the upload method in Client class and ConcurrentClient class:

Client.upload(path, source, [, options])

ConcurrentClient.upload(localPath, remotePath)

Arguments are inverted and it could be annoying if you want to switch between clients in a programmatic way.

Delete file/folder with one function

Currently there are 2 methods, first to delete a file and second to delete a folder and all of it's contents. I plan to make third method, which will take file path: string as argument and decide what use.

Concurrent file transfer

Invalid file date in SFTP listenings

In some cases, Client.readDir() returns files with invalid date when used with sftp protocol.

How to add different authentication handlers?

Hi, i can't figure how to add keyboard-interactive authentication handler (such as ssh2)
How can i do?

thanks so much

TransferClient: check if local folder exists

Check if local path exists, if not create folders recursively.

TSL certificates

Qusly-Core currently doesn't support them yet. I will need some help to implement it.

Blank file/folder

I want to make a method, which will create a blank file/folder with available name. For example there are folders: New folder, New folder (2) and it will create New folder (3).

Transfer manager

• splitting transfer
• new transfer event
• progress event

Change tree traversal names

I plan to change these names in 1.1.8.

• Tree class to TraversalClient
• ITreeItem to ITraversalItem
• ITreeOptions to ITraversalOptions

Double breaking lines after plain text file upload using FTP

After uploading a file in plain text, every line is ended with a new breaking line. This does not affect sftp.

Example

Original file

Lorem ipsum dolor sit amet consectetur adipisicing elit. Illum dignissimos
culpa enim esse nulla. Alias, libero sit. Recusandae quo, tempora, iusto
inventore veniam adipisci praesentium, iure et dolorem atque odit?

After upload

Lorem ipsum dolor sit amet consectetur adipisicing elit. Illum dignissimos

culpa enim esse nulla. Alias, libero sit. Recusandae quo, tempora, iusto

inventore veniam adipisci praesentium, iure et dolorem atque odit?

Write unit tests