m4nuc / async-busboy Goto Github PK

async-busyboy doesn't unsubscribe from custom onFile function on error.

Have issue with filePromises. This arg not passing to custom onFile callback

Hi i just tried out this tool. It's awesome. Is there a way or an example you can give for tracking the uploading progress? Is this something that can be done on the client side right away or do i have to implement some sort of custom logic on the server?

Hey everybody.

Could someone update busboy to version 1.3?

I was trying to debug and fix a bug described in #42. It took me a while to realize, it is already fixed in master branch, just not released to latest version (1.1.0).

Could you please release new version from master?

I am using async-busboy to read a .txt file uploaded from multipart form in postman. However, it can't read the file content so the buffer in the returned "files" object is always empty, and the length of the file is zero.

My code:
router.post( '/attachment',async (ctx, next) => {
    try {
        const {files, fields} = await asyncBusboy(ctx.req);
        ctx.code = 200;
        ctx.body = {
            code: 200,
            data: files,
        };
    } catch (err) {
        log.error(err);
        ctx.throw(500, 'Internal Server Error');
    }
  },
);

The returned file object:
[
    {
        "_readableState": {
            "objectMode": false,
            "highWaterMark": 65536,
            "buffer": {
                "head": null,
                "tail": null,
                "length": 0
            },
            "length": 0,
            "pipes": null,
            "pipesCount": 0,
            "flowing": null,
            "ended": false,
            "endEmitted": false,
            "reading": false,
             "sync": true,
            "needReadable": false,
            "emittedReadable": false,
            "readableListening": false,
            "resumeScheduled": false,
            "destroyed": false,
            "defaultEncoding": "utf8",
            "awaitDrain": 0,
            "readingMore": false,
            "decoder": null,
            "encoding": null
        },
        "readable": true,
        "domain": null,
        "_events": {},
        "_eventsCount": 1,
        "path": "/tmp/0d812ea775d47-test1.txt",
        "fd": null,
        "flags": "r",
        "mode": 438,
        "start": 0,
        "end": null,
        "autoClose": true,
        "pos": 0,
        "bytesRead": 0,
        "fieldname": "",
        "filename": "test1.txt",
        "encoding": "7bit",
        "transferEncoding": "7bit",
        "mime": "text/plain",
        "mimeType": "text/plain"
    }
    ]
}

I understand that file caching is inevitable in a promise-based form parser, since to be able to have an overview of the whole form in one go you also need to, well, read and process the whole body in one go. This is obvious if you stop and think about it for a second, but, being in a rush to just get stuff done, this has escaped my attention, until a slightly later "duh" moment. A quick note, warning people about the difference in nature between async-busboy and co-busboy / busboy, somewhere high up in the readme, could save some time for somebody else in a hurry.

Just in case, this matters, because if you're interested in doing any processing/spying on a file (e.g. checking its magic bytes or calculating a hash), piping it through your transforms while it's downloading is more performant than saving it on the disk and then reading from there.

I think it would be nicer to have files as an object of keys -> ReadStreams so it would be easier to validate the existence certain files

I have midllware for validate fields formData

const checkAndSendErrorValidateMiddleware = (schema, transformError) => async (ctx, next) => {
  const { fields, files } = await asyncBusboy(ctx.req)
   
  const filesFields = files.reduce((acc, fileReadStream) => ({ ...acc, [fileReadStream.fieldname]: fileReadStream }), {})
    
  const resultValidate = validate({ ...fields, ...filesFields }, schema)
  const errors = transformError(resultValidate.errors)
  if (resultValidate.errors.length !== 0) {
    ctx.body = JSON.stringify({
      errors,
      data: null,
    })
 }
 await next()
}

and controler midlleware

const controler = async (ctx, next) => {
  const { fields, files } = await asyncBusboy(ctx.req)

  const instanceModel = await db.findById(fields.id)
 // e.t.c
})

when I connect them in a chain, then in controler fields, files empty. Why?

route.post('/url', checkAndSendErrorValidateMiddleware(validateScheme), controler)

Quoting the doc: "When the consumer stream drained the request stream, files will be automatically removed, otherwise the host OS should take care of the cleaning process."

What would be the best practice for reading the files e.g. to write them in a data base. I'm currently using fs.readFileSync(files[0].path) and deleting the file with fs.unlink() function.

Edit: Extending the question, because it's probably related:
To get the file size one must use fs.stat(PATH) but that's only possible when you have the path already. But how to skip a file upload that exceeds a certain size?

See PR #16 for description.

hi, i has a question

for example:

{fields, files} = await asyncBusboy(ctx.req)

i can get the mime, path, filename and others information of files[0] but the file's size.

how to get size? is must to use the fs.stat(path) or read to buffer ?

apiroute.post("/upload",async function (ctx,next) {
    asyncBusboy(ctx.req).then(function (formData) {
        console.log(formData);
    })
});
console  ->  { fields: { modName: '23333', mod: 'undefined' }, files: [] }

this files is empty
use koa-router and async-busboy

whit demo

 var bd = new FormData();
                bd.append("modName","23333");
                bd.append("mod",sj.file);
                    $.ajax({
                        url: '/api/upload',
                        type:"post",
                        cache:false,
                        data:bd,
                        processData: false,
                        contentType:false,
                        success:function () {
                            console.log("OK");
                        }
                    });
            }

Hi

So I have a question I want to do an async/await pattern with the onFile custom handler.

I tried several ways and I dont' think I found one that works out.

I know this may be very simple, but I couldn't figure it out myself.

It seems that node 14 has some issue with this lib, the following code hangs:

const { files, fields } = await asyncBusboy(context.req, config);

node 13 is fine.
Thanks.

Would it be possible to implement the file.on('limit') event? I would really like to have an error thrown like shown below or exposing the listeners or having same options for callbacks, not sure what would be the best practice here...

function onFile( //...
//...
  file.on('limit', () => {
    const err = new Error('Reach file size limit');
    err.code = 'Request_file_size_limit';
    err.status = 413;
    throw err;
  })

const {files, fields} = await asyncBusboy(ctx.request);
It's show 'TypeError: request.on is not a function'.

hi,
is there any way to writestream to desired path in local ?

Send the files with csv or xlsx in formdata, But async busboy just keep on waiting for the response and ending up with 504 gateway timeout. But if I use .xls file extension , It successfully parsing that incoming file. Could not find what's the issue for other file formats like .csv or .xlsx.

Sometimes I found that asyncBusboy not responding because promise never resolved or rejected.
I traced inside asyncBusBoy source code and I found that sometimes request's 'close' event handler called
before busboy's 'finish' handler ( before onEnd called).
I modified source code like "request.on('close', onEnd)" instead of "request.on('close', cleanup)" and it works perfectly.
I'm afraid I'm doing wrong and make some side effects. Replacing handler to onEnd instead of cleanup is safe?

^^ Same thing as #42

I found an issue when trying to use array values without a defined key name. This can be easily replicated.

I am unsure if this is an issue in the wrapper or the original code, though the error occurs in the reconcile function, which is not present in the synchronous version; hence why I am sharing the issue here.

<input type="password" name="password[]"> <!-- Should be treated as password[0] -->
<input type="password" name="password[]"> <!-- Should be treated as password[1] -->

The actual error occurring:

TypeError: Cannot create property 'undefined' on string ''
    at reconcile (Path\To\Project\node_modules\async-busboy\index.js:208:24)
    at reconcile (Path\To\Project\node_modules\async-busboy\index.js:206:12)
    at reconcile (Path\To\Project\node_modules\async-busboy\index.js:206:12)
    at onField (Path\To\Project\node_modules\async-busboy\index.js:84:5)
    at emitMany (events.js:127:13)
    at Busboy.emit (events.js:204:7)
    at Busboy.emit (Path\To\Project\node_modules\busboy\lib\main.js:38:33)
    at UrlEncoded.end (Path\To\Project\node_modules\busboy\lib\types\urlencoded.js:205:14)
    at Busboy.emit (Path\To\Project\node_modules\busboy\lib\main.js:31:36)
    at finishMaybe (_stream_writable.js:510:14)
    at endWritable (_stream_writable.js:520:3)
    at Busboy.Writable.end (_stream_writable.js:485:5)
    at IncomingMessage.onend (_stream_readable.js:511:10)
    at Object.onceWrapper (events.js:293:19)
    at emitNone (events.js:86:13)
    at IncomingMessage.emit (events.js:188:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)

Putting the following works around the issue, however creates problems when dynamically creating elements based on user interaction:

<input type="password" name="password[0]">
<input type="password" name="password[1]">

Only sometimes when sending files via multipart/form-data and uploading them to S3 I am getting the following error:

Uncaught Error: ENOENT: no such file or directory, open '/var/folders/fh/w1h08fy12fngm0x499_3xr300000gn/T/filename.jpg'
      at Error (native)

When I log files variable, it always lists all files as ReadStreams with path set to /var/folders/fh/w1h08fy12fngm0x499_3xr300000gn/T/filename.jpg, however. Plus, when I do ls -la /var/folders/fh/w1h08fy12fngm0x499_3xr300000gn/T/ on a os.tmpDir() dir afterwards, desired files exist inside:

const { files, fields } = await asyncBusboy(ctx.req)

Wondering whether async-busboy causing the issue when creating ReadStreams or something with Promises I use for uploading files to S3…
The issue happens only sometimes. Most of the times, uploading is works successfully.

I test the lib in koa2, I found only picture file may be upload successfully, the .txt .execl files was unable to upload successfully! Is there options to set? Such as file type, upload path and so on.
BTW, I have no function named checkFiles and uploadFilesToS3, so I'd commend the lines for those codes.
Thank, please.

Hi,

we forked busboy and fixed two critical bugs in the package, which could cause the node-process to crash or to hang. We also improved the performance and added some new features. It does not have breaking changes so it is a drop-in-replacement for busboy. It has a code coverage of about 95%.

https://github.com/fastify/busboy/blob/master/CHANGELOG.md
https://github.com/fastify/busboy
https://www.npmjs.com/package/@fastify/busboy

for tracking reasons:
fastify/busboy#68

var {files, fields} = await parseFile(ctx.req, {autoFields: true});
fs.readFileSync(files[0].path); // <buffer > (a empty buffer, because writing is unfinished)

is that current?

Now I have to code like this:

return new Promise((resolve, reject) => {
  let index = 0;
  let interval = setInterval(() => {
    let data = fs.readFileSync(file.path);
    if (data) {
      resolve(parse(data));
      clearInterval(interval);
    } else if (index > 10) {
      reject(new RestError(400, 'FILE_LOAD_ERR', 'no file loaded'));
      clearInterval(interval);
    }
    index++;
  }, 100);
});

and it returned current buffer.

A common use case is to use array fields with no specified indexes, e.g.:

<input type="checkbox" name="fruits[]" value="apple">
<input type="checkbox" name="fruits[]" value="orange">
<input type="checkbox" name="fruits[]" value="banana">

However, upon receiving more than one such field, e.g. fruits[]=apple and fruits[]=banana, the whole Node process crashes with this stack trace:

RangeError: Maximum call stack size exceeded
    at Function.keys (<anonymous>)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:193:22)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)
    at reconcile ([redacted]/node_modules/async-busboy/index.js:202:12)

There are 2 problems here. I propose the following solutions:

1. There should be no process crash, in any circumstances. Upon any error, even internal, the exception should be thrown. It seams that the underlying busboy module poorly handles errors inside registered event handlers.
   Suggested solution: wrap any internal field and file processing function with a try..catch, which would reject the main promise.
2. This use case is very common and should be handled properly. And making your own parser can be difficult and sometimes even unsafe.
   I think, in general, TJ’s qs module could (and maybe even should) be used here for robust field parsing and aggregating. Off the top of my head:

const qs = require('qs')
const fieldList = [
	{name: 'fruits[]', value: 'apple'},
	{name: 'fruits[]', value: 'banana'},
]
const fields = qs.parse(fieldList
	.map(({name, value}) => encodeURIComponent(name) + '=' + encodeURIComponent(value))
	.join('&'))

Documentation is wrong with current file

(filePromises, fieldname, file, filename, encoding, mimetype)

filePromises is the first argument, not the file object

I find a bug.

A small file which maybe 9k can not be pushed into 'files'.I debug it and perhaps it returned before the stream was pushed into the files.

I get what's wrong.

the busboy closed before the writeStream open.

so if i code like this

function onFile(files, fieldname, file, filename, encoding, mimetype) {
  const tmpName = file.tmpName = new Date().getTime()  + fieldname  + filename;
  const saveTo = path.join(os.tmpDir(), path.basename(tmpName));
  const writeStream = file.pipe(fs.createWriteStream(saveTo));
    const readStream = fs.createReadStream(saveTo);
    readStream.fieldname = fieldname
    readStream.filename = filename
    readStream.transferEncoding = readStream.encoding = encoding
    readStream.mimeType = readStream.mime = mimetype;
    files.push(readStream);
}

then it work.

is it a bug or there is anyother reason why you code such as?

Hi,

After the server receive the formdata and have been successfully parsed by async-busboy you built. I need to save the file to some other directory. How can I make it?

The reason why I didn't use co-busboy is that it act like a middleware. But I expect the saving operations can be processed in a specific route.

Here's the co-busboy example:

var parse = require('co-busboy')

app.use(function* (next) {
  // the body isn't multipart, so busboy can't parse it
  if (!this.request.is('multipart/*')) return yield next

  var parts = parse(this)
  var part
  while (part = yield parts) {
    if (part.length) {
      // arrays are busboy fields
      console.log('key: ' + part[0])
      console.log('value: ' + part[1])
    } else {
      // otherwise, it's a stream
      part.pipe(fs.createWriteStream('some file.txt'))
    }
  }
  console.log('and we are done parsing the form!')
})

What I expect:

const router = require('koa-router)()
router.post('/api/upload', async (ctx, next) => {
  const { fields, files } = await asyncBusboy(ctx.req)
  files.map(file => {
    // saving file
  }
  ...
}

(By printing out the file variable in the map function above and the part variable in the while loop. I notice that file is kind of like a ReadStream object while part is a FileStream object. )

Sorry about asking a question that is not fully related to your repo. I've tried some other parser and methods but it seems like it still cannot work properly. Could you give me some advice about how to solve this problem?

Much appreciated