sharpart555 / nexline Goto Github PK

Hello again!

Thanks for fixing that last bug. I suppose this one is more of a feature request really. Consider the following test program:

#!/usr/bin/env node
"use strict";

import fs from 'fs';

import gunzip from 'gunzip-maybe';
import nexline from 'nexline';

(async () => {
    "use strict";
    
	let handle_reader = fs.createReadStream("/tmp/test.txt");
	let extractor = gunzip();
	handle_reader.pipe(extractor);
	
	let line_reader = nexline({
		input: extractor // Could also be an array of inputs
	});
	
	console.log(`First line: ${await line_reader.next()}`);
})();

I'm trying to read a number of different (potentially) gzipped files line-by-line, but it looks like nexline gets upset by duplex streams (gunzip-maybe instances show Duplexify { .... } in when console.log()ged). It generates an error message like this:

(node:26326) UnhandledPromiseRejectionWarning: Error: Invalid input. Input must be one of these: string, buffer, readable stream, file descriptor
    at nexline (/home/bryan-smithl/Documents/repos/PhD-Code/ingester/node_modules/nexline/nexline.js:43:11)
    at multiInputWrapper (/home/bryan-smithl/Documents/repos/PhD-Code/ingester/node_modules/nexline/multiInputWrapper.js:25:7)
    at file:///home/bryan-smithl/Documents/repos/PhD-Code/ingester/backends/RadarCaesar/test.mjs:16:20
    at file:///home/bryan-smithl/Documents/repos/PhD-Code/ingester/backends/RadarCaesar/test.mjs:21:3
    at ModuleJob.run (internal/modules/esm/module_job.js:110:37)
    at async Loader.import (internal/modules/esm/loader.js:141:24)
(node:26326) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:26326) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

System info: Node.js version: v13.9.0, npm version: 6.13.7, Ubuntu 18.04.4 LTS

For my next trick, I'm planning to share process.stdin of a master control process across multiple worker processes. From there, I'm want to read from said file descriptor line-by-line from the multiple worker processes.

You can probably see where this is going, but I'd like to know whether nexline has an internal buffer. If so, this could complicate matters considerably, as the buffer will be local to a single worker process as opposed to shared across all workers.

Many thanks in advance :-)

I'm reading large file (10 MB->1GB) and then processing each line with slow function, then at the end of file (last line), the nl.next() fire a timeout error ( can't catch this error):

UnhandledPromiseRejectionWarning: Error: Stream read timeout!
    at Timeout.setTimeout [as _onTimeout] (somedrive:\somewhere\node_modules\nexline\reader\streamReader.js:31:13)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)

while (true) {
        const line = await nl.next().catch((err) => { return null }); // -> error here and can't catch this error
        if (line === null) {
            break; // If all data is read, returns null
        }
        await slowfunction(line);
}

Single byte character is ok but multibyte character possibly get broken.

As the title says. Will nexline behave as expected with TCP sockets?

I've been using nexline for a while now, and it's so convenient! I wish standard Node.js had a n async readline implementation.

Unfortunately, I've run into a pretty deal-breaking bug. All of a sudden nexline has started silently exiting the entire Node.js process when it reaches the end of the stream instead of returning null. Consider the following test file:

apple
orange
banana
pear

....and the following Node.js script:

import nexline from 'nexline';

(async () => {
    "use strict";
	let reader = nexline({
		input: process.stdin
	});

	let items = [];
	while(true) {
		console.log("Getting next line");
		const line = await reader.next();
		console.log(`Got next line: ${line}`);
		if(line === null) { console.log("was null, breaking"); break; }
		
		items.push(line);
		console.log(`Read ${items.length} objects so far`);
	}
	console.log(`Done, read ${items.length} in total`);
})();

Watch what happens when I execute it:

$ cat /tmp/test.txt 
apple
orange
banana
pear
sbrl@box:~/path/to/PhD$ cat /tmp/test.txt | node ./test.mjs 
(node:10920) ExperimentalWarning: The ESM module loader is experimental.
Getting next line
Got next line: apple
Read 1 objects so far
Getting next line
Got next line: orange
Read 2 objects so far
Getting next line
Got next line: banana
Read 3 objects so far
Getting next line
Got next line: pear
Read 4 objects so far
Getting next line
sbrl@box:~/path/to/PhD$ echo $?
0

It correctly reads the 4 lines of input, but when it tried to read the 5th it dies instead of returning null. It doesn't even throw an error that can be caught - or indeed return a non-zero exit code.

If I force the Node.js process to stick around a bit longer with a sneaky setTimeout at the top of the file (below the import) like this:

setTimeout(() => console.log("the future is now"), 2000);

...it has no effect other than delaying Node.js' exit.

System details:

sbrl@box:~$ node --version
v13.8.0
sbrl@box:~$ npm --version
6.13.7
sbrl@box:~$ uname -a
Linux box 5.3.0-29-generic #31-Ubuntu SMP Fri Jan 17 17:27:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
sbrl@box:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 19.10
Release:	19.10
Codename:	eoan

Can anyone help me please? This is pretty annoying.

Edit: I've also tried Node v12.16.0, but it has no effect either and does not change anything.