Node module to sniff the start of a stream (non-destructively) to detect the file type and encoding when you don't have the luxury of being able to restart the stream again.
It does so by using buffer-peek-stream to get the first 16KB of the stream then send that to mmmagic (which uses libmagic). Before it's finished the peek stream will unshift the bytes it's received back onto the origin stream thereby making it appear as if the origin stream was new.
npm install stream-mmmagicconst magic = require('stream-mmmagic');
const input = fs.createReadStream('somefile.csv');
const [mime, output] = await magic.promise(input);
console.log('TYPE:', mime.type);
console.log('ENCODING:', mime.encoding);
output.pipe(process.stdout);
//- TYPE: text/plain
//- ENCODING: us-ascii
//- <the file content>var magic = require('stream-mmmagic');
var input = fs.createReadStream('somefile.csv');
magic(input, function (err, mime, output) {
if (err) throw err;
console.log('TYPE:', mime.type);
console.log('ENCODING:', mime.encoding);
// will print the *whole* file
output.pipe(process.stdout);
});
//- TYPE: text/plain
//- ENCODING: us-ascii
//- <the file content>A magic file is bundled with the mmmagic npm module but if you want to use your own then set the path to the file on
the magicFile option.
const magicFile = '/usr/share/magic';
magic(input, {magicFile}, callback);Use {splitMime: false} option to get back the original mime string instead of a split object.
const [mime] = magic.promise(input, {splitMime: false});
console.log(mime);
//- text/plain; charset=us-asciiAs the input stream starts to get data the first 16KB is buffered and sent to libmagic for analysis to get file type and
encoding. 1KB is more than enough for detecting file type with a standard magicFile but the reliabilty of getting the
correct encoding is increased the more bytes are buffered. The tradeoff is performance and memory use.
Set peekBytes to the number of bytes you want buffered and sent to libmagic. For best results do not set below 256
bytes.
// somefile.txt is a utf8 file where the first doublebyte char is after the first 1KB of the file
const input = fs.createReadStream('somefile.txt');
const [{encoding}, output] = magic.promise(input, {peekBytes: 1024});
console.log(encoding);
// not detected as utf8 because the first doublebyte char wasn't until later in the stream
//- us-ascii
const [{encoding}, output] = magic.promise(input, {peekBytes: 16384});
console.log(encoding);
// now we're peeking 16KB into the file libmagic gets that first doublebyte char and knows it's utf8
//- charset=utf8MIT
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent