webpack / memory-fs Goto Github PK

flag it as deprecated plz

Hi! I'm really curious about your normalize() function.

At first glance it like it handles a lot of interesting cases, but I tested it on a bunch of different strings and it seemed to work exactly like path.normalize. What am I missing?

This has not been published to npm in a while. I'm interested in the fixes to exception handling for compatibility with mkdirp. Currently I've got a link to this github repo in my package.json, but I'd rather use a semver.

Unlike the real Node API, the async variants have their callbacks called synchronously. This is really odd, and can lead to bugs for people using this as a mock, especially if they're calling the functions asynchronously.

Could these callbacks be wrapped in a setImmediate, so they are a) actually called async in the first place and b) scheduled as macrotasks like in Node?

_

I try to use memfs to replace this library, and found some problem

import { fs } from 'memfs'
compiler.outputFileSystem = fs // TS2741: Property 'join' is missing in type 'IFs' but required in type 'OutputFileSystem'.

You are looking for memfs instead :) You're welcome.

The original Node implementation of rmdirSync throws an error if the directory is not empty. The implementation in memory-fs quietly deletes the directory an all its contents. rmdirSync effectively behaves like rmdirpSync should.

This is a problem when using memory-fs to mock the native file system and trusting that it behaves similarly.

const MemoryFileSystem = require('memory-fs');

const fs = new MemoryFileSystem();
fs.mkdirpSync('/a/b/c');
fs.rmdirSync('/a/b'); // This should fail. b is not empty.
console.log(fs.readdirSync('/a')); // Empty

Hello,
since the last commit of yesterday, I have problems deploying with the updated dependencies: when I try to do npm install, I get this error:

npm ERR! code ETARGET
npm ERR! notarget No matching version found for memory-fs@^0.5.0
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.
npm ERR! notarget
npm ERR! notarget It was specified as a dependency of 'enhanced-resolve'
npm ERR! notarget

TBH I'm not sure why I have this issue, since enhanced-resolve removed this dependency months ago; and TBH I have neither memory-fs nor enhanced-resolve among my dependencies, it is deeply nested.

npm seems to have the updated version of the package (0.5.0); at the moment I solved the issue manually installing the package ( npm install https://github.com/webpack/memory-fs/tarball/v0.5.0 )

I just got the following error from memory-fs:

Error: no such file or directory
    at MemoryFileSystem.readFileSync (/myproject/node_modules/memory-fs/lib/MemoryFileSystem.js:108:10)
    at MemoryFileSystem.readFile (/myproject/node_modules/memory-fs/lib/MemoryFileSystem.js:298:21)
    at devRoute (/myproject/backend/webpack.js:36:29)
    at Layer.handle [as handle_request] (/myproject/node_modules/express/lib/router/layer.js:95:5)
    at next (/myproject/node_modules/express/lib/router/route.js:131:13)
    at Route.dispatch (/myproject/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/myproject/node_modules/express/lib/router/layer.js:95:5)
    at /myproject/node_modules/express/lib/router/index.js:277:22
    at Function.process_params (/myproject/node_modules/express/lib/router/index.js:330:12)
    at next (/myproject/node_modules/express/lib/router/index.js:271:10)
    at Layer.handle [as handle_request] (/myproject/node_modules/express/lib/router/layer.js:91:12)
    at trim_prefix (/myproject/node_modules/express/lib/router/index.js:312:13)
    at /myproject/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (/myproject/node_modules/express/lib/router/index.js:330:12)
    at next (/myproject/node_modules/express/lib/router/index.js:271:10)
    at SendStream.error (/myproject/node_modules/serve-static/index.js:121:7)

As you can see, there's no mention of the path it was trying to use to find a file/directory. However, when I looked at the code it was clear that it intended to share that path:

if(!isDir(current[path[i]])) 
    throw new MemoryFileSystemError(errors.code.ENOENT, _path); // Note the ",  _path"

and indeed, if I added a console .log ...

if(!isDir(current[path[i]])) console.log(_path)

it logged the path successfully. So evidently there's something wrong with MemoryFileSystemError which is causing it to ignore or discard it's second argument.

When I run fs.readFile(somePath, { encoding: "utf8" }, (err, content) => {}), which (I think) works with the native Node fs module, it throws an error TypeError: Unknown encoding: [object Object]. It seems to work if I just pass "utf8", rather than an object.

I'm currently migrating a small project to use Babel v6, but I'm getting the titular error with this trace when attempting to run webpack-dev-server:

TypeError: Cannot read property 'replace' of undefined
at Tapable.join (C:\project\node_modules\memory-fs\lib\join.js:11:73)
at Tapable.<anonymous> (C:\project\node_modules\enhanced-resolve\lib\FileAppendPlugin.js:14:19)
at Tapable.applyPluginsParallelBailResult (C:\project\node_modules\tapable\lib\Tapable.js:139:14)
at Tapable.<anonymous> (C:\project\node_modules\enhanced-resolve\lib\Resolver.js:103:8)
at Tapable.Resolver.forEachBail (C:\project\node_modules\enhanced-resolve\lib\Resolver.js:196:3)
at Tapable.doResolve (C:\project\node_modules\enhanced-resolve\lib\Resolver.js:102:7)
at Tapable.resolve (C:\project\node_modules\enhanced-resolve\lib\Resolver.js:45:14)
at Tapable.resolve (C:\project\node_modules\enhanced-resolve\lib\UnsafeCachePlugin.js:23:14)
at C:\project\node_modules\webpack\lib\NormalModuleFactory.js:169:12
at C:\project\node_modules\async\lib\async.js:356:13

It appears the error is in join.js which begins with:

module.exports = function join(path, request) {
    if(request == "") return normalize(path);
    ...
}

Obviously, FileAppendPlugin.js is calling join with undefined as the second argument but undefined != "", so this escape only works if an empty string is passed, and execution continues if undefined is passed (and the rest of the function assumes a value). Is this function intentionally tossing its toys, or is this a simple oversight that could be handled with something like:

if (!request) return normalize(path);

?

Sure, I can move onto working out why FileAppendPlugin is passing undefined (assumedly erroneously), but this problem remains.

I'm trying to create an empty file. So I'm calling fs.writeFileSync('/foo.txt', '');. However, this throws an exception with the message "No content".

I don't see any reason why it shouldn't be allowed to pass an empty string as file content.

Is it possible to make webpack use an in memory file system rather than the normal file system? I need to run webpack completely in memory. all files are already in memory and I want to start a webpack process.

Also react to changes to these files to trigger a rebundle.

Just wondering if it is even possible? And if so, with this project?

This seems wrong. I am trying to make a set of unit tests work cross platform by sticking to path methods for building paths. When i use path.resolve('/') on windows i get C:\, and on *nix, unsurprisingly, i get /. However, if i test to see if root exists:

function clear(): void {
  console.log(`check ${root}`)
  if (fs.existsSync(root)) {
    console.log(`iterate over ${root}`)
    fs.readdirSync(root).forEach(dir => {
      fs.rmdirSync(path.join(root, dir))
    })
  }
}

It is true on windows, and false on *nix. It seems to me that the root path should always exist...

Everything is in the title, this library doesn't works well. It doesn't even follow any standard. Even the copyright in README.md is old. And the code hasn't been updated for almost 2 years now.

Hi, the release of this package has not updated for a long time, we have to install it by git dep for using recently commits for a while.
Also webpack4 has launched, why not bump a minor version for this nice lib at this moment?
Feel free to close if it bothered.

It should remove the whole things, but instead it throws MemoryFileSystemError(errors.code.EPERM, _path)

This can be a small fix to remove everything instead of throw an error.

tons of prs and open issues...should I just switch to rollup?

pathToArray() unnecessarily changes the drive letter case when not *nix path[0] = path[0].toUpperCase();.

This mutation causes a problem in tasks run from c:. For example, memory-fs-stream _read() will fail because root was "c:..." but they keys on this.fs.data had been changed by pathToArray to "C:"

I don't see a test asserting this behavior. I'm happy to submit a PR, but wanted to verify this behavior wasn't intentional/required.

Continuing discussions from here - https://twitter.com/TheLarkInn/status/855369900914855937

At my recent talk at React London I mentioned that we use MemoryFS as the input filesystem for webpack in our bundling service - https://www.youtube.com/watch?v=mkUK2SZ8moY . One issue we came across when making this system scale was that as soon as we swapped out the input filesystem, rebuild times went through the roof. After digging into the webpack core I discovered that the compiler was trying to use the last modified timestamp (mtime) in order to work out whether it needed to rebuild each file. Since MemoryFS does not support mtime, webpack is unable to use its module cache and ends up rebuilding every file on every compilation.

We ended up monkey-patching support for timestamps into the methods we needed. It's not pretty, but here is the code extracted from our project:

const _ = require('lodash')
const MemoryFS = require('memory-fs')

const mfs = new MemoryFS()
mfs.lastModified = {}

// memory-fs doens't support stats by default, so we wrap all relevant methods
// to make it work.
wrap('writeFileSync', writeFile)
wrap('unlinkSync', unlink)
wrap('rmdirSync', rmdir)
wrap('statSync', stat)
wrap('mkdir', mkdir)

function mkdir (fn, args) {
  // mfs doesn't support supplying the mode!
  if (typeof args[2] === 'function') {
    return fn.apply(mfs, [args[0], args[2]])
  } else {
    return fn.apply(mfs, args)
  }
}

function writeFile (fn, args) {
  const filePath = args[0]
  const result = fn.apply(mfs, args)
  mfs.lastModified[filePath] = new Date()
  return result
}

function unlink (fn, args) {
  const filePath = args[0]
  const result = fn.apply(mfs, args)
  delete mfs.lastModified[filePath]
  return result
}

function rmdir (fn, args) {
  const dir = args[0]
  const result = fn.apply(mfs, args)
  mfs.lastModified = _.reduce(mfs.lastModified, (memo, mtime, filePath) => {
    if (filePath.indexOf(dir) !== 0) {
      memo[filePath] = mtime
    }
    return memo
  }, {})
  return result
}

function stat (fn, args) {
  const filePath = args[0]
  const stats = fn.apply(mfs, args)
  stats.mtime = mfs.lastModified[filePath]
  return stats
}

function wrap (method, fn) {
  const oldFn = mfs[method]
  mfs[method] = () => fn(oldFn, arguments)
}

@sokra @TheLarkInn

0.3.0 started requiring errno, which doesn't work in strict mode:

rvagg/node-errno#13

I have a use case where I would like to be able to fall back to another filesystem if a file is not found in the virtual file system. To enable this functionality and not fork memory-fs, I propose creating a function to throw errors that by default would throw the existing MemoryFileSystemError but would allow enterprising consumers the ability to override the function to provide alternate behavior.
For example:

class MemoryFileSystem {
...
  throwError(fn, args, error) {
    throw new MemoryFileSystemError(...error);
  }
...
  statSync(_path) {
    ...
    else {
      return this.throwError('statSync', arguments, [errors.code.ENOENT, _path, "stat"]);
      // previously
      // throw new MemoryFileSystemError(errors.code.ENOENT, _path, "stat");
  }
...
}

Example Usage:

const MemoryFS = require("memory-fs");
const fs = new MemoryFS();
fs.throwError = function(fn, args, error) {
  // do custom things
};

I'm happy to contribute back but want to see what others think of the idea first.

We need a testcase for #21.

`function MemoryFileSystem(data) {
	this.data = data || {};
    this.data[""]=true;
    this.data["/"]={
        isMount:false,
        mounts:{}
    };
}
MemoryFileSystem.prototype.mount = function(fs,path){
    try{
        this.writeFileSync(path,fs.data);
    }catch(e){
        throw new Error("Cannot mount "+path+" error: "+e.message);
    }
    fs.data["/"].isMount=true;
    this.data["/"].mounts[path]=fs;
}
MemoryFileSystem.prototype.unmount = function(path){
    if(this.data["/"].mounts[path]!==undefined){
        var ff=this.data["/"].mounts[path];
        ff.data["/"].isMount=false;
        this.rmdirSync(path);
        delete this.data["/"].mounts[path];
    }else{
        throw new Error("Path not a mount point!");
    }
}`

resulting in succes:

===================createing both folders
{ '': true,
  '/': { isMount: false, mounts: {} },
  a: { '': true } }
{ '': true,
  '/': { isMount: false, mounts: {} },
  b: { '': true } }
===================creating a file on f2
{ '': true,
  '/': { isMount: false, mounts: {} },
  a: { '': true } }
{ '': true,
  '/': { isMount: false, mounts: {} },
  b:
   { '': true,
     'one.txt': <Buffer 68 65 6c 6c 6f 20 70 65 6f 70 6c 65> } }
===================mounting f2 to f1
{ '': true,
  '/': { isMount: false, mounts: { '/m1': [Object] } },
  a: { '': true },
  m1:
   { '': true,
     '/': { isMount: true, mounts: {} },
     b:
      { '': true,
        'one.txt': <Buffer 68 65 6c 6c 6f 20 70 65 6f 70 6c 65> } } }
{ '': true,
  '/': { isMount: true, mounts: {} },
  b:
   { '': true,
     'one.txt': <Buffer 68 65 6c 6c 6f 20 70 65 6f 70 6c 65> } }
===================deleting file on f2 from f1
{ '': true,
  '/': { isMount: false, mounts: { '/m1': [Object] } },
  a: { '': true },
  m1: { '': true, '/': { isMount: true, mounts: {} }, b: { '': true } } }
{ '': true, '/': { isMount: true, mounts: {} }, b: { '': true } }
===================unmounting f2 from f1
{ '': true,
  '/': { isMount: false, mounts: {} },
  a: { '': true } }
{ '': true,
  '/': { isMount: false, mounts: {} },
  b: { '': true } }

just replace with the coe iprovided, (edited from npm source)

I'm trying to use this package on windows but i keep getting error : throw new MemoryFileSystemError(errors.code.ENOENT, _path)

In orer to debug the issue, i tried this and am still getting the same error

var MemoryFileSystem = require("memory-fs")
var fs = new MemoryFileSystem()

console.log(__dirname)

console.log(fs.readdirSync(__dirname))

Ignoring tests saves ~14KB.

if we config webpack like this
config.output.filename(js/[name].js?${VERSION.commit}).end();,

webpack will cut off the query of the fileName, when write into the MemoryFileSystem instance.

const queryStringIdx = targetFile.indexOf("?"); if (queryStringIdx >= 0) { targetFile = targetFile.substr(0, queryStringIdx); }

but we will read the cache content with a query name, so that we can't read the correct file.

maybe we can cut off the query when read the fileContent from MemoryFileSystem instance

In pathToArray function, if the path looks like \\SERVER\folder\subfolder\abc.txt, it will throw MemoryFileSystemError(errors.code.EINVAL, path).

This is causing issues with Node.js >= 6.0.0 when bundling with Webpack on a UNC path.

Repro steps:

1. Mapped \\SERVER\folder to Z:
2. Setup a webpack.config.js that has an entry based on __dirname

Expected:
Webpack should bundle successfully

Actual:
Node.js >= 6.0.0: Webpack failed because pathToArray does not recognize UNC path
Node.js < 6.0.0: Bundle successfully

This is because under Node.js >= 6.0.0. Any JS modules on a mapped drive Z:, the __dirname will be relative to \\SERVER\folder\, instead of Z:\. And Webpack will fail because memory-fs could not recognize UNC path correctly.

There is currently no workaround as the new behavior on __dirname is consistent across all modules. The only way is to use process.cwd() instead but the solution is limited.

This is a blocking issue for Webpack bundler to run on Azure Web App with Node.js >= 6.0.0.

The correct behavior of pathToArray should be recognizing \\SERVER\folder and turn it into ['\\SERVER', 'folder'].

description

when I use webpack v4 in this way:

const serverCompiler = webpack(serverConfig)
const MFS = require('memory-fs')
const mfs = new MFS()
serverCompiler.outputFileSystem = mfs

and show the warning below:
DeprecationWarning: Tapable.plugin is deprecated. Use new API on .hooks instead

fs api: https://nodejs.org/api/fs.html#fs_fs_rename_oldpath_newpath_callback

I'm not sure if this issue is with webpack-dev-server or memory-fs, however when I serve from a file path which includes spaces on Windows things do not work.

My bundles generate 404 when served through webpack-dev-server. I tried to debug and it seemed like at some point the spaces were being converted to %20, and when readdirSync is called from here it is trying to do a check which fails because it is comparing the file path with spaces against the file path with %20.

I hope this is enough information to go on, but if not I will try to put together a minimal example reproducing the issue.

Trying to join paths of kind 'C:/your/path/here' with relative requests like './../../my/module/path' result in 'C:/your/path/here\my\module\path' instead of 'C:/your\my\module\path'.

This is due to converting slashes in the request, but not in the path when invoking join.

MemoryFileSystem.priotoype.meta loops through 0...n-1 entries in the split file path. Suppose, /path/to/existing/files exists but /path/to/existing/files/plus does not. Then, sending /path/to/existing/files/plus/extra causes the meta method to return null. This causes an issue in the conditional inside statSync(). When isDir() is called, it tests the type of the incoming object. But if that object is null, typeof evaluates to "object", which I believe is unintended.

I believe the fix for this is to return undefined inside MemoryFileSystem.priotoype.meta instead of null.

https://github.com/webpack/memory-fs/blob/master/lib/MemoryFileSystem.js#L39
I think it should be /^[A-Za-z]+/ instead of /^[A-Za-z]:/ over there.

Hi @sokra,
would it be possible to release a new minor version of this package with license file in it? Right now is not possible to use it due to missing license file.

Thank you a lot.

fs.join is not part of fs module and we need deprecate this and remove in next major release

Ref: webpack/webpack-dev-middleware#370