dherman / defense-of-dot-js Goto Github PK

'import' should be preferred over 'require' for newer modules.

If the package has a CLI that uses ES6 imports, how should we declare that the CLI is ES6 module ?

"bin.module": {} or "module.bin": {}

or, should the "modules.root" take care of that internally, how should that work?

Update:

Also, for node --module cli.js, one can simply specify - #!/usr/bin/env node --module and execute that as ./cli.js.

But if you run node cli.js it would be script mode instead of module mode and would throw.

If require loads CJS and import loads ESM don't things "just work"?

The only thing I can think of off the top of my head is if a library updates to ESM and you were previously loading it using require you need to start loading using import. But that's just a major semver breaking change like any other major semver breaking change. You go look at the docs, see what changed, update your code, move on.

Are there other reasons that this solution was rejected?

The last line of agnostic-usage:

Because this proposal retains the .js suffix, applications that require a module that ends in .js, a standard practice, will continue to work even if the package in question updates to use standard modules and does not publish a poly-package.

Should clarify that "in versions of Node which support standard modules" a poly-package may not be required (though maybe a better example should be used).

In the what about section there's an nvm example. It's not clear to me how nvm loads js files. I use it to switch node versions. Maybe a better example could be used.

Realizing that it's possible to monkeypatch require('module')._load to polyfill the behavior specified here in Node, I figured someone might have released a polyfill already, but a cursory search hasn't turned up anything. Does anyone know of existing Node polyfills for this proposal, or should I work on one?

The what-about-transitional-packages blurb references:

This allows for truly gradual upgrades of large codebases. Keep in mind, since requires can be dynamic (and test frameworks quite often do make use of this feature), it may not always be possible to mechanically upgrade a single file in a large codebase without breaking other parts of the application.

Alludes to the poly-packages bit

The transpiler would rewrite any imports from "./lib/" in module.js to "./" in the transpiled CommonJS modules. Since standard modules use a declarative syntax for imports, it is possible to do this transformation soundly and reliably.

But it isn't super clear. A link back to that section would be helpful. I also feel like this is an understood and established gotcha in the age of babel plugins.

As part of the exercise, we ended up with a handful of possibilities around package.json. The three top alternatives that were considered:

1. shim
2. modules.root
3. modules.resolver

We decided to push for option 2 from above, knowing that we could pivot if needed.

I think it is worth to compile the other two alternatives, for historical reasons. This issue is a recollection of option 3 thoughts.

disclaimer: this is @caridy's recollection of memories from the discussions with @dherman and @wycats; might be missing some important pieces.

Rationale

Today, node only support extending the loading mechanism at the process level, applying AOP on require.extensions to support new extensions, and other goodies. This is the mechanism used by babel-register, and you can see how it works here.

The problem with this mechanism, and the babel-register approach, is that it affects all modules required by your program, with no distinction.

Note: part of our analysis was to look into existing NPM pkgs, to see how many of them have babel-register as a dependency, to corroborate our hypothesis that since this is a process-level mechanism, packages should not attempt to patch the loading mechanism of the process.

Note: Because of the work that we have done on the module loader spec, we will love to have a custom resolver mechanism in node that can be configured per package.

Questions

• Can a package control its resolution process?
• Can a package dictates how to evaluate modules depending on the runtime conditions?
• How can we align node resolving mechanism with the loader spec?

Note: From the forward looking point of view, once we get the loader in browsers, it will be easy to control almost every aspect of the fetching, parsing, and instantiation process for every module to be evaluated, including segmentation per packages.

Proposal

Any package (or a folder with a package.json) could choose its own resolver, which is used by new versions of node to determine the path and type of the module in question.

modules.resolver directive

Add a new directive called modules.resolver in package.json, e.g.:

{
     ...
     "modules.resolver": "./path/to/resolver.js",
     ...
}

This directive can also be a reference from another package declared as a dependency:

{
     ...
     "modules.resolver": "node-awesome-resolver/lib/standard",
     ...
}

As a result, once the package is resolved by node for the first time, and package.json cached in memory, the module denoted by modules.resolver will also be evaluated and cache.

Node Resolution Algo

When requiring or importing a module, node will determine what package (a folder) contains the module in question, and whether or not that package contains a custom resolver. If a resolver is available, node will call the default export function from the resolver, by passing the requested path, and some meta information about the package.

Example 1: ES Standard Package

An example of a very dummy resolver:

export default function (requested, pkg) {
    return {
       type: 'standard',
       path: pathlib.relative(requested, pkg.root),
    }
};

As a result, any requested module from this package will be parsed as ES Standard Module, and will be resolved from the root of the package.

Example 2: Poly-Package

export default function (requested, pkg) {
    return {
       type: 'standard',
       path: pathlib.relative(requested, pkg.root + '/src'),
    }
};

As a result, any requested module from this package will be parsed as ES Standard Module from the src/ folder in new versions of node that will understand the modules.resolver directive, otherwise it will fallback to the normal resolution for CJS modules relative to the package root.

Example 2: .mjs vs .js

export default function (requested, pkg) {
    const ext = pathlib.extname(required);
    const file = pathlib.relative(requested, pkg.root);
    let t = 'cjs';
    if (ext === '.mjs') {
         t = 'standard';
    } else if (fslib.existsSync(requested + '.mjs')) {
         t = 'standard';
         file += '.mjs';
    }
    return {
         type: t,
         path: file,
    };
};

As a result, for any requested module from this package, if the extension is explicitly .mjs, or a file exists with .mjs extension, it be parsed as ES Standard Module, otherwise it will fallback to cjs.

Features

• Any package can define its own resolver without affecting the node process
• Resolvers can be shared via NPM
• Node can provide one default resolver
• Let the winner to emerge from the community
• It is a one-line to be added into package.json
• Poly-packages
• Apps and Modules requiring files from arbitrary folders can be added.
• Flexible mechanism to add more features in the future (e.g.: transpilation)
• WHATWG Loader resolver hook can tap into node's resolver, for easy integration.
• Testing compat-mode for poly-packages is possible.

Resolver API

Some preliminary work around the API of the resolver is specified here using TypeScript:

interface Package {
  root: string;
  json: PackageJSON
}

interface PackageJSON {
  // you know ... the object representing package.json
}

interface ResolvedModule {
  type: 'standard' | 'cjs';
  path: string; // absolute path
}

interface Resolver {
  resolve(requested: string, fromPackage: PackageJSON): ResolvedModule;
}

Would it be reasonable to presume a default directory for modules.root? All of my packages are currently structured and published in a format nearly identical to the "poly-packages" format described here, but none declare modules.root. However, in all cases this value would simply be src. Perhaps that would be enough?

In agnostic-usage

If lodash publishes a poly-package that also contains "module": "main.js", new versions of Node.js will prefer the standard module, while older versions of Node.js (such as the 4.x LTS series), will see the "main": "index.js" entry in the package.json and use that.

The "module": "main.js" should be a "main": "index.js".

I asked on Twitter, but you might have missed it:

"npm-module" directive would be package.json-compatible and standardizeable as e.g. "use module". Thoughts?

Have you considered identifying modules by a directive at the start of their body? It can work now, is compatible with everything here, and as an approach can be standardized in ECMAScript.

As the poly-packages section stats

The transpiler would rewrite any imports from "./lib/" in module.js to "./" in the transpiled CommonJS modules. Since standard modules use a declarative syntax for imports, it is possible to do this transformation soundly and reliably.

This issue to a placeholder for someone to tackle a babel plugin to handle this.

I'm new here (both at this repo and with ES modules in general), but I was reading through the blog posts and I think I would also be unhappy with the .mjs solution. Mainly because of syntax highlighting, but also because it would be impossible to serve a single file to the server and the browser, since browsers don't know .mjs (some utility scripts could be useful on both front and back ends). I would much prefer to keep .js.

I like the idea of using metadata in a package file to indicate modules, but I was wondering…

module.json would be for ESM what package.json is for CJS. In order to be a non-breaking change, Node could learn to look for this new file, and if both files are present, module.json would take precedence over package.json. Years into the future, when CJS is dead, module.json would be the new standard. Any thoughts on this?

https://gist.github.com/sokra/27b24881210b56bbaff7

Edit: fix a couple small bugs.

I was thinking of an idea that might speed up the package.json approach, making it much quicker to check, which should keep the loading time minimal. The package.json will still need to be located, but that can be done in a similar search as what's done for node_modules now.

• Make the package.json fields relevant for this (e.g. modules, modules.root) queryable in a cache trie, so it can be quickly tested, almost constant time in practice.
• If no package.json was found in the above search, then the package.json should be found, and the file's parent directory added to the cache.

I've also prepared this gist to show what I mean better. One of those is a very high-level, unoptimized version, and the other is a more low-level, optimized equivalent that avoids allocation, although neither is tested.

With that gist, you would basically do this:

// In lib/module.js

const check = NativeModule.require("internal/check-path.js");
const path = NativeModule.require("path");
const hasOwn = Object.prototype.hasOwnProperty;

// ...

function loadPackageDirs(pkg, requestPath) {
  if (hasOwn.call(pkg, "modules")) {
    for (let i = 0; i < pkg.modules.length; i++) {
      check.addPath(path.resolve(pkg.modules[i]));
    }
  }

  if (hasOwn.call(pkg, "modules.root")) {
    check.addPath(path.resolve(pkg["modules.root"]));
  }

  if (hasOwn.call(pkg, "module")) {
    const dir = hasOwn.call(pkg, "main") ? pkg.module : requestPath;
    check.addPath(path.resolve(dir));
  }
}

function readPackage(requestPath) {
  // normal loading, declares `pkg`
  loadPackageDirs(pkg);
  return pkg;
}

function isModule(file) {
  // Check this first, since usually, the package has already been loaded. It's a
  // cheap check, anyways.
  if (check.hasPath(file)) {
    return true;
  }

  let last = path.dirname(file)
  let current;

  // The dirname of the root is the root.
  while (current !== last) {
    if (!hasOwn.call(packageCache, current)) {
      if (fs.statSync(path.join(current, "package.json")).isDirectory()) {
        readPackage(current);
        break;
      }
    }

    current = last;
    last = path.dirname(current);
  }

  // This is only true if a package.json has been loaded
  if (check.hasPath(file)) {
    return true;
  }

  return path.basename(file).startsWith("module.");
}

I really liked that you stand up to the TC "decision" (hopefully) since I am also not quite happy with the current state.

That being said given that I call the current proposal 4d I would like to add an few concerns I have with 4d and would like to offer a 4f (jumping over e). Maybe it could be useful?!

Concerns

• It adds three important properties (modules, module and modules.root) to package.json which have to be learned.
• The difference between module and main is not clear by looking at the code. (It is easy to mistake module for main when skimming the package)
• When looking at the files in a folder it is not easy to distinguish between CommonJS and ES6 files since the rules might be complex depending on the modules property definition.
• For a package to be supporting both CommonJS and ES6 it is required to pack both variants into the same .tar.gz. Which means that when downloading the package that supports both it will be twice as big.

Proposal 4f

Based on those concerns I thought if there might be a way to incorporate that into your work and well this is what I have come up with:

One new property to package.json

In order to figure out if .js files in a package are CommonJS or ES6 we add a new property "syntax": "es6" to the package.json. All files in this package will automatically be treated as ES6 while "syntax": "commonjs" would specify that the files are treated as CommonJS.

Backwards compatibilty through syntax variants

Old Node.js versions do not support ES6. Loading a package that contains es6 would thusly fail. In order to support this case we add support for syntax variants into npm! A new version of npm (say npm@4) supports a new "publish-time-only" property of the package.json: "variants": {"commonjs": "./common_js"}.

Every key in this property points to a folder. On publish npm stores two variants of the package in the package storage: One with all folders except the "./commonjs" folder and one where the "./commonjs" folder content is treated as root.

This way we have two packages stored in the npm registry. The old packages would all be treated as commonjs.

Old npm versions would automatically install the CommonJS variant of the package. The new version of npm would by-default install the es6 variant of the package a new --syntax=CommonJS flag would allow to request the CommonJS variant of it in case npm@4 runs on a old version of node.

(Of course npm could simply assume --syntax=CommonJS if npm runs in an old version of node. Yet I still think a flag is good-to-have)

Npm could even go as far and install the latest fitting CommonJS version of a package in case --syntax=CommonJS.

Transistion support through Babel

Packages that want to port their code from CommonJS to babel but have a hard time to are hard to up-date all at once could for-the-intermediate-time (until all modules are ported) simply use Babel (and "syntax": "commonjs"). This would add a processing penalty and more devDependencies to this case but it would work.

Benefits of this approach

• (like in 4e) deep linking require statements (require("lodash/array.js) would still work.
• (like in 4e) we only deal with the .js file ending.
• syntax is easier to understand and differentiate from the other properties in the package.json.
• Since npm only downloads one variant the downloaded amount would not grow.
• The penalty on transistionary packages would motivate package maintainers to transist completly.
• Variants of a package could open the door to frontend specific packages.
• When someone chooses to learn about supporting older versions of Node.js she has no chance of missing out complex configuration that could end up bloating the registry.
• Backwards compatibility could be added to ES6 package with a comparably simple babel script.

One last concern

With the simplicity of this comes the obvious lack of the possibility that we can't properly mix CommonJS and ES6 modules. The reason as to why this would be required is because ES6 doesn't support all the features of CommonJS (namely: dynamic imports). I see this as task of TC39 to improve the ES6 specification to support dynamic imports.

I would be very happy to hear your comments.

I have been thinking about this subject heavily as I have recently pulled my big personal project out of NPM.

In the current JS module ecosystem there are two primary problems:

1. module specification
2. open distribution

module specification problem

The ES2015 module definition should be considered the ratified standard. This convention is in the specification with intention for adoption all environments where JS executes whether that be browser, Node, or other environments.

Common.js and Require.js are defacto standards that exist and are popularized in the absence of a formal standard. A formal standard is now present, and in coming months will become ubiquitous across the browser landscape. This satisfies both qualities that allowed the defacto approaches to achieve popularity, which suggests defacto approaches will be relegated to a historic status whose support is necessary for backwards compatibility.

The ideal approach for a resolution is to focus development and support upon the new standard and provide an abstraction layer in Node for dealing with synchronous require. An abstraction layer may very well kill the recent performance wins that came with Node v6.0.0, but it is still the most advisable approach for forwards compatibility. At some future time if legacy require should be killed simply remove the abstraction.

The cost benefit of the abstraction layer approach is that all of the risks and costs associated exist primarily in the present and not in the future, unlike a dual support approach. Just get this right the first time and then maintain it only as support for ES2015 module support in Node changes.

open distribution problem

Currently the defacto standard for distribution of JavaScript modules is NPM. The problems are that NPM is not completely open and it is limited to JavaScript. The web is, as a platform, completely open between the IETF and W3C and that platform is not limited to JavaScript.

My solution to this problem is to propose a URI scheme, module which can point to a module repository. Since this approach is based upon URI scheme it would allow universal uniqueness without conflict or assistance. Privacy is free. For private modules move the module repository behind your organization firewall. Since it would be based upon a URI scheme it would also be:

• OS independent
• Technology independent
• Platform independent
• Language independent (both human and programming)
• Application independent
• Law independent (liabilities would be vested upon infringing artifacts and not the module system)

This is the first time I have disclosed the idea. I leave for Warrant Officer Candidate School next week where I will be without phone or internet and was waiting to work on this after graduation.

Makes me wonder whether Node 7 (or 8, whichever introduces these changes) should ship strictly with support for module syntax .js files and nothing else. Just ship our ideal future state and ignore the past.

The community can still monkey-patch global.require() at runtime to bring back support for script syntax (assuming there's sufficient plumbing for this), and can experiment with whatever package.json or extension conventions they think will fly. So backwards compatibility is an "exercise for the reader".

This is probably a bad idea, of course, just be looking at our previous io.js fork not to mention the Python 2 to 3 problem. But it seems like the current .mjs decision, which I'm personally okay with, has a "decision made by someone else, ruining my day" problem with the broader community. /shrug

Cross posted here:

• nodejs/node-eps#3 (comment)

One potential problem with the described system for poly-packages is that they will have to point their transpilation output to the source of the project. This makes it slightly more difficult to gitignore and clean up built files, since they will now have to be specified directly. When the build output is inside a folder, then the entire folder can be cleaned and gitignored. With this proposal other js files that are in the root (like index.js, test.js, etc) will have to be handled. Maybe not the biggest issue.

OK, I know there are huge subtleties, but why not just assume the developer knows best and if an import is used, with a .js extension or some other, then just presume it's an es6 module?

Just treat any file using "import" as a module with the transitive closure of imports derived from the file. "import" means it's a module.

I'm sorry if this is naive, but I really don't want "Determining if source is an ES Module" to spiral down a never ending debate.

This is a placeholder to create a proof of concept for package.json parsing and resolution.

The defined behavior of using module.js as a stand-in for a module field in package.json is problematic. Consider the following case:

• A package exists where there is no main or module field defined in package.json
• This package does not use standard module syntax
• This package has two files in the root directory:
  • index.js
  • module.js
• A require or import is run on this module

According to your resolution rules, as I understand them, Node would attempt to parse this module as a standard ES module. However, if this module is indeed not coded with the standard ES module syntax, this require will fail.

That is backwards-incompatible behavior. I suggest that module.js is only interpreted as a module field in package.json if:

• main field is undefined
• module field in undefined
• No index.js file exists in the root directory of the package