jspm / generator Goto Github PK

View Code? Open in Web Editor NEW

157.0 157.0 19.0 976 KB

JSPM Import Map Generator

License: Apache License 2.0

JavaScript 33.03% TypeScript 64.14% HTML 2.32% Shell 0.50%

generator's Introduction

JSPM CLI

JSPM CLI is the package management of the JSPM project, supporting import map package management.

Getting Started | Documentation | FAQ

Resolution against node_modules for local development workflows.
Versioned, locked dependency management against the local package.json.
Tracing and installing the full dependency tree of an application.
Complete NPM-like module resolution that supports conditional environments and package entry points.
Support for a wide range of CDNs, such as jspm.io, jsDeliver, esm.sh and unpkg.
Import map extraction/injection into HTML files, with module preloading and integrity attributes.

See the documentation and getting started guide on jspm.org.

Contributing

Build and test workflows use Chomp.

License

Apache-2.0

generator's People

Contributors

Stargazers

Watchers

Forkers

demoorjasper jayakrishnanamburu zhoukekestar mesteery fusionstrings jimmywarting lugoues therakeshpurohit isabella232 ponomarevlad fairhopeweb aslemammad bubblyworld abbijamal pvh dpikt standardgalactic yowainwright haikyuu

generator's Issues

Additional method signature produces incorrect d.ts

generator/src/generator.ts

Line 36 in 82ac2f4

async install (target: string): Promise<void>;

^ the extra signature for install (of just string param) is unnecessary as the union type string | Install already covers it.
But, it also leads to a incorrect signature in the generated d.ts...

install(target: string): Promise<void>;

Modify import map after loading @jspm/generator

I'm successfully using @jspm/generator to generate my import map. Then adding that map to the document head and executing modules. This however only works with es-module-shims and the module-shim/importmap-shim script types. If I don't use the shim I get An import map is added after module script load was triggered.

Is there a way to use @jspm/generator in the browser without initializing a module so I can avoid the An import map is added after module script load was triggered. error?

defaultProvider customization

It would be useful to have deeper defaultProvider customization - in particular varying the default provider based on the parent URL and the registry being looked up.

If we extend the defaultProvider option to support a double map of URL and registry keys, this could enable these flexible features in a relatively straightforward way:

defaultProvider: {
  './localapp/': 'nodemodules',
  'https://cdn.com/': {
    npm: 'jspm',
    github: 'github'
  }
}

Where import('./localapp/app.js') containing in turn import('pkg') would search for pkg in the local node_modules, while import('pkg') on https://cdn.com/ would use the JSPM CDN npm sources.

More informative subpath error

When getting the No "..." subpath defined in package error, it would be useful to note:

What subpaths are defined so that the user can more easily correct.
A URL / link to how to create an override to fix the issue.

This is important to be able to be guided to the solution when hitting this roadblock.

Support for passing an existing import map as the base for the generator.

The use-case is when you already have an import map but you want to add extra dependencies to it.

`whitespace` option ignored

I think the htmlGenerate whitespace option is supposed to "pretty print" the output when set to true. It appears setting to true or false has no effect, and the output is always minified / has no newlines:

import { Generator } from '@jspm/generator';

const generator = new Generator({
  mapUrl: import.meta.url,
  env: ['browser', 'module'],
});

const outHtml = await generator.htmlGenerate(`
  <!doctype html>
  <script type="module">import 'react'</script>
`, { esModuleShims: true, whitespace: true });

console.log(outHtml);

Output:

  <!doctype html>
  <!-- Generated by @jspm/generator - https://github.com/jspm/generator --><script async src="https://ga.jspm.io/npm:[email protected]/dist/es-module-shims.js" crossorigin="anonymous"></script><script type="importmap">{  "imports": {    "react": "https://ga.jspm.io/npm:[email protected]/index.js"  },  "scopes": {    "https://ga.jspm.io/": {      "object-assign": "https://ga.jspm.io/npm:[email protected]/index.js"    }  }}</script><script type="module">import 'react'</script>

Version:

  "name": "@jspm/generator",
  "version": "1.0.0-beta.28",

feat: option for nodemodules to create absolute urls

I am using

import { Generator } from '@jspm/generator';

const generator = new Generator({
  defaultProvider: 'nodemodules',
});

// Install a new package into the import map
await generator.install('lit');

console.log(JSON.stringify(generator.getMap(), null, 2));

to generate an import map that I am using on multiple files

index.html
page-a/index.html
page-b/index.html
...

Expected output

{
  "imports": {
    "lit": "/node_modules/lit/index.js"
  },
  "scopes": {
    "/node_modules/": {
      "@lit/reactive-element": "/node_modules/@lit/reactive-element/development/reactive-element.js",
      "lit-element/lit-element.js": "/node_modules/lit-element/development/lit-element.js",
      "lit-html": "/node_modules/lit-html/development/lit-html.js"
    }
  }
}

Actual output

{
  "imports": {
    "lit": "./node_modules/lit/index.js"
  },
  "scopes": {
    "./node_modules/": {
      "@lit/reactive-element": "./node_modules/@lit/reactive-element/development/reactive-element.js",
      "lit-element/lit-element.js": "./node_modules/lit-element/development/lit-element.js",
      "lit-html": "./node_modules/lit-html/development/lit-html.js"
    }
  }
}

relative URLs will only work on the root index.html as soon as we are in a subfolder it fails.
For example in page-a/index.html it will then search for page-a/node_modules/lit/index.js which does not exist.

Suggestion

I assume we would need some sort of flag for that? 🤔
or should it be absolute by default and add a flag to be relative? 🤔

How about

const generator = new Generator({
  absoluteNodeModulesUrls: true, // potentially defaults to true and to get relative you put false here
  defaultProvider: 'nodemodules',
});

PS: I also tried with

const generator = new Generator({
  mapUrl: '/',
  defaultProvider: 'nodemodules',
});

but that gave an error Invalid URL: /

The HTML generator crashes when the input HTML contains an empty importmap

The following input will crash the HTML generator. This happens for all HTML files with an empty importmap:

<!DOCTYPE html>

<script type="importmap"></script>

conditional lazy imported packages are preloaded

https://jspm.dev/stream-consumers loads "node:buffer" and "/npm:fetch-blob@3" even doe they are not necessary...

Catch double "/" in any resolver functions

These are causing identity issues currently.

Checkout API

Currently import maps generated by JSPM generator are always made comprehensively against the JSPM CDN.

There could be benefit in having a checkout API that enables a way to bring CDN workflows back to the local file system.

Consider an import map has already been generated for the generator, then we might want to perform the following steps:

const fsMap = generator.checkout({
  // if any package has only one import with no deps,
  // don't check out the whole folder and instead rename
  // the module to just be the name of the package itself
  singleFileCheckouts: true,
  
  // if a package only has one version, skip the version in the package name
  singleVersionNaming: true,
  
  // if a package only comes from one registry, skip the registry version name
  singleRegistryNaming: true,
  
  // required option to actually do any work
  checkoutUrls: {
    'https://ga.jspm.io/': './third_party/jspm/',
    'https://unpkg.com/': './third_party/unpkg/',
  }
);

// Output is a map of URLs to local paths
// The import map returned then works against these local paths directly
returning {
  // single file package
  'https://ga.jspm.io/npm:[email protected]/': {
    singleFile: true, // enabled by "singleFileNaming" option above
    target: './third_party/jspm/mod.js',
    traced: ['mod.js'],
    static: ['package.json', 'app.test.js']
  }
  
  // directory package
  'https://ga.jspm.io/npm:[email protected]/': {
    singleFile: false,
    target: './third_party/jspm/dep/',
    traced: ['index.js', 'dep.js'],
    static: ['package.json', 'asset.txt']
  }
  
  // multi-version package (and for its single-file variant)
  'https://ga.jspm.io/npm:[email protected]/': {
    singleFile: false,
    target: './third_party/jspm/[email protected]/',
    traced: ['index.js', 'dep.js', 'dynamic-import.js'],
    static: ['package.json', 'asset.txt', 'unused.js']
  }
}

The import map is then updated to work against the new file mappings, and the download operation can be completed with a simple fetch loop over the files. Non-JS files could be included in the file map for directory packages along with their associated mappings.

Support external / skip / ignore

Possibly in inputMap itself we could support an entry like false to mean external / skip.

Or another means of defining this.

CJS / ESM selection case - styled-components

Styled components broke recently at https://generator.jspm.io/#U2NhYGBkDM0rySzJSU1hKC6pBFK6yfm5Bfl5qXklxQ6mesZ6xgCnze/3JgA.

This is due to the CJS variant being used from https://ga.jspm.io/npm:@emotion/[email protected]/dist/stylis.browser.cjs.js instead of the .esm.js variant.

With the dominance change in #86, issues like this might be more likely.

Steps:

Test the latest version of this generator source against it to see if it's still doing that.
Try changing the priority in the PR at https://github.com/jspm/generator/pull/86/files to instead do a !wasCJS variant on all the conditionals to check if the ESM priority can be made
Possibly consider allowing priority to be user-defined to resolve cases like this in future or alternative customization approaches.

inputMap dedupes

In https://github.com/jspm/generator/blob/main/test/api/inputmap.test.js, the custom react mapping should be treated like an override (as it is supposed to be).

The result should be that the override also takes precedence for scoping to ensure scope deduping of custom mappings. The issue here might be versioning, but as a starting point that seems like better deduping logic and follow-up features could allow customization of this deduping.

Any plans for CLI support?

I just used the online generator and it was really good. But I think using an import map generator in the CLI which automatically generates import map from package.json would be very handy. I'm wondering if there are any plans to support this approach, too.

fetch contention

In the browser build, just naively calling fetch causes some serious bottlenecks, it's important for the native browser fetch implementation to use a manual queuing system to avoid this.

Eg something similar to guybedford/es-module-shims#184 would avoid these issues.

Unexpected token 'export' when pulling in single-spa as a dependency

Sorry if this is the wrong place to ask this kind of specific question. I am using this library to generate some import maps. One of the dependencies is single-spa. The package is referencing version 5.9.3. When I pipe it through the generator using these settings

    defaultProvider: 'jspm',
    env: ['production', 'browser']

I am getting this

"single-spa":"https://ga.jspm.io/npm:[email protected]/lib/umd/single-spa.min.js"

However, at the end of that file that it is pulling down is

;export default n;

Is there a reason that is being added? I don't see that in the npm version and I am not setting the env to 'module'. Am I missing some configuration somewhere? Thanks!

EnsureBuild failure modes

For bad package lookups, ensureBuild can stall the generator.

For example looking up [email protected] either an existing package or non-existing package.

It is important to verify that:

ensureBuild handles not found packages properly
ensureBuild when hitting a build that stalls will report the issue eventually with a timeout

Import map is inserted before the last script tag, not the first

With the HTML generation, if you have no importmap script tag and you have several other script tags, the importmap will generate just before the last script tag instead of just before the first script tag. If I add an empty importmap script where I want it, the generation is then handled correctly.

Input:

<!DOCTYPE html>

<script type="module">
    import "react";
</script>

<body>
    <script type="text/plain">
        Hello World
    </script>
</body>

Output:

<!DOCTYPE html>

<script type="module">
    import "react";
</script>

<body>
    <!-- Generated by @jspm/generator - https://github.com/jspm/generator -->
    <script async src="https://ga.jspm.io/npm:[email protected]/dist/es-module-shims.js" crossorigin="anonymous"></script>
    <script type="importmap">
    {
      "imports": {
        "react": "https://ga.jspm.io/npm:[email protected]/index.js"
      },
      "scopes": {
        "https://ga.jspm.io/": {
          "object-assign": "https://ga.jspm.io/npm:[email protected]/index.js"
        }
      }
    }
    </script>
    <link rel="modulepreload" href="https://ga.jspm.io/npm:[email protected]/index.js" />
    <link rel="modulepreload" href="https://ga.jspm.io/npm:[email protected]/index.js" />
    <script type="text/plain">
        Hello World
    </script>
</body>

Workaround:

<!DOCTYPE html>

<script type="importmap">
    {}
</script>

<script type="module">
    import "react";
</script>

<body>
    <script type="text/plain">
        Hello World
    </script>
</body>

Dependency aliasing

Reported on Discord giving a resolution error -

in package.json:

"dependencies": {
    "local-store": "npm:store@^2.0.12",
   }

https://ga.jspm.io/npm:@lit-async/[email protected]/directives/server-until.js

"dependencies": {
    "lit": "npm:@lit-async/lit@latest",
    "lit-html": "npm:@lit-async/lit-html@latest",
    "@lit/reactive-element": "npm:@lit-async/reactive-element@latest"
  },

Add a simple binary?

I’m not sure if that is possible, but I’d find it useful if I could generate an import map like this:

npx @jspm/generator node_modules/

Deno stdlib support

Support construction of mappings for Deno execution in the generator.

Pattern trailers support

We don't yet support pattern trailers per the Node.js support added. In addition #* in imports cuts off a character right now but should work too.

Support "cascade" interaction for inputMap

When setting the inputMap it is useful for the RHS targets that are package names, which are then resolved based on the known map.

Effectively this is calling traceInstall on the RHS of inputMap for all RHS targets that are bare specifiers.

Module not found

How do can I resolve these errors?

JspmError: Module not found: https://ga.jspm.io/npm:[email protected]/browser/index.js imported from https://ga.jspm.io/npm:[email protected]/dist/index.js

Consider support for cache busting mapping

Import maps can be useful for implementing local application code cache busting.

The way this could be specified is that a given path should have all remaps containing querystrings like:

{
  cachebust: { filter: '/some/path/', querystring: '?custom' }
}

where that would then handle remapping all traced URLs within that path within the import map using URL mappings.

Because Chrome doesn't yet support multiple import maps, there is a benefit to being able to handle user code features like this within this same generator API.

generator.d.ts is out of date

The current generator.d.ts for beta.11 doesn't reflect the true types in generator.ts - 'installMap' for example, is missing from GeneratorOptions

Top-level generator installs do not use resolutions object authoritatively

Support node: style install targets for Node.js builtins

Support import 'node:fs' as importing the Node.js builtin.
Generate maps in the import map like { "imports": { "node:fs": "/path/to/fs.js" } as URL mappings.

Generator CDN API throwing 401 Unauthorized error for jspm.system provider

I am trying to generate import map using the CDN API using "jspm.system" provider. Few days ago it was working, but now it is throwing 401 Unauthorized error! Please check the screenshot attached.
** Please note that it is working for other providers like "unpkg"!

fetch encapsulation

Properly encapsulate fetch versions between browser, Node.js, and Deno using the "imports" property in the package.json.

Trace "deoptimization"

When tracing an "eval" or full import wildcard, we should deoptimize the generation to fall back to a full generation mode for that package, whereby all resolvable entries and dependencies are traced by default.

Add a node provider

While working locally it would be nice to be able to use the local resources.

e.g. something like this

import { Generator } from '@jspm/generator';

const generator = new Generator({
  /*
   * Default: 'jspm'
   * Supported: 'jspm', 'jspm.system', 'skypack', 'jsdelivr', 'unpkg', 'node'.
   */
  defaultProvider: 'node',
});

// Install a new package into the import map
await generator.install('lit-element');

console.log(JSON.stringify(generator.getMap(), null, 2));
/*
 * Outputs the import map: 
 * {
 *    "imports": {
 *      "lit-element": "/node_modules/lit-element/lit-element.js",
 *      "lit-html": "/node_modules/lit-html/lit-html.js",
 *      "@lit/reactive-element": "/node_modules/@lit/reactive-element/reactive-element.js",
 *      "@lit/reactive-element/decorators/": "/node_modules/@lit/reactive-element/decorators/"
 *    }
 * }
 */

what do you think?

--
PS: adjusted request to use the name "node" instead of "local"

fetch include / exclude rules

It could be useful to support URL-based include and exclude pattern rules for the base fetch implementation.

This would support a basic security mechanism whereby you could eg restrict a build to just files of a certain folder on your system, or just files on certain CDNs but not others.

For example:

opts = {
  urlFilter: new URL('./localdir/', import.meta.url)
}

would restrict to local URLs only within the given folder. We could also allow regular expressions or functions returning true / false.

Babel Helpers non-determinism

When generating an import map for @material-ui/core, we've got a variation on the Babel helpers where typeof chooses the esm variant or not depending on the determinism of the importer.

See https://generator.jspm.io/#U2RgYGBkDM0rySzJSU1hcMhNLEktykzM0S3N1E/OL0p1MNEzNNQz0XdMBvJSMvPzAHy8WL8xAA.

Deno build

Full Deno build for JSPM Generator.

devDependencies scoping

Currently devDependencies is always used as a fallback if a dependency has no other range declaration.

But it turns out this is sometimes incorrect eg in the case of react-query, it has an optional dependency on react-dom but doesn't specify a version range. yet the version in devDependencies to React 16 is overly restrictive (https://ga.jspm.io/npm:[email protected]/package.json).

The generator should always ignore "devDependencies" for packages deemed non-local. The hard part is then just scoping which packages are deemed local versus non-local. For such a definition "devDependencies" restriction can then be performed, using an implicit "*" assumption instead as we do for no "devDependencies" match.

Non-deterministic resolution

Notice how @babel/runtime/helpers/typeof's mapping randomly switches between https://ga.jspm.io/npm:@babel/[email protected]/helpers/typeof.js and https://ga.jspm.io/npm:@babel/[email protected]/helpers/esm/typeof.js

Screen.Recording.2021-11-20.at.19.44.58.mov

In use version set and import map lockfile extraction

Currently the lockfile version extraction from import maps used in the htmlGenerate function works best with the direct scoped form of import maps which is not the default output (where scopes correspond to package boundaries for versioning resulting in duplication).

Instead the version resolver should be updated in the installer to deal in terms of an "in use package version set", where resolutions are prioritised from this pool based on the resolution rules, using new versions only when semver forces this or when the latest: true option is used.

The version collection from the import map would then work seamlessly with the version resolution in prepopulating this in use version set.

//cc @vovacodes this feature will likely solve your use case for the lockfile here.

Only extract staticDeps, dynamicDeps on finishInstall

Install finish is the only operation that has the final version resolution solution, so only at this stage should we be extracting the staticDeps and dynamicDeps lists, otherwise there is a risk of using an intermediate resolution that would have been orphaned by an alternate resolution.

Custom resolutions

There seem to be two major ways we could support custom resolutions:

Copy the Yarn style "resolutions" field as an input option to the generator, then use those as authoritative singular version resolutions.
Attempt to do something more general like a full map field that allows forcing map values. Something like map: { imports: { react: 'npm:react@17' } where package lookup in the map target is treated as sugar. This then extends to supporting any custom mappings within the import map generation.

Perhaps both (1) and (2) is the way to go as well, instead of having the RHS target sugar in (2).

This feature definitely needs to be implemented, the question is just how best to do it.

URL mappings support

URL mappings should be supported as a post-mapping for all non-bare-specifier resolutions. Simple example to support is query string mappings in inputMap.

Utility functions

Wondering about useful utility functions further for the generator.

Eg:

Exposing resolver
Exposing package metadata lookups
Exposing exports resolutions etc

This really comes down to use cases though.

Support arbitrary existing import map as input lock

Instead of treating all mappings of inputMap as authoritative, we should always treat inputMap as first going through extractLockAndMap to prepopulate the lockfile and custom maps separately.

If you want an authoritative inputMap you could just use new Generator({ inputMap, freeze: true }) to retain the existing behaviour.

lockfile support

We currently have lockfile support in the installer via https://github.com/jspm/generator/blob/main/src/install/installer.ts#L60 along with the install option freeze, which when used will even freeze any updates of dependencies from that provided lockfile.

It would be great to expose this option to the Generator constructor so that users could pass a custom lockfile.

The lock format is currently typed at https://github.com/jspm/generator/blob/main/src/install/installer.ts#L36, and takes the following form:

{
  "file:///path/to/pkg/": {
    "dep": "https://ga.jspm.io/npm:pkg@version/"
  }
}

The main base URL used can just be generator.baseUrl for the local resolutions. It is effectively the URL of the package.json folder that defines the dependencies.

It is important that all URLs are fully normalized - thus the initial input into the lockfile system needs to deal with normalization into this format.

The standard library resolutions for Node.js use a special symbol | in the lockfile to indicate packages that are installed to export values.

For example, fs would be written:

{
  "file:///path/to/baseurl/": {
    "fs": "https://ga.jspm.io/npm:@jspm/[email protected]/|fs"
  }
}

Which ensures that it will always resolve to the correct environment fs library against the JSPM core package on the CDN. This is the only exception case to worry about.

To generate this lock format from an input map should be relatively straightforward based on effectively just iterating over the import map and collecting the resolutions.

We could then automatically support treating a file with "imports" or "scopes" as an import map input, versus the direct lockfile input. We could possibly even handle automatic support for npm and Yarn lockfile formats as further work in future but that isn't necessary for the MVP.

Steps:

Expose "lockfile" option, supporting this format exactly, just with an iteration that converts all URLs into absolute URLs relative to the baseUrl of the generator so that a proper normalized lockfile is passed in.
Support detecting "imports" and "scopes" on this object then converting an import map input into this same lockfile format through an iterative reduction. For a given URL https://path/to/module.js, to determine the package boundary, there is already a resolver.getPackageBase(url) function which can be reliably used for any URL in a way that works with the provider hooks properly.
Profit!

Browser build

Proper browser build is still pending...

Resolve symlinks for monorepo patterns

Appling realpath on the fetch implementation for file: URLs should help with monorepos.

Support environment multi-branching

Instead of having the generator trace for a single environment solution, it can be useful to support the ability to trace all possible environment solutions at the same time into a conditional import map.

The benefit of this is when plugging the generator into bundlers as a linker for bundling.

The environment resolution can then be deferred until import map serialization time, allowing eg a single generation operation to output a map for Deno, a map for the browser and another map for Node.js.

RangeError in resolver

Test case:

import { Generator } from '#dev';

const generator = new Generator();

await generator.install({ target: '[email protected]', subpath: './esm/vs/editor/editor.api' });

output:

➜  generator git:(main) ✗ npm test

> @jspm/[email protected] test
> node test/test.js

............> /*---------------------------------------------------------------------------------------------
file:///Users/mgibson/github/generator/lib/install/resolver.js:339
                    console.log('  ' + ' '.repeat(col - 1) + '^');
                                           ^

RangeError: Invalid count value
    at String.repeat (<anonymous>)
    at Resolver.analyze (file:///Users/mgibson/github/generator/lib/install/resolver.js:339:44)
    at async TraceMap.traceUrl (file:///Users/mgibson/github/generator/lib/tracemap/tracemap.js:240:64)
    at async TraceMap.trace (file:///Users/mgibson/github/generator/lib/tracemap/tracemap.js:144:13)
    at async file:///Users/mgibson/github/generator/lib/tracemap/tracemap.js:254:30
    at async Promise.all (index 84)
    at async TraceMap.traceUrl (file:///Users/mgibson/github/generator/lib/tracemap/tracemap.js:253:9)
    at async TraceMap.trace (file:///Users/mgibson/github/generator/lib/tracemap/tracemap.js:206:17)
    at async Generator.install (file:///Users/mgibson/github/generator/lib/generator.js:32:13)
    at async file:///Users/mgibson/github/generator/test/errors/range-error.test.js:5:1
...

staticDeps and dynamicDeps as full graph

Currently staticDeps and dynamicDeps are just arrays of modules, without further graph information.

It could be useful to actually expose the entire graph for these in the generation to be able to lookup arbitrary dependency information.

We currently have a getAnalysis method which returns the analysis per module. It could be useful to effectively get the whole graph in one:

generator.getGraph() = () => ({
  [url: string]: {
      staticDeps: { specifier: string }[],
      dynamicDeps: { specifier: string }[]
  }
})

The current getAnalysis could be extended to support this form relatively easily.