Coder Social home page Coder Social logo

loader-runner's Introduction

loader-runner

import { runLoaders } from "loader-runner";

runLoaders({
	resource: "/abs/path/to/file.txt?query",
	// String: Absolute path to the resource (optionally including query string)

	loaders: ["/abs/path/to/loader.js?query"],
	// String[]: Absolute paths to the loaders (optionally including query string)
	// {loader, options}[]: Absolute paths to the loaders with options object

	context: { minimize: true },
	// Additional loader context which is used as base context

	processResource: (loaderContext, resourcePath, callback) => { ... },
	// Optional: A function to process the resource
	// Must have signature function(context, path, function(err, buffer))
	// By default readResource is used and the resource is added a fileDependency

	readResource: fs.readFile.bind(fs)
	// Optional: A function to read the resource
	// Only used when 'processResource' is not provided
	// Must have signature function(path, function(err, buffer))
	// By default fs.readFile is used
}, function(err, result) {
	// err: Error?

	// result.result: Buffer | String
	// The result
	// only available when no error occurred

	// result.resourceBuffer: Buffer
	// The raw resource as Buffer (useful for SourceMaps)
	// only available when no error occurred

	// result.cacheable: Bool
	// Is the result cacheable or do it require reexecution?

	// result.fileDependencies: String[]
	// An array of paths (existing files) on which the result depends on

	// result.missingDependencies: String[]
	// An array of paths (not existing files) on which the result depends on

	// result.contextDependencies: String[]
	// An array of paths (directories) on which the result depends on
})

More documentation following...

loader-runner's People

Contributors

alexander-akait avatar chalker avatar commanderroot avatar dependabot[bot] avatar edwardbetts avatar jugglinmike avatar niieani avatar noscripter avatar olleolleolle avatar opichals avatar sokra avatar tarang9211 avatar thelarkinn avatar vankop avatar xandris avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

loader-runner's Issues

Should `result.result` be an array?

Hey there!

I was using the lib and discovered that result.result that I get is an array. But README says:

// result.result: Buffer | String
// The result
// only available when no error occured

I'm not quite sure whether there an error or not, but probably type declaration in the README should be different.

Avoid System.import?

Hi, I've run into an interop problem between systemjs-builder and webpack running concurrently in the same process. In short, loader-runner picks up on the System global installed by systemjs due to this code path.

I don't know if it's really loader-runner's fault, but I ended up solving my problem by forking it and cutting out that part. And now that webpack 2 went with import(...) instead of System.import(...), is it still needed?

Issue with System module undefined error when running webpack 3.4.1 from custom gulp task

Given the following config(in file webpack.config.js) using either ts-loader or awesome-typescript-loader:

const path = require('path');
let _root = path.resolve(__dirname);
var fromRoot = function (...args) {
    args = Array.prototype.slice.call(args, 0);
    return path.join.apply(path, [_root].concat(args));
};
module.exports = {
    context: fromRoot( "src" ),
    devtool: "cheap-module-eval-source-map",
    entry: {
        "app/polyfills": "./app/polyfills.ts",
        "app/vendor": "./app/vendor.ts"
    },
    resolve: {
        modules: [
            fromRoot( "src", "node_modules" ),
            fromRoot( "node_modules" )
        ],
        extensions: ["*", ".js", ".ts"],
        alias: {
            build: fromRoot( "build", "src", "public" ),
            jquery: fromRoot( "src", "node_modules", "jquery", "src", "jquery" )
        }
    },
    module: {
        rules: [{
                test: /\.ts$/,
                use: [
                    {
                        loader: "ts-loader",
                        options: {
                            configFileName: fromRoot( "src", "tsconfig.json" ),
                            compilerOptions: {
                                rootDir: fromRoot( "src" ),
                                outDir: fromRoot( "build", "src", "public" )
                            }
                        }
                    }
                ]
            }
        ]
    },
    output: {
        path: fromRoot("build", "src", "public"),
        publicPath: "/",
        filename: "[name].js",
        chunkFilename: "[id].chunk.js"
    }
};

export = webpackMerge(commonConfig, {
    devtool: "cheap-module-eval-source-map",
    output: {
        path: fromRoot("build", "src", "public"),
        publicPath: "/",
        filename: "[name].js",
        chunkFilename: "[id].chunk.js"
    }
    
}, webpackDevConfig);

Running webpack works.

Given the following file, test.js:

var webpack = require("webpack");
var webpackConfig = require("./webpack.config.js");

var compiler = webpack(webpackConfig, (err, stats) => {	
    if (err) {
        console.error(err);
    };
});

Running node ./test.js works.

But given the following gulp task:

var webpack = require("webpack");
var webpackConfig = require("./webpack.config.js");
var starting = true;
gulp.task("ugh", function(cb) {
    var compiler = webpack(webpackConfig, (err, stats) => {	
        if (err) {
            console.error(err);
        };
        if (starting) {
            starting = false;
            cb();
        }
    });
});

Running gulp ugh yields these errors:

Unhandled Rejection at: Promise Promise {
<rejected> TypeError: Cannot read property 'default' of undefined
at /Users/mitchellmorris/Sites/projects/sandbox/node_modules/loader-runner/lib/loadLoader.js:4:66
} reason: TypeError: Cannot read property 'default' of undefined
at /Users/mitchellmorris/Sites/projects/sandbox/node_modules/loader-runner/lib/loadLoader.js:4:66

Unhandled Rejection at: Promise Promise {
<rejected> TypeError: Cannot read property 'default' of undefined
at /Users/mitchellmorris/Sites/projects/sandbox/node_modules/loader-runner/lib/loadLoader.js:4:66
} reason: TypeError: Cannot read property 'default' of undefined
at /Users/mitchellmorris/Sites/projects/sandbox/node_modules/loader-runner/lib/loadLoader.js:4:66

From what I can tell given the following block that System(in node_modules/loader-runner/lib/loadLoader.js) only defined when leveraging gulp and emitting an undefined result(i.e. module).

module.exports = function loadLoader(loader, callback) {
    if(typeof System === "object" && typeof System.import === "function") {
        console.log("This only happens using gulp.");
        System.import(loader.path).catch(callback).then(function(module) {
            console.log("further more module is undefined!");
            loader.normal = typeof module === "function" ? module : module.default;
            // ...
        });
    } else {
        console.log("This only happens using node ./test.js or webpack and is successfull.");
        // ...
    }
}

Can anyone help please?

Add a way to share data across loaders for a given resource

It would be great if there was a way to store some meta data related to the compiled resource across the whole stack of loaders.

I'm trying to parse front-matter early on to then use the collected values later for generating extra source a couple of loaders later. Unfortunately, the meta argument I pass through this.callback gets lost after the next loader.

Would you be open to having the loaderContext used to invoke each loader bear an extra property (resourceMeta?) where loaders could store metadata to keep during the run?

Debugging SystemJS?

I'm trying to debug an issue I'm having with System.import within loader-runner right now, but honestly I'm not sure where System is even coming from in the first place. It's not part of this module, or webpack core.

The issue is that when it's given a path to import, it is unable to correctly resolve requires or imports from inside that file. Confusingly, it tries to resolve from the root of the webpack project, not even any node_modules folder. So for example, I have something like this file being System.imported:

const someModule = require('some_module')

module.exports = function someLoader (source) {
  return source
}

When you run System.import with the path to this file, it throws an error from trying to resolve some_module relative to the project root, where it certainly does not exist. It seems like it resolves it according to its baseURL property, which is set as the project root.

It seems like it comes from SystemJS, the System object is called SystemJSNodeLoader, and defines its own version as 0.19.31 Node.

'this.fs' does not work from inside the module

Hi, I'm running into this issue when trying to load a file from inside the loader:
`ReferenceError: fs is not defined

  35 |     console.log(this.getDependencies());
  36 | 
> 37 |     fs.readFile(fileToSearch, 'utf8', (error, data) => {
     |     ^`

Error doesn't happen when running from webpack config. Only with loader-runner.

Could you help me out?

Missing license

Hello, is it possible to incorporate the LICENSE file inside the repo and npm package ? Now it's missing. It's implied by MIT license that if licensed under the MIT the source code must be accompanied by full license text. If the license file is not in the repo nor npm package and lot of people cannot use it.

Thank you

We've already provided a PR solving this: #27

@ChALkeR, @TheLarkInn, @sokra could you help us pls or anybody from your webpack org ?

Cannot find module if the path contains illegal character

Expected Behaviour

Loaders work if the path contains # character.

Actual Behaviour

Webpack build fails with ERROR (tested on ts-loader and css-loader):

../../a#/ts-loader-path-issue/src/index.ts 39 bytes [not cacheable] [built] [code generated] [1 error]

ERROR in ../../a#/ts-loader-path-issue/src/index.ts
Module build failed (from ../../a#/ts-loader-path-issue/node_modules/ts-loader/index.js):
Error: Cannot find module 'C:\a#\ts-loader-path-issue\node_modules\ts-loader\index.js'
Require stack:
- C:\a#\ts-loader-path-issue\node_modules\loader-runner\lib\loadLoader.js
- C:\a#\ts-loader-path-issue\node_modules\loader-runner\lib\LoaderRunner.js
- C:\a#\ts-loader-path-issue\node_modules\webpack\lib\NormalModule.js
- C:\a#\ts-loader-path-issue\node_modules\webpack\lib\NormalModuleFactory.js
- C:\a#\ts-loader-path-issue\node_modules\webpack\lib\Compiler.js
- C:\a#\ts-loader-path-issue\node_modules\webpack\lib\webpack.js
- C:\a#\ts-loader-path-issue\node_modules\webpack\lib\index.js
- C:\a#\ts-loader-path-issue\node_modules\webpack-cli\lib\webpack-cli.js
- C:\a#\ts-loader-path-issue\node_modules\webpack-cli\lib\bootstrap.js
- C:\a#\ts-loader-path-issue\node_modules\webpack-cli\bin\cli.js
- C:\a#\ts-loader-path-issue\node_modules\webpack\bin\webpack.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
    at Function.Module._load (internal/modules/cjs/loader.js:725:27)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (C:\a#\ts-loader-path-issue\node_modules\v8-compile-cache\v8-compile-cache.js:159:20)
    at loadLoader (C:\a#\ts-loader-path-issue\node_modules\loader-runner\lib\loadLoader.js:19:17)
    at iteratePitchingLoaders (C:\a#\ts-loader-path-issue\node_modules\loader-runner\lib\LoaderRunner.js:182:2)
    at runLoaders (C:\a#\ts-loader-path-issue\node_modules\loader-runner\lib\LoaderRunner.js:395:2)
    at NormalModule.doBuild (C:\a#\ts-loader-path-issue\node_modules\webpack\lib\NormalModule.js:631:3)
    at NormalModule.build (C:\a#\ts-loader-path-issue\node_modules\webpack\lib\NormalModule.js:775:15)
    at C:\a#\ts-loader-path-issue\node_modules\webpack\lib\Compilation.js:1236:12

Steps to Reproduce the Problem

On Windows 10 with run these commands in PowerShell (tested with 5.1):

mkdir C:\a#
cd C:\a#
git clone https://github.com/csutorasa/ts-loader-path-issue.git
cd ts-loader-path-issue
npm install
.\node_modules\.bin\webpack
# if webpack fails set execution policy and run the last command again
# Set-ExecutionPolicy Bypass -Scope Process -Force

On Linux run these commands in bash (tested with 5.0.17):

mkdir -p ~/a#
cd ~/a#
git clone https://github.com/csutorasa/ts-loader-path-issue.git
cd ts-loader-path-issue
npm install
node_modules/.bin/webpack

Location of a Minimal Repository that Demonstrates the Issue.

https://github.com/csutorasa/ts-loader-path-issue (ts-loader repo)
https://github.com/csutorasa/loader-path-issue (css-loader repo)

Loader cannot load ES6 script

I have a ES6 angular 13 project that runs angular-cli with customWebpackConfig ES6 script:

angular.json:

"customWebpackConfig": {
   "path": "./custom-webpack.config.mjs"
},

custom-webpack.config.mjs

import path from 'path';
import webpack from "webpack";
export default function(config) {
    config.module.rules.unshift({
        test: /app\.ts$/i,
        exclude: [/node_modules/],
        use: [{loader: path.resolve('./custom-loader.mjs')}]
    });
    return config;
};

custom-loader.mjs

import _sourceMap from "source-map";
import fs from 'node:fs';
export default function(content, sourceMap) {...}

What is the current behavior?

 Error: Module build failed (from ./custom-loader.mjs):
Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /custom-loader.mjs
    at Module.load (internal/modules/cjs/loader.js:935:11)
    at Function.Module._load (internal/modules/cjs/loader.js:778:12)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at loadLoader (/node_modules/loader-runner/lib/loadLoader.js:19:17)
    at iteratePitchingLoaders (/node_modules/loader-runner/lib/LoaderRunner.js:182:2)
    at runLoaders (/node_modules/loader-runner/lib/LoaderRunner.js:398:2)
    at NormalModule._doBuild (/node_modules/@angular-devkit/build-angular/node_modules/webpack/lib/NormalModule.js:819:3)
    at NormalModule.build (/node_modules/@angular-devkit/build-angular/node_modules/webpack/lib/NormalModule.js:963:15)
    at /node_modules/@angular-devkit/build-angular/node_modules/webpack/lib/Compilation.js:1371:12

On the other hand, when I add the parameter type: "module" in the use array:
use: [{loader: path.resolve('./custom-loader.mjs'), type:"module"}] I receive an error:

Generating browser application bundles (phase: setup)...An unhandled exception occurred: Invalid configuration object. Webpack has been initialized using a configuration object that does not match the API schema.
 - configuration.module.rules[0].use[0] has an unknown property 'type'. These properties are valid:
   object { ident?, loader?, options? }

But, if I remove this validation the ES6 script is executing successfully.
Instead of changing Webpack validation schema I also tried to update the LoaderRunner.js, line 78 and ProgressPlugin.js, line 405. It also worked:
obj.type = value.type || value.options && value.options.type || '';

It looks like this feature already exists. It just isn't enabled?

Other relevant information:
webpack version: 5.74
Node.js version: 14 and 16
Operating System: MacOS
Additional tools:
@angular-devkit/build-angular: 13.3.9

refactoring to ES6?

Hey, looking to contribute to this! Does any of this codebase require refactoring to ES6?

Support `context.exec` feature?

I use the lib to test my custom loader, and my loader depend on context.exec.

So, is there a plan to support context.exec in the future?

How to indicate that a loader uses ES modules

How do you indicate via config or options that a loader should be loaded using import instead of require?

How do you get this logic to kick-in?

I have a loader that is written with ES modules, and my package.json specifies "type": "module". I am running Jest using the node option --experimental-vm-modules and my tests written with ES modules are all working fine except for this one which has the following error: NonErrorEmittedError: (Emitted value instead of an instance of Error) Error: Must use import to load ES Module.

Relevant stack trace:

      at processResult (node_modules/webpack/lib/NormalModule.js:718:12)
      at node_modules/webpack/lib/NormalModule.js:827:5
      at node_modules/loader-runner/lib/LoaderRunner.js:399:11
      at node_modules/loader-runner/lib/LoaderRunner.js:185:11
      at loadLoader (node_modules/loader-runner/lib/loadLoader.js:33:11)
      at iteratePitchingLoaders (node_modules/loader-runner/lib/LoaderRunner.js:182:2)
      at runLoaders (node_modules/loader-runner/lib/LoaderRunner.js:397:2)
      at NormalModule.doBuild (node_modules/webpack/lib/NormalModule.js:781:3)
      at NormalModule.build (node_modules/webpack/lib/NormalModule.js:928:15)

Loader options functions are not serialized correctly

If you pass loader options that have a function within them, something like

{
    loader: 'css-loader',
    options: {
        url: {
            filter:
                (url) => {
                    if (url.includes('abc')) {
                        return false;
                    }
                    return true;
                }
        }
    }
} 

then this will be not converted to a function in the worker. Instead the values are missing.
Simply JSON.stringify is called, which returns an empty object for url.
Instead, we should go through each child and if it is a function we need to call toString.
Later in the worker, we then need to set an eval + stringified function, so that it gets set correctly.

Feature proposal: Add getOptions to loaderContext inside runLoaders

Feature Proposal

Introduces getOptions function inside loaderContext.

The implementation of this function should mirror the one inside webpack NormalModule.
I think we can use loader context in the same way of webpack to extract loader's options and validate them if a schema is provided (Include schema-utils and json-parse-better-errors (or maybe directly use JSON.parse) as dependencies).

Feature Use Case

getOptions will become available to loaders called via loader-runner.
thread-loader is one of them and has an open issue about undefined this.getOptions.


I have created a simple reproduction repository for reference.
If you agree I can try to do a PR.

I encountered some bad cases when using thread-loader on windows

When I used thread-loader to speed up compilation, I encountered a bad case on windows. When I refrence the es5-ext package, thread-loader will report the following error:

ERROR in ./node_modules/es5-ext/string/#/ends-with/index.js
Module build failed (from ./node_modules/thread-loader/dist/cjs.js):
Thread Loader (Worker 0)
EISDIR: illegal operation on a directory, read
    at PoolWorker.fromErrorObj (D:\Development\test-thread-loader\node_modules\thread-loader\dist\WorkerPool.js:346:12)
    at D:\Development\test-thread-loader\node_modules\thread-loader\dist\WorkerPool.js:219:29
    at mapSeries (D:\Development\test-thread-loader\node_modules\neo-async\async.js:3625:14)
    at PoolWorker.onWorkerMessage (D:\Development\test-thread-loader\node_modules\thread-loader\dist\WorkerPool.js:173:34)
    at D:\Development\test-thread-loader\node_modules\thread-loader\dist\WorkerPool.js:146:14
    at Socket.onChunk (D:\Development\test-thread-loader\node_modules\thread-loader\dist\readBuffer.js:40:9)
    at Socket.emit (events.js:314:20)
    at Socket.Readable.read (_stream_readable.js:507:10)
    at Socket.read (net.js:625:39)
    at flow (_stream_readable.js:1007:34)
 @ ./src/index.js 1:0-49 3:12-20

The minimum unit test for this error can be viewed at https://github.com/YourWildDad/test-thread-loader

I think it is because the loader-runner splits the resource fragment, which leads to abnormal reading of the file.

Please help to fix this error, good luck.

ES6 import error

ES6 import error

// error
import { runLoaders } from "loader-runner";

image

// OK
const { runLoaders } = require(`loader-runner`);

image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.