tkoenig89 / express-static-gzip Goto Github PK

I've noticed that when I run my gulp task to zip the files into the public folder, I have to restart my server because it doesn't pick up the new files. Can this be fixed?

The Vary header is only used by caching proxies and the failure case can be for example that:

1. user1 fetches app.js without Accept-Encoding header (or with Accept-Encoding value that has no precompressed file such as bzip2)
2. the express-static-gzip does not Vary header in response
3. proxy caches the result
4. user2 asks for the resource with Accept-Encoding: br, gzip
5. the proxy finds the uncompressed version and serves that one (because the response did not have proper Vary)

If the uncompressed also had a Vary header the proxies would not get 'poisoned' with uncompressed versions. This is normally not a problem for internal applications, but with the wild west of internet and it's caches it would be much safer to have the header correctly implemented.

See HSLdevcom/digitransit-ui#1074 (comment)

I just started to use this nice library and integrated it into our nodejs backend which is intended to serve both - static and dynamic content.

There is one behaviour which I did not expect: API requests to our REST endpoint stopped working if their path was ending with "/".

The issue is, that indexFromEmptyFile is enabled by default and thus every request is manipulated if it is ending with "/". - Even if the serve-static middleware does not handle the request.
In our scenario, the next middleware - nestjs - cannot handle the request anymore as the path does not match.

when using a maxAge option in the object as the second arg to expressStaticGzip, I get the TS error 'maxAge' does not exist in type 'ExpressStaticGzipOptions' which is correct. That type is missing from the options-type. Howver, it's used as an example in the readme maxAge: 123,. Which is correct?

Hi @tkoenig89

I'm using this middleware to serve static compressed files using brotli compression, and came across an issue whereby if my server is probed with an invalid URI path then the middleware throws an error rather than it being caught and handled gracefully.

app.use(
    '/',
    expressStaticGzip('dist/client', {
      enableBrotli: true,
      orderPreference: ['br'],
      index: false,
      serveStatic: {
        maxAge: 31536000000,
        immutable: true,
        setHeaders: (res, currPath) => {
          if (currPath.includes('/remoteEntry.js')) {
            res.setHeader('Cache-Control', 'public, max-age=0');
          }
        },
      },
    }),
  );

If I try to access the following URL on the server http://localhost:5000/%c0 a 500 error is thrown with a URI Error: URI Malformed coming from the decodeURIComponent function being called here

Any idea on how I can resolve this?

Thanks
James

Hello, compressed files are served only from root directory.

I guess problem is in line 95

Should be

    findCompressedFilesInDirectory(filePath);

instead of

    findCompressedFilesInDirectory(fs, filePath);

Like serve-static, express-static-gzip should also be able to serve a single file.
eg:
expressApp.use('/service-worker.js', expressStaticGzip('service-worker.js'));

Currently, the error is:

Error: ENOTDIR: not a directory, scandir 'C:\test\dist\service-worker.js'
    at Object.fs.readdirSync (fs.js:909:18)
    at findAllCompressionFiles (C:\test\node_modules\express-static-gzip\index.js:127:24)
    at expressStaticGzip (C:\test\node_modules\express-static-gzip\index.js:28:9)
    at serve (C:\test\myserver.js:344:10)
    ...

I don't want "changeUrlFromEmptyToIndexHtml" function to change my req.url. The only way to avoid it is to pass { index: false } option to "expressStaticGzip", but this option is also passed to "ServeStatic". This is not what I expected to do. Please, make passing "index" to "serve-static" or calling "changeUrlFromEmptyToIndexHtml" optional

one of the requirements for serving brotli encryption is a https protocol.

right now, when setting up express-static-gzip with brotli enabled, any request not over https fails.

would it be possible to verify req protocol is https before trying to serve .br files?

There are a error at server side render (Isomorphic react render). The plugin are serving the .gz at server side. Otherwise, that file should be a .js, without the .gz extension.

As I understand the code, it loads a list of available compressed files into a sort of cache at startup, and the middleware looks for the compressed version of the requested file in that cache. There are several potential problems with this approach that may catch people off guard:

• if a new compressed file is added after startup, it will never be served by express-static-gzip. I think it's reasonable to support files being added later, what if someone is gzipping files after they're uploaded and then serving them back later?
• if a compressed file is removed for any reason, I think the code will fail to serve the uncompressed file instead
• what if there are so many files in the directory that they can't all fit into memory?
• if there are a large number of files, it increases startup time

All of these problems would be alleviated by not building any kind of cache and instead just checking for the existence of foo.js.gz at the time that foo.js is requested. I don't think it would be difficult to perform that check when the request is made, so it seems like a win-win type of change.

But I haven't dug deeply into the guts of how you build that cache so maybe there's something I'm missing; is there some reason you can't just check the fs for the compressed files upon request?

Sorry if this is a dumb question but have you thought about removing the serveStatic part?

As it is I call something like

app.use("/", expressStaticGzip("/my/rootFolder/"));

and it uses both this code and serveStatic but that means it's hard coded to serve static, and a certain version and you have a dependency that needs to be updated etc.

Wouldn't be just as easy yet more flexible to just call next and if I want static files I could do

app.use("/", expressStaticGzip("/my/rootFolder/"));
app.use("/", express.static("/", staticOptions));

And then I could use other things as well?

The middleware is not serving gzip files as intended.I tried reloading the server but still not working. Here's my code:

server/config/express.js

var path = require('path');
var express = require("express");
var expressStaticGzip = require("express-static-gzip");
var app = express();

module.exports= function() {
    app.use(express.static('./public'));
    app.use("/", expressStaticGzip(path.resolve(__dirname, '../../public/app/assets/js/')));

module.app = app;

return module;

}

public/app/assets/js/

• common.min.js 500KiB
• common.min.js.gz 180 KiB

When I try from the browser to access example.com/app/assets/js/common.min.js the uncompressed file is served always.

I'm using node v12.10.0, expressjs 4.17.1 and express-static-gzip 2.0.5

I compress files beforehand with webpack, so I have for example bundle.js, bundle.js.gz and bundle.js.br If I leave everything default and just use gzip when I inspect response headers in browser I see no Content-Encoding. There is also no indication of size difference which is interesting because if I inspect with curl I see Content-Encoding: gzip and also some online tests are reporting that I am using gzip.

If I turn on option for brotli, everything is working as it should. If I inspect response Headery in browser I see Content-Encoding and I also see the size difference, the size of original file and the size of brotli zip version.

So I am not sure in the end if gzip is enabled.

Hi, I'm using express-static-gzip in vue ssr (no nuxt.js) and it doesn't work and I dont know why.
I can't see content-enconding in my header

My code

import path from 'path';
import express from 'express';
import fs from 'fs';
import expressStaticGzip from 'express-static-gzip';

import vueServerRenderer from 'vue-server-renderer';
import dotenv from 'dotenv';
import { getTemplateFromRequest } from './template.mjs';
import serverBundle from '../dist-etu-ssr/vue-ssr-server-bundle.json';
import clientManifest from '../dist-etu-ssr/vue-ssr-client-manifest.json';
import { startApiRoutes } from './router/index.mjs';
import { preloadServer } from '../utils/server.mjs';
import redirects from './redirects/redirects.json';

const { createBundleRenderer } = vueServerRenderer;
const __dirname = path.dirname(new URL(import.meta.url).pathname);

dotenv.config();
preloadServer();
const server = express();

server.use('/dist-etu-ssr', expressStaticGzip('../dist-etu-ssr', {
    enableBrotli: true,
    orderPreference: ['br', 'gz']
}));
server.use(express.json());

startApiRoutes(server);

server.get('*', (req, res) => {
    redirects.forEach((element) => {
        if (element.pattern !== 'undefined') {
            const params = req.url.match(element.pattern);

            if (params === null) {
                return;
            }

            let redirectUrl = element.to;

            element.to.match(/([1-9])/g).forEach((number) => {
                redirectUrl = redirectUrl.replace(number, params[number]);
            });

            if (redirectUrl !== 'undefined') {
                res.redirect(redirectUrl);
            }
        }
    });
etc ...

My server conf is based from
https://github.com/horprogs/vue-ssr-hmr/blob/master/app/server.js

for some reason when i enable express-static-gzip, it ignores the cache-control headers i set for my public folder with express.static. it should have max-age=604800 but instead my brotli and gzip compressed files return with max-age=0. i've also tried using other express caching middleware, but even though it works on my pages, it won't work on static assets served via brotli. how, then, can i set max-age of these objects?

  // static assets
  const oneYear = { maxAge: 604800000 }
  const staticFiles = [
    { file: 'browserconfig.xml', path: 'public/browserconfig.xml' },
    { file: 'favicon.ico', path: 'assets/img/favicon/favicon.ico' },
    { file: 'favicon', path: 'assets/img/favicon/' },
    { file: 'manifest.json', path: 'public/manifest.json' },
    { file: 'package.json', path: './package.json' },
    { file: 'public', path: 'public/' },
    { file: 'sw.js', path: './sw.js' }
  ]

  let robotsEnv
  process.env.ENVIRONMENT === 'production' ? robotsEnv = 'production' : robotsEnv = 'development'
  staticFiles.push({ file: 'robots.txt', path: `public/robots.txt.${robotsEnv}` })

  if (process.env.ENVIRONMENT === 'local') {
    staticFiles.push({ file: 'assets', path: 'assets/' })
    staticFiles.push({ file: 'src', path: 'src/' })
  }

  app.use('/public/', expressStaticGzip(`${process.env.DOC_ROOT}public/`, { enableBrotli: true }))

  for (let file of staticFiles) {
    app.use(`/${file.file}`, express.static(`${process.env.DOC_ROOT}${file.path}`, oneYear))
  }

  app.use(cacheControl({
    maxAge: 1800,
    public: true
  }))

Hello,
I create a server side rendered react app, for compression I use webpack plugin to compress js files into gzip and brotli. It compressed successfully. But, I got an issue about response header. It always return content type gzip.

If I only compress the js files into brotli, and run the project. The response header return content of application/javascript;charset=UTF-8.

My project runs locally using nodemon (No SSL), I use Latest Chrome and Latest Express static gzip.

is there any configuration that I need to use to make the header return content type of brotli?

This is my code screenshot and the response header:
https://res.cloudinary.com/spacesam/image/upload/v1564329266/code-gzip.png
https://res.cloudinary.com/spacesam/image/upload/v1564329215/gzip.jpg

Just now read https://medium.com/@rajaraodv/two-quick-ways-to-reduce-react-apps-size-in-production-82226605771a#.9woyhpj4o

And saw that in step 4 it has no check Accept-Encoding:

4. Finally, add this middleware to Express to return .js.gz so you can still load bundle.js from the html but will receive bundle.js.gz
app.get('*.js', function (req, res, next) {
  req.url = req.url + '.gz';
  res.set('Content-Encoding', 'gzip');
  next();
});

After 5 min of googling I found your awesome lib. Exactly what is needed! Thanks!

I have a react app that has a round like /homepage/. This maps to a component and not an index.html file. However when I try to use expressStaticGzip and the server hits /homepage/ it'll try to serve /homepgae/index.html and crash.

Hi,
I am trying to use the middleware express-static-gzip.

It seems, that I found a bug. If the name of the folder containing static gzipped files contains '.gz',
no pre-compressed content will be delivered.

Steps to reproduce:

1. create folder "wwwroot.gzipped" in the application root
2. put gzpped files into the folder, e.g. test.html.gz
3. register the folder as express-static-gzip root folder:
   app.use("/", expressStaticGzip('/wwwroot.gzipped'));
4. start the application, by me it works on localhost:8080
5. try to get http://localhost:8080/test.html
6. result: error 404, "Cannot GET /test.html"

Renaming the folder to "wwwroot-gzipped" helps to avoid the problem.

The error is in index.js;167

    /**
     * Adds the compression to the file's list of available compressions
     * @param {string} filePath
     * @param {Compression} compression
     */
    function addCompressionToFile(filePath, compression) {
// !! in the next line there is an error - fileExtension is removed also from the middle of filePath !!
        var srcFilePath = filePath.replace(compression.fileExtension, "").replace(rootFolder, "");
        var existingFile = files[srcFilePath];
        if (!existingFile) {
            files[srcFilePath] = { compressions: [compression] };
        } else {
            existingFile.compressions.push(compression);
        }
    }

So, in the line

        var srcFilePath = filePath.replace(compression.fileExtension, "").replace(rootFolder, "");

fileExtension is removed also from the middle of filePath :(

I am using

• express-static-gzip version 1.1.1,
• node.js version 8.11.1,
• express version 4.16.3

Best regards
yarick123

I want by backend express server to serve gziped bundle.js.

My server.js (I included only the part responsible for serving gzip)

app.use("/", expressStaticGzip(DIST_DIR, {}));

My index.html

<html>
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="minimum-scale=1, initial-scale=1, width=device-width" />
    <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap" />
    <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons" />
    <title>Architects of Belarus</title>
  </head>
  <body>
    <div id="app"></div>
    <script type="text/javascript" src="bundle.js.gz"></script></body>
</html>

When I load the page, there are the following errors in the console:

The script from “http://localhost:3000/bundle.js.gz” was loaded even though its MIME type (“application/gzip”) is not a valid JavaScript MIME type.
localhost:3000
SyntaxError: illegal character
bundle.js.gz:1

I believe that this problem would be solved if I could set the MIME type to 'application/javascript'

I've followed your guide (using Brotli) and I'm able to render my root route '/' and nested routes as well, but of course refreshing breaks the implementation you've provided due to my client-side routing. Is there any other options to use which supports this issue? I realise I'd usually send back the index.html file on any 404 error, however I'm not sure how to accomplish this since your code snippet (if I understand correctly) handles GET requests, but I don't see an example for nested routes there.

Any info is appreciated!

If a subdirectory of the directory being scanned has permissions preventing content from being read from there, findAllCompressionFiles will fail.

It looks plausible to just wrap this in a try-catch block and return on error.

EACCES: permission denied, scandir '[redacted]/client/wip/.cache'
    at Object.fs.readdirSync (fs.js:904:18)
    at findAllCompressionFiles ([redacted]/node_modules/express-static-gzip/index.js:132:24)
    at findAllCompressionFiles ([redacted]/node_modules/express-static-gzip/index.js:139:17)
    at expressStaticGzip ([redacted]/node_modules/express-static-gzip/index.js:30:9)

It's how it's called in serve-static. Renaming it allows easier migration from serve-static.

Hi!
I have the following:
app.use('/include_public', expressStaticGzip(__dirname + '/include_public/'));
I have (right now) only one file gzipped using the gzip -9 bootstrap.css command on linux
It does not get served as the css is missing form the page
can you offer any guidance?
Thanks!

It might be better to add express as a peer dependency, so it is always used the static middleware of the app's express installation.

Just want to make sure is it correct behavior ?

sources

Gzip is always on top in the sorted array of compression types.

I would like to use serveStatic setHeaders option with a function that checks for mime type, like so:
if (serveStatic.mime.lookup(path) === 'text/html') { ... }
It would be nice to expose serveStatic or mime so I don't need to install an other npm package.