Comments (16)
I stand corrected—I've discovered that it is in fact possible to set brotli compression level (1-11). A little obscure, but there you are:
const options: zlib.BrotliOptions = {
params: {
[zlib.constants.BROTLI_PARAM_QUALITY]: 7
}
}
For koa:
koa.use(koaCompress({
br: {
params: {
[zlib.constants.BROTLI_PARAM_QUALITY]: 7
}
}
}))
With a 12MB json payload typical for my app:
BROTLI
Level 1 Compression 88.7% Time 36ms
Level 2 Compression 89.3% Time 72ms
Level 3 Compression 89.6% Time 74ms
Level 4 Compression 91.1% Time 112ms
Level 5 Compression 92.0% Time 180ms
Level 6 Compression 92.1% Time 222ms
Level 7 Compression 92.2% Time 302ms
Level 8 Compression 92.3% Time 397ms
Level 9 Compression 92.4% Time 545ms
Level 10 Compression 93.1% Time 6441ms
Level 11 Compression 93.6% Time 33956ms
GZIP
Level 1 Compression 87.4% Time 75ms
Level 2 Compression 87.8% Time 79ms
Level 3 Compression 88.2% Time 91ms
Level 4 Compression 89.1% Time 115ms
Level 5 Compression 89.4% Time 135ms
Level 6 Compression 89.9% Time 169ms
Level 7 Compression 90.0% Time 215ms
Level 8 Compression 90.2% Time 505ms
Level 9 Compression 90.3% Time 557ms
So this makes things a lot more interesting. For the kind of json data I work with, there seems to be a large sweet spot where brotli has the edge: both faster and smaller file size!
You may need to run some experiments with your own data to tune the compression level (1-11 for brotli, 1-9 for gzip).
from compress.
I noticed this as well, running in node 14 and using streams. Enabling brotli compression causes a 3+ second delay before transfer of bytes begins on requests that experience <20ms of latency when using gzip.
This suggested to me that it was buffering the entire file into memory before sending it. While this may be the case, it is clear this would be a change in node's zlib, as this library seems very sensible and treats the different encoders the same.
Given the disastrous performance here, it may be best to change the default such that brotli is disabled.
from compress.
The brotli implementation in NodeJS does not seem to allow the compression level to be specified (1-11). This is unfortunate. Given the time taken to compress, it appears that the default level is high (11?).
Some experiments with a 3MB csv file on 2.9GHz i7:
NodeJS.zlib.brotliCompress
Level - Compression 81% Speed 5924ms
NodeJS.zlib.gzip
Level 1 Compression 67% Speed 45ms
Level 2 Compression 69% Speed 48ms
Level 3 Compression 70% Speed 55ms
Level 4 Compression 71% Speed 67ms
Level 5 Compression 72% Speed 90ms
Level 6 Compression 73% Speed 113ms
Level 7 Compression 73% Speed 131ms
Level 8 Compression 73% Speed 164ms
Level 9 Compression 73% Speed 169ms
https://github.com/foliojs/brotli.js
I assume these times cannot be directly compare to above (JS vs NodeJS.native?).
Level 1 Compression 71% Speed 159ms
Level 2 Compression 72% Speed 142ms
Level 3 Compression 73% Speed 126ms
Level 4 Compression 74% Speed 174ms
Level 5 Compression 76% Speed 216ms
Level 6 Compression 77% Speed 256ms
Level 7 Compression 77% Speed 373ms
Level 8 Compression 78% Speed 428ms
Level 9 Compression 78% Speed 548ms
Level 10 Compression 81% Speed 5801ms
Level 11 Compression 81% Speed 13457ms
I conclude that brotli is a poor choice for runtime compression.
On the other hand, it is ideal for build-time compression!
Further Reading:
https://blogs.akamai.com/2016/02/understanding-brotlis-potential.html
from compress.
I noticed this as well, running in node 14 and using streams. Enabling brotli compression causes a 3+ second delay before transfer of bytes begins on requests that experience <20ms of latency when using gzip.
This suggested to me that it was buffering the entire file into memory before sending it. While this may be the case, it is clear this would be a change in node's zlib, as this library seems very sensible and treats the different encoders the same.
Given the disastrous performance here, it may be best to change the default such that brotli is disabled.
+1, please change the default back to gzip.
According to https://www.opencpu.org/posts/brotli-benchmarks/, brotli has an edge of <25% in compression ratio, but the compression speed is 70 times slower than gzip. Definitely not sensible to choose brotli for on-the-fly compression.
from compress.
I think koa-compress can use brotli by default so long as it sets the compression level to something suitable for doing realtime compression.
NodeJS default is Brotli L11, which is great for offline compression, but woeful for realtime compression.
Since koa-compress is all about realtime compression, it must not use the NodeJS default.
I suggest Brotli L5. From my experiments (above), this works out 3x faster then default GZIP L9 compression. Of course your mileage will vary.
from compress.
Before Upgrade:
After:
from compress.
I just tested compress({ br: false })
with 5.0.1
and it's fast. So it appears to be something about it trying to use Brotli on Node 10 that's making it 10-20 times slower…
from compress.
So maybe koa-compress should use a lower default compression level?
from compress.
iirc one isn't set, but i'll take a PR to have a lower default
from compress.
Brotli L5 seems like the more reasonable default.
from compress.
I've set for L4 after my own tests. L5 still seemed a bit sluggish, with little gain, as you can see in the example above.
from compress.
got burned by this today where our app that normally takes 2s to download was taking 30-40s. setting brotli default level to something like 3 seems like a sane choice.
from compress.
Given the disagreements on the desirable brotli level, and brotli only gives a marginal improvement over gzip (~3%), can we switch back the default to gzip which has been time-tested and works for everybody?
from compress.
a default low level makes sense
from compress.
Any progress..?
from compress.
Related: #121
from compress.
Related Issues (20)
- remove jest --forceExit
- tests: remove callbacks HOT 1
- Brotli used, even if not in Accept-Encoding header (v4.0.0) HOT 4
- Not using the correct encoding HOT 2
- If no Accept-Encoding header is sent, koa-compress may compress the response HOT 9
- Can this be used without koa? HOT 1
- koa-compress not compressing response body. HOT 1
- 4.x seems to be significantly slower (not related to brotli!) HOT 4
- Expand documentation HOT 12
- Brotli Defaults HOT 13
- "flush" as it is used in tests HOT 1
- Q: `options.defaultEncoding` default `idenity`, typo? HOT 1
- Error [ERR_HTTP2_HEADERS_SENT]: Cannot set headers after they are sent to the client HOT 1
- Content-Length Not Set HOT 2
- 5.1 did not set brotli default quality to 4 correctly HOT 1
- An in-range update of eslint-plugin-flowtype is breaking the build 🚨 HOT 7
- An in-range update of supertest is breaking the build 🚨 HOT 4
- Check that response hasn't been sent already HOT 2
- consider adding `response.originalLength` property HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from compress.