Comments (12)
Success!!!
from algernon.
I updated Algernon to be able to stream large files. I tried both with your test case and the regression test.
Two new flags were also added:
--timeout=N
for setting a timeout in seconds, when serving large files (but there is range support, so if a download times out, the client can continue where it left).
--largesize=N
for setting a threshold for when a file is too large to be read into memory (the default is 42 MiB).
Please test. :)
from algernon.
Thanks for testing. I will add a better test and fix this.
from algernon.
I will create a reproducible test case and then fix this.
from algernon.
This is a great project! I really love all the built-in features.
Unfortunately, this issue has re-emerged.
It looks like the server is completely loading the file into some kind of cache or a buffer before sending it to the client: https://github.com/xyproto/algernon/blob/master/engine/handlers.go#L369
Shouldn't all static files be streamed directly from the disk to the client? (That's the default behaviour for all of the other webservers I've used.)
Additionally, files that are already compressed (eg: zip, jpg, mp4, ogg, etc.) can be streamed directly to the client without gzipping.
Could we have a function that streams static files?
from algernon.
Same issue.
Strangely, the process also balloons up to many multiples of the size of the file it's serving. When I first start algernon, the process uses 293MB of memory. Then I load a 75MB video, and the process's memory usage grows to 771MB.
Is algernon is copying the memory block as it's being passed around between functions?
Would it be a large architectural change to pass a file handle as a response, and have that stream to the client?
from algernon.
Not compressing the file before checking if it would fit in the cache in a compressed state fixed this.
from algernon.
Thanks for reporting. I suspect that the streaming issues are in debug mode only. Could you please try the -x
flag and see if it streams correctly?
from algernon.
I am able to reproduce the issue.
from algernon.
The issue is that this line is trying to read in the entire file:
https://github.com/xyproto/datablock/blob/8c3914e5c4fe78accb948fad712c0080dfb5ded9/filecache.go#L437
I need to change the code to only read the file if:
- caching is enabled and it is under a certain size
- it's one of the supported data types that will need to be interpreted (like Lua scripts) and under a certain size
from algernon.
Added a fix to master
. Please test.
from algernon.
Just tested it:
$ ./algernon --httponly --nocache --server ~/bigfiles :8889
When I try to get a big file, this happens:
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< Last-Modified: Thu, 07 May 2015 08:36:32 GMT
< Server: Algernon 1.12.1
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Rate-Limit-Duration: 1
< X-Rate-Limit-Limit: 10.00
< X-Rate-Limit-Request-Forwarded-For:
< X-Rate-Limit-Request-Remote-Addr: 127.0.0.1:34934
< X-Xss-Protection: 1; mode=block
< Date: Thu, 24 Jan 2019 00:55:57 GMT
< Content-Length: 18
<
seeker can't seek
from algernon.
Related Issues (20)
- low ulimit on macOS HOT 1
- Lua tables that contains objects are not rendered with pongo2 HOT 13
- Authrefresh crashes randomly at startup HOT 2
- Command line arg --internal=filename ignores filename HOT 3
- Can Algernon be configured to run over SSL but with a self-signed cert
- ability to close the response stream early HOT 5
- Integrate pure-go Svelte compiler just like plenti HOT 7
- gopherlua doesn't support package.loadlib HOT 3
- MultiMarkdown CriticMarkup Support
- lua to wasm HOT 2
- .alg files are persisted to /tmp HOT 2
- sqlite support HOT 2
- examples for data validations
- Create more documentation and learning materials (like tutorial + reference) HOT 7
- Vhost handling HOT 1
- Automatic ACME / LetsEncrypt HOT 4
- Unable to serve files with embedded new line HOT 2
- Algernon flags precedence issue HOT 1
- Support LuaX HOT 1
- [tutorial] show how to embed audio files as <audio> tags HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from algernon.