nicolasazrak / caddy-cache Goto Github PK
View Code? Open in Web Editor NEWCaching middleware for caddy
License: Mozilla Public License 2.0
Caching middleware for caddy
License: Mozilla Public License 2.0
Great module!!!
Is a cache control planned to delete full cache or single cached pages? For example with a HTTP request or an admin api?
The logic to support Surrogate-Control
would basically the same as Cache-Control
, but the header is specifically meant for reverse proxy <-> origin server controls (Cache-Control is meant for end browsers)
See: https://www.w3.org/TR/edge-arch/
Often used by CDNs:
https://docs.fastly.com/guides/tutorials/cache-control-tutorial
https://www.nuevocloud.com/documentation/getting-started/cache-headers-cache-control-surrogate-control-and-expires
One caveat: It's not clear to me if it'd the cache plugins job to strip the Surrogate-Control
header from the response, or if that should be handled by the proxy
plugin.
Would it possible to serve only static cache content to the client and refresh the cache content from the backend webserver by an given interval?
Exclude from caching by path to reach dynamic cms backends.
I try to serve a static copy of a dynamic website because of security problems with some cms out there... Serve the page as static copy should be absolute secure and the backend path would be saved by caddy basicauth.
Thanks for you work. It has been very helpful 👍
Is there a way I can limit the amount of memory used for caching. I have seen caddy run out of memory and not accept subsequent connections after a few hours when caching is enabled.
I would not expect it to use so much memory given that the amount of cache-able content we have is limited. Can you please tell me if there is anything wrong/missing with the configuration
https://domain:443, https://domain:443 {
bind 0.0.0.0
proxy / https://origin:443 {
transparent
keepalive 5
}
cache {
match_path /static
path /tmp/caddy_cache
default_max_age 0
}
tls ./keys/cert ./keys/key
errors /var/log/caddy_error.log {
rotate_size 100
}
}
Following graph shows memory usage climbing up when caching is enabled. With no caching and only as a reverse proxy memory usage seems stable.
Any help would be greatly appreciated. Thanks!
Hello!
Are there any plans to submit this plugin to the Caddy website? It would be pretty useful!
Thanks 😄
Could you explain more about how to build this with Caddy? I've looked at the docs and I'm still not sure how to begin.
Static content will not see great benefits.
Would it be possible to add an option to store the cache in memory instead of in disk? This could presumably improve performance so that you then would see performance improvements, even for static content.
Caddy is losing caching information after server restart, then it takes time for downloading files for the first time. Is it possible to make caching persistent
As the title suggests, the cache directive in the Caddyfile is not recognized after building caddy with this plugin via xcaddy.
Try to use this kind of config
example.com {
gzip
cache
push
proxy / 1.1.1.1
}
http2 push dosn't work
With
example.com {
gzip
push
proxy / 1.1.1.1
}
http2 push will work
I have a JavaScript file that is 2 megabytes in size and I canceled the request when I didn't fully receive the file when I first requested it.
Requesting this file again will always be pending. Looking at the source website access record discovery request did not return to the source, just waiting. There will be no timeout errors.
Title is pretty self explanatory. When using the cache
directive with relative path
makes it in the root where caddy was started, not relative to the root
directive as expected.
Without this, it makes snippets using cache
useless
src/github.com/mholt/caddy/caddyhttp/httpserver/responsewriterwrapper.go:41 - *cache.StreamedRecorder is not a closeNotifier
StreamedRecorder is not closeNotifier error .
I'm trying out the cache plugin on my personal site and am seeing some images being downloaded with a Content-Length of 1, for example: https://matt.life/resources/images/lake.jpg
This is downloaded with a content of Ü
and a content-range header of just the last byte. You can see the rest of them like this on https://matt.life.
Hi~I would like to know the progress of “ Add a configuration to not use query params in cache key” . Thanks you very much :)
https://github.com/nicolasazrak/caddy-cache/blob/master/cache.go#L25 uses Header.Set to add individual header values, but uses https://godoc.org/pkg/net/http/#Header.Set, which replaces all values with this one - this is wrong.
I'd use
m := w.Header()
for k, values := range response.HeaderMap {
m[k] = append(m[k][:0], v...)
}
When used in conjuction of the cors plugin, it returns the cors headers from the cached version instead of the currrent request.
I have no idea how caddy plugins work, but I imagine this plugin should take effect first and hand off the cached value to the cors plugin so it can update the headers accordingly.
Caddy's import path (and Go module name) has changed from
github.com/mholt/caddy
to
github.com/caddyserver/caddy
Unfortunately, Go modules are not yet mature enough to handle a change like this (see https://golang.org/issue/26904 - "haven't implemented that part yet" but high on priority list for Go 1.14) which caught me off-guard. Using Go module's replace
feature didn't act the way I expected, either. Caddy now fails to build with plugins until they update their import paths.
I've hacked a fix into the build server, so downloading Caddy with your plugin from our website should continue working without any changes on your part, for now. However, please take a moment and update your import paths, and do a new deploy on the website, because the workaround involves ignoring module checksums and performing a delicate recursive search-and-replace.
I'm terribly sorry about this. I did a number of tests and dry-runs to ensure the change would be smooth, but apparently some unknown combination of GOPATH, Go modules' lack of maturity, and other hidden variables in the system or environment must have covered up something I missed.
This bash script should make it easy (run it from your project's top-level directory):
find . -name '*.go' | while read -r f; do
sed -i.bak 's/\/mholt\/caddy/\/caddyserver\/caddy/g' $f && rm $f.bak
done
We use this script in the build server as part of the temporary workaround.
Let me know if you have any questions! Sorry again for the inconvenience.
In Caddy 1.0.3 I've been seeing "superfluous response.WriteHeader
" and "wrote more than the declared Content-Length
" warnings / errors output in the logs.
I managed to track it down to when the http.cache
plugin is used.
I've already noted this issue on the forum:
https://caddy.community/t/caching-plugin-causing-superfluous-response-writeheader-and-wrote-more-than-the-declared-content-length/6262/
The version I tested with was downloaded from here: https://caddyserver.com/download/windows/amd64?plugins=http.cache&license=personal&telemetry=off
http://localhost {
proxy / http://127.0.0.1
cache
log stdout
}
http:// {
status 418 /
log stdout
}
λ caddy -log stdout
Activating privacy features... done.
2019/09/27 18:57:35 [INFO][cache:0xc000196230] Started certificate maintenance routine
Serving HTTP on port 80
http://localhost
http://
2019/09/27 18:57:35 [INFO] Serving http://localhost
2019/09/27 18:57:35 [INFO] Serving http://
127.0.0.1 - - [27/Sep/2019:18:57:38 +0100] "GET / HTTP/1.1" 418 17
2019/09/27 18:57:38 http: superfluous response.WriteHeader call from github.com/caddyserver/caddy/caddyhttp/httpserver.(*ResponseRecorder).WriteHeader (recorder.go:61)
2019/09/27 18:57:38 [Error] failed to write body: http: wrote more than the declared Content-Length
::1 - - [27/Sep/2019:18:57:38 +0100] "GET / HTTP/1.1" 418 17
2019/09/27 18:58:23 [INFO] SIGINT: Shutting down
2019/09/27 18:58:23 [INFO][cache:0xc000196230] Stopped certificate maintenance routine
Removing the cache
line from the Caddyfile results in the errors / warnings not displaying.
at times I've seen erroneous additional text at the bottom of responses, mostly during error responses. e.g.
503 Service Unavailable
503 Service Unavailable
Instead of just:
503 Service Unavailable
This has been seen in successful responses (200) too (at the bottom of HTML responses) - but not captured at the time.
Whilst this could be two bugs - it's likely related.
Had a crash on Caddy instance on Ubuntu receiving traffic.
This "out of memory" error seems to be "http.cache" related as the instance hasn't crashed until today - this crash is after having caching enabled for ~24-48 hours with a restart in the middle.
Crash Log:
https://gist.github.com/DeanMarkTaylor/000b02b4fb4ff7ede8ac3afa6c766bfd
caddyserver/caddy#2137 Described here
Currently, it appears that even if I use the "advanced usage" approach and specify a particular path to be cached, other paths can automatically be cached as well.
In my case, I would like to instruct caddy-cache only to cache the /api
path, and leave everything else out of the cache, regardless of what headers it may include. However, even if I specify match_path /api
in the cache
directive of my Caddyfile, I see other files from outside that path appearing in the cache.
I'm not sure if this behavior is by design or not. But the documentation seems to suggest that only the "basic usage" approach should result in this automatic behavior, and the presence of any "advanced usage" should restrict caching to just the path the user specified.
Hi there,
I have looked through the issues and can't find anything about this. I might be missing something here. I have enabled the cache
module in Caddy. I can see the cached files in tmp when I do ls -lah /tmp
. However, after two minutes, the file is removed. It's not always consistent though. Sometimes it's more than a few minutes.
What is the logic behind this? Let's say I use to cache images from imgproxy
. I don't want to fetch the image every time. For my use case, I would want to cache it for over an hour. How do I do that?
Thanks!
Is there a placeholder that can be used to log out the cache status same as status_header?
Thanks!
I’m trying to reverse proxy and cache a HLS/M3U8 live streaming. In HLS live streaming, m3u8 playlist file contains video “chunk” file name with TS extension which is exactly 10 seconds long. So I’m trying to cache those TS files. There is no reason to download same TS files for every new user in my reverse proxy server.
example.com/live/index.m3u8
example.com/live/feed865/1000.ts
example.com/live/feed346/1010.ts
example.com/live/feed424/1020.ts
…
and my caddyfile
proxy /tv example.com/live {
without /tv
header_downstream Access-Control-Allow-Origin "*"
header_downstream -Server
}
cache {
match_path /tv/
match_header Content-Type application/octet-stream # only for ts file
status_header X-Cache-Status
default_max_age 1m # in live streaming, cached file don't have any value longer than 10 seconds.
path C:\caddy\tmp\caddy-cache
}
I’m having two issues using this configs.
MyServer # I added custom name using header
Caddy # I guess it’s adding from cache plugin
Accept-Ranges: bytes
Cache-Control: max-age=2592000
Content-Length: 43391
Content-Type: image/jpeg
Date: Tue, 28 May 2019 04:31:02 GMT
Etag: “qqn4spr”
Last-Modified: Sun, 19 May 2019 06:23:52 GMT
Server: Caddy
Server: Maddy
X-Cache-Status: hit
Using the cache
directive in the same host as the redir
directive breaks the functionality of conditions in the latter. The conditions are not properly evaluated and depending if a statement or a negated statement is used, all or none of the requests are redirected, regardless if the condition matches or not.
In this first example, it will redirect no requests:
example.com {
cache
redir {
if {uri} is /1
/2
}
}
In this second example, it will redirect all requests:
example.com {
cache
redir {
if {uri} not /1
/2
}
}
Removing the cache
directive from these examples restores the desired behavior.
It would be useful if Caddy could treat different cached paths differently, like how Caddy's header
directive works. However, looking at setup.go
it appears that DefaultMaxAge
and other options are global to the whole middleware.
If I try to specify multiple cache blocks in my Caddyfile, Caddy doesn't complain, but I don't see the behavior I would expect:
cache {
match_path /foo
default_max_age 1m
status_header First-Block-Header
}
cache {
match_path /bar
match_path /baz
default_max_age 5m
status_header Second-Block-Header
}
In this case I'm not sure whether Caddy is actually instantiating multiple instances of caddy-cache, or if one of the blocks is just getting ignored, but I see all responses using a single same status header.
I suppose a revised syntax would look more like the syntax of header
:
cache /foo {
default_max_age 1m
status_header First-Block-Header
}
cache /bar /baz {
default_max_age 5m
status_header Second-Block-Header
}
Can a new release be cut so that the cache_key
can be used?
as follows:
......
header / -Server
......
The response header will not show "server: Caddy"
if I enable cache, like:
......
header / -Server
cache {
.......
}
......
'server: Caddy' will show again.
Given the following header: The plugin should cache for 30 seconds (s-maxage) and can then serve the cached variant to subsequent requests for another 90 seconds (6 hours if the background server returns an error) while asynchronously refreshing the cached variant.
Cache-Control: max-age=60, s-maxage=30, stale-while-revalidate=90, stale-while-error=21600
Hi,
I am getting this error when trying to get the caddy-cache plugin
go get github.com/nicolasazrak/caddy-cache
../../../../nicolasazrak/caddy-cache/setup.go:171: c.IncrNest undefined (type *caddy.Controller has no field or method IncrNest)
Which versions of Caddy servers are supported at this time?
Dimitar
This plugin’s cache is a shared cache. It should prefer the value specified in the s-maxage (shared cache= value over the max-age (client cache), and only use the max-age as a fallback if s-maxage isn’t present.
Example (should cache for 60 seconds):
Cache-Control: max-age=900, s-maxage: 60
I’m seeing a bunch of http: multiple response.WriteHeader calls output in the logs.
I tried removing the cache config sections consisting of the following:
cache {
status_header X-Cache-Status
}
And ran for over 24 hours without an occurrence of the log:
http: multiple response.WriteHeader calls
Put the config sections back in and saw an occurrence of the offending log entry after just 5 minutes and then several after that within 2-3 minutes.
Related thread in forum including further details:
https://caddy.community/t/http-multiple-response-writeheader-calls/4667/1
Now the situation is that 404 502 will cache
More information here https://caddy.community/t/rewrite-nginx-rewrite-block-to-caddy/3582/26
It seem not to be controlled.
https://www.example.cc/usr/themes/Single/css/kico.css
response headers:
accept-ranges: bytes
content-length: 27270
content-type: text/css; charset=utf-8
date: Wed, 31 Oct 2018 07:14:08 GMT
etag: "pg9yotl1i"
last-modified: Mon, 08 Oct 2018 10:05:17 GMT
server: Caddy //this header maybe upstream's or local caddy's
server: NaibaCDN //this header is setting by myself
status: 200
x-cache-status: hit
And this is my Caddyfile
www.example.cc {
tls [email protected]
header / Server NaibaCDN
cache
proxy / https://www.example.cc {
transparent
header_downstream -Server
}
}
I'm already set header_downstream
and http.header
why has two Server header?
Hi,
Don't you think that caddy-cache
's licence should be changed to anything more friendly?
Apache2 maybe, because mholt/caddy
has Apache2.
caddy-cache
is a great project, but I (and maybe many others) can't use it because of the licence. Most of the golang's projects and caddy
itself use non-*GPL and it's okay. Although LGPL is a less strict, in case of golang it has no sense because there is no dynamic linking.
(Please take this with a large grain of salt, as I'm very new to this area. This could be a total non-issue due to my own errors.)
Summary: It seems like caddy-cache is caching and serving successful (but empty) 304 responses to conditional requests using the If-None-Match
header. This results in the cache sometimes serving an empty page to some visitors.
Details: I think the sequence of events I see happening is:
GET index.html
200 (OK) [contents of index.html]
. and caddy-cache stores the contents of index.html.GET index.html
but includes the If-None-Match
and ETag
headers, so that the server can avoid re-sending the contents of index.html if it hasn't changed.304 (Not Modified) [empty response]
GET index.html
Again, I could have something wrong here, so please don't hesitate to question my theory.
In case it's important, here's the relevant portion of my Caddyfile:
cache {
match_path /api
}
header / Cache-Control "max-age=3600"
header /api Cache-Control "max-age=30"
My small website has an expensive API, so I'd like to cache its results for 30s. But I don't particularly need to cache the rest of my site. I've added the cache
directive in an attempt to selectively cache the /api
path, but since this appears to cache the rest of the site anyway (see #26) I put the extra header
directive in so at least I can control how long the rest gets cached. I don't know if this issue occurs with other configurations but I'm guessing it's likely.
I see that my use of header
means that all responses from Caddy will include a max-age, even the 304 responses, but I would have thought the cache would be smart enough to recognize a 304 response and avoid caching it.
I'd like to be able to selectively cache assets based on a header, such as Content-Type
. This would be useful for caching things like images, while still serving HTML directly.
i.e.,
cache {
match {
header Content-Type image/*
}
}
Cache is not appliable to websocket.
I just tested and it responsed with 502
08/Aug/2017:23:27:41 +0800 [ERROR 502 /ws] *cache.Response is not a hijacker
Maybe apply a filter to detect websocket and then skip it?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.