ss098 / certmagic-s3 Goto Github PK

Please reference issue: caddyserver/caddy#4780

The plugin should implement the Stringer interface, which would let it control how the struct is serialized as a string with %v.

Caddy needs to convert the struct to a string to use as an ID as a unique key for the storage plugin. The plugin can control what that ID is by being a Stringer.

Admittedly, this wasn't well documented, so that's partly our-bad.

Either way, we can't recommend using the S3 plugin, because it doesn't properly implement the locking mechanisms required by the storage interface.

Originally posted by @francislavoie in caddyserver/caddy#4780 (comment)

Hi there!

ss098, may I ask you to publish your package in https://caddyserver.com/download? That would allow us to download it from the package manager.

I think that do that is really easy, based on the community forum you should:

1. Create an account in https://caddyserver.com/account/create
2. Send your package using https://caddyserver.com/account/

Thats it and will be really helpful!

Thanks so much for your work!

Hello,

It seems like in mid 2021 you guys rewrote the package to remove the Redis dependency, is that correct?
I'm reading the current code and it appears that the implementation for the Lock and Unlock method that previously used Redis are now empty:

func (s3 S3) Lock(ctx context.Context, key string) error {
	return nil
}

func (s3 S3) Unlock(ctx context.Context, key string) error {
	return nil
}

I was wondering if the lock is not needed anymore in the current implementation and if the current implementation supports multiple caddy instances running in parallel accessing the same s3 storage space

Hi.

Working with custom docker images based on Caddy 2.3.0, 2.4.0, 2.4.1 with current certmagic-s3 and format-encoder modules

1. Problem

At first start when I got no certs at all and bucket folder is empty I'm getting an error that is shown in a full log section below.
Second problem - current error does not stop/kill docker container and system think that everything is fine there. As a result systemd did not throw any on-failure notifications.

2. Environment I'm working with

Got an AWS cluster with next set:

• Caddy node (Ubuntu 18.04.5 LTS)
• Application node (Ubuntu 18.04.5 LTS)
• Redis
• s3 bucket

Redis and S3 bucket has correct configuration and accessible from node where Caddy is working.
Caddy and App working as a docker containers started by systemd service

a. Command i use to start caddy

/usr/bin/docker run --rm --name='proxy' \
--mount type=bind,source=/etc/caddy_config.json,destination=/caddy_config.json \
--publish 80:80 \
--publish 443:443 \
artemius/caddy:2.4.1 caddy run --config /caddy_config.json

b. Caddy Dockerfile:

FROM caddy:2.4.1-builder AS builder
RUN xcaddy build --output ./caddy \
    --with github.com/ss098/certmagic-s3 \
    --with github.com/caddyserver/format-encoder

FROM caddy:2.4.1
COPY --from=builder /usr/bin/caddy /usr/bin/caddy

c. My complete JSON config:

{
  "apps": {
    "tls": {
      "certificates": {
        "automate": ["wp-test.dev-art.domain.com"]
      }
    },
    "http": {
      "servers": {
        "dev-art": {
          "listen": [":443"],
          "routes": [
            {
              "match": [{"host": ["wp-test.dev-art.domain.com"]}],
    "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [{"dial": "<APPLICATION_NODE_IP>:3000"}]
                }
              ]
            }
          ]
        }
      }
    }
  },
  "storage": {
    "module": "s3",
    "host": "s3.amazonaws.com",
    "bucket": "test_bucket",
    "access_key": "<AWS_ACCESS_KEY>",
    "secret_key": "<AWS_SECRET_KEY>",
    "prefix": "test_folder",
    "redis_address": "<AWS_REDIS_IP>:<AWS_REDIS_PORT>",
    "redis_password": "",
    "redis_db": 0
  }
}

3. Full log output:

{"level":"info","ts":1622292835.266702,"msg":"using provided configuration","config_file":"/caddy_config.json","config_adapter":""}
2021/05/29 12:53:55 [INFO] Redirecting sink to: stdout
2021/05/29 12:53:55 [INFO] Redirected sink to here (stdout)
{"level":"info","ts":1622292835.2699473,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":1622292835.271092,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"dev-art","https_port":443}
{"level":"info","ts":1622292835.2714522,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"dev-art"}
{"level":"info","ts":1622292835.2722864,"logger":"caddy.storage.s3","msg":"Load: test_folder/certificates/acme-v02.api.letsencrypt.org-directory/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.crt"}
{"level":"info","ts":1622292835.2732973,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002fda40"}
{"level":"info","ts":1622292835.9622266,"logger":"caddy.storage.s3","msg":"Load: test_folder/certificates/acme.zerossl.com-v2-dv90/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.crt"}
{"level":"info","ts":1622292836.0014627,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["wp-test.dev-art.domain.com"]}
{"level":"info","ts":1622292836.0025342,"logger":"caddy.storage.s3","msg":"Load: test_folder/certificates/acme-v02.api.letsencrypt.org-directory/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.crt"}
{"level":"info","ts":1622292836.00381,"logger":"caddy.storage.s3","msg":"Exists: test_folder/certificates/acme-v02.api.letsencrypt.org-directory/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.crt"}
{"level":"info","ts":1622292836.0050566,"logger":"tls","msg":"cleaning storage unit","description":"&{0xc00072f6e0 0xc00037d970 s3.amazonaws.com test_bucket <AWS_ACCESS_KEY> <AWS_SECRET_KEY> dev-art Redis<<AWS_REDIS_IP>:<AWS_REDIS_PORT> db:0> 0xc00073f230 map[] <AWS_REDIS_IP>:<AWS_REDIS_PORT>  0}"}
{"level":"info","ts":1622292836.0395818,"logger":"caddy.storage.s3","msg":"Load: test_folder/certificates/acme.zerossl.com-v2-dv90/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.crt"}
{"level":"info","ts":1622292836.0770361,"logger":"caddy.storage.s3","msg":"Exists: test_folder/certificates/acme.zerossl.com-v2-dv90/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.crt"}
{"level":"info","ts":1622292836.14719,"logger":"caddy.storage.s3","msg":"Exists: test_folder/certificates/acme-v02.api.letsencrypt.org-directory/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.crt"}
{"level":"info","ts":1622292836.1494465,"logger":"caddy.storage.s3","msg":"Store: test_folder/rw_test_4873102611796815485, 10240 bytes"}
{"level":"info","ts":1622292836.152547,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","ts":1622292836.153608,"msg":"serving initial configuration"}
{"level":"info","ts":1622292836.156261,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1622292836.1844337,"logger":"caddy.storage.s3","msg":"Exists: test_folder/certificates/acme.zerossl.com-v2-dv90/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.crt"}
{"level":"info","ts":1622292836.2113404,"logger":"caddy.storage.s3","msg":"Load: test_folder/rw_test_4873102611796815485"}
{"level":"info","ts":1622292836.2222416,"logger":"caddy.storage.s3","msg":"Store: test_folder/rw_test_3162212477631280723, 10240 bytes"}
{"level":"info","ts":1622292836.2527869,"logger":"caddy.storage.s3","msg":"Delete: test_folder/rw_test_4873102611796815485"}
{"level":"info","ts":1622292836.2739127,"logger":"caddy.storage.s3","msg":"Load: test_folder/rw_test_3162212477631280723"}
{"level":"info","ts":1622292836.2978468,"logger":"tls.obtain","msg":"acquiring lock","identifier":"wp-test.dev-art.domain.com"}
{"level":"info","ts":1622292836.2989197,"logger":"caddy.storage.s3","msg":"Lock: issue_cert_wp-test.dev-art.domain.com"}
{"level":"info","ts":1622292836.3187115,"logger":"caddy.storage.s3","msg":"Delete: test_folder/rw_test_3162212477631280723"}
{"level":"info","ts":1622292836.3603976,"logger":"tls.obtain","msg":"acquiring lock","identifier":"wp-test.dev-art.domain.com"}
{"level":"info","ts":1622292836.3625963,"logger":"caddy.storage.s3","msg":"Lock: issue_cert_wp-test.dev-art.domain.com"}
{"level":"info","ts":1622292836.3638575,"logger":"tls.obtain","msg":"lock acquired","identifier":"wp-test.dev-art.domain.com"}
{"level":"info","ts":1622292836.3648925,"logger":"caddy.storage.s3","msg":"Exists: test_folder/certificates/acme-v02.api.letsencrypt.org-directory/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.crt"}
{"level":"info","ts":1622292836.4058213,"logger":"caddy.storage.s3","msg":"Exists: test_folder/certificates/acme.zerossl.com-v2-dv90/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.crt"}
{"level":"error","ts":1622292836.424822,"logger":"caddy.storage.s3","msg":"Cannot lock key: issue_cert_wp-test.dev-art.domain.com"}
{"level":"error","ts":1622292836.425877,"logger":"tls","msg":"job failed","error":"wp-test.dev-art.domain.com: obtaining certificate: unable to acquire lock 'issue_cert_wp-test.dev-art.domain.com': redislock: not obtained"}
{"level":"info","ts":1622292836.4893322,"logger":"caddy.storage.s3","msg":"Load: test_folder/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json"}
{"level":"info","ts":1622292837.5782793,"logger":"caddy.storage.s3","msg":"Store: test_folder/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json, 141 bytes"}
{"level":"info","ts":1622292837.6305573,"logger":"caddy.storage.s3","msg":"Store: test_folder/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.key, 227 bytes"}
{"level":"info","ts":1622292837.6845737,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["wp-test.dev-art.domain.com"]}
{"level":"info","ts":1622292837.6846461,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["wp-test.dev-art.domain.com"]}
{"level":"info","ts":1622292838.1838276,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"wp-test.dev-art.domain.com","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"info","ts":1622292838.1849308,"logger":"caddy.storage.s3","msg":"Store: test_folder/acme/acme-v02.api.letsencrypt.org-directory/challenge_tokens/wp-test.dev-art.domain.com.json, 350 bytes"}
{"level":"info","ts":1622292838.5495484,"logger":"tls","msg":"served key authentication certificate","server_name":"wp-test.dev-art.domain.com","challenge":"tls-alpn-01","remote":"3.120.130.29:36250","distributed":false}
{"level":"info","ts":1622292838.7121632,"logger":"tls","msg":"served key authentication certificate","server_name":"wp-test.dev-art.domain.com","challenge":"tls-alpn-01","remote":"66.133.109.36:23904","distributed":false}
{"level":"info","ts":1622292838.7668262,"logger":"tls","msg":"served key authentication certificate","server_name":"wp-test.dev-art.domain.com","challenge":"tls-alpn-01","remote":"54.189.22.122:31250","distributed":false}
{"level":"info","ts":1622292838.8055172,"logger":"tls","msg":"served key authentication certificate","server_name":"wp-test.dev-art.domain.com","challenge":"tls-alpn-01","remote":"18.116.86.117:13532","distributed":false}
{"level":"info","ts":1622292839.3147287,"logger":"caddy.storage.s3","msg":"Delete: test_folder/acme/acme-v02.api.letsencrypt.org-directory/challenge_tokens/wp-test.dev-art.domain.com.json"}
{"level":"info","ts":1622292839.363895,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-v02.api.letsencrypt.org/acme/order/125255113/10033391100"}
{"level":"info","ts":1622292840.3172622,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-v02.api.letsencrypt.org/acme/cert/0306c525ab0978c5477ce58df8d3020bb67e"}
{"level":"info","ts":1622292840.320495,"logger":"caddy.storage.s3","msg":"Store: test_folder/certificates/acme-v02.api.letsencrypt.org-directory/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.crt, 5357 bytes"}
{"level":"info","ts":1622292840.372044,"logger":"caddy.storage.s3","msg":"Store: test_folder/certificates/acme-v02.api.letsencrypt.org-directory/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.key, 227 bytes"}
{"level":"info","ts":1622292840.4263725,"logger":"caddy.storage.s3","msg":"Store: test_folder/certificates/acme-v02.api.letsencrypt.org-directory/wp-test.dev-art.domain.com/wp-test.dev-art.domain.com.json, 168 bytes"}
{"level":"info","ts":1622292840.496127,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"wp-test.dev-art.domain.com"}
{"level":"info","ts":1622292840.4964314,"logger":"tls.obtain","msg":"releasing lock","identifier":"wp-test.dev-art.domain.com"}
{"level":"info","ts":1622292840.496462,"logger":"caddy.storage.s3","msg":"Release lock: issue_cert_wp-test.dev-art.domain.com"}
2021/05/29 12:54:00 panic: certificate worker: runtime error: invalid memory address or nil pointer dereference
goroutine 53 [running]:
github.com/caddyserver/certmagic.(*jobManager).worker.func1()
  github.com/caddyserver/[email protected]/async.go:58 +0xa5
panic(0x179c840, 0x2a18990)
  runtime/panic.go:965 +0x1b9
github.com/bsm/redislock.(*Lock).Release(0x0, 0xc000ad5000, 0x36)
  github.com/bsm/[email protected]/redislock.go:174 +0x40
github.com/ss098/certmagic-s3.(*S3).Unlock(0xc000106fa0, 0xc000885ad0, 0x28, 0xc000168380, 0x1)
  github.com/ss098/[email protected]/s3.go:115 +0x137
github.com/caddyserver/certmagic.releaseLock(0x1fab858, 0xc000106fa0, 0xc000885ad0, 0x28, 0x1, 0x1)
  github.com/caddyserver/[email protected]/storage.go:246 +0x49
github.com/caddyserver/certmagic.(*Config).obtainCert.func1(0xc000aabce0, 0xc000049f80, 0x1d, 0xc000107360, 0xc000885ad0, 0x28)
  github.com/caddyserver/[email protected]/config.go:448 +0x85
github.com/caddyserver/certmagic.(*Config).obtainCert(0xc000107360, 0x1f9c370, 0xc00053ea80, 0xc000049f80, 0x1d, 0xc000753600, 0x0, 0x0)
  github.com/caddyserver/[email protected]/config.go:532 +0x326
github.com/caddyserver/certmagic.(*Config).ObtainCert(0xc000107360, 0x1f9c370, 0xc00053ea80, 0xc000049f80, 0x1d, 0x100000000000000, 0x7ff88429bf40, 0x0)
  github.com/caddyserver/[email protected]/config.go:428 +0x135
github.com/caddyserver/certmagic.(*Config).manageOne.func1(0x0, 0x0)
  github.com/caddyserver/[email protected]/config.go:329 +0x95
github.com/caddyserver/certmagic.(*jobManager).worker(0x2a22720)
  github.com/caddyserver/[email protected]/async.go:73 +0x103
created by github.com/caddyserver/certmagic.(*jobManager).Submit
  github.com/caddyserver/[email protected]/async.go:50 +0x12d

Sorry, because of NDA, main domain was changed to domain.com, it was containing company name. Other parts/structure as it is

4. Tested with:

Caddy 2.3.0, 2.4.0, 2.4.1
Redis 4.0.10, 5.0.6

Hello.
Just ran into this module. Just wondering why not using the AWS GO SDK (github.com/aws/aws-sdk-go/service/s3) ?
Among other things, it will allow to pick the AWS Credentials directly using the SDK credentials chain.
Thanks :)

Would you mind adding tags going forward? Would make builds a bit more reproducible. :) Cheers!

Hi,

Can the module be updated to look for SSL files on local disk before checking S3. If the file exists locally, then s3 request is not needed. This significantly improves TTFB even if the certificates aren’t loaded in memory. Some body implemented something similar here. https://github.com/thekonqueror/certmagic-s3

Best regards,

Using xcaddy build master --with github.com/ss098/certmagic-s3 I get the error below (seems to be also happening for redistls storage module). Building against 2.4.6 everything seems fine.

2022/04/18 10:19:02 [INFO] exec (timeout=0s): /usr/bin/go get -d -v
2022/04/18 10:19:05 [INFO] Build environment ready
2022/04/18 10:19:05 [INFO] Building Caddy
2022/04/18 10:19:05 [INFO] exec (timeout=0s): /usr/bin/go mod tidy
2022/04/18 10:19:05 [INFO] exec (timeout=0s): /usr/bin/go build -o /root/caddy/caddy -ldflags -w -s -trimpath
# github.com/ss098/certmagic-s3
/root/go/pkg/mod/github.com/ss098/[email protected]/s3.go:127:2: cannot use s3 (type S3) as type certmagic.Storage in return argument:
        S3 does not implement certmagic.Storage (wrong type for Delete method)
                have Delete(string) error
                want Delete(context.Context, string) error
2022/04/18 10:19:09 [INFO] Cleaning up temporary folder: /tmp/buildenv_2022-04-18-1018.2928695872
2022/04/18 10:19:09 [FATAL] exit status 2
root@apt-cacher:~/caddy# go version
go version go1.17.8 linux/amd64

One of my caddy instances crashed with this error log:

Jul 14 05:12:59 venus caddy[833]: fatal error: concurrent map writes
Jul 14 05:12:59 venus caddy[833]: goroutine 2341366 [running]:
Jul 14 05:12:59 venus caddy[833]: runtime.throw(0x1a597ec, 0x15)
Jul 14 05:12:59 venus caddy[833]: runtime/panic.go:1117 +0x72 fp=0xc002a176e8 sp=0xc002a176b8 pc=0x438652
Jul 14 05:12:59 venus caddy[833]: runtime.mapassign_faststr(0x17ecde0, 0xc00038c390, 0xc000740cf0, 0x30, 0x30)
Jul 14 05:12:59 venus caddy[833]: runtime/map_faststr.go:211 +0x3f1 fp=0xc002a17750 sp=0xc002a176e8 pc=0x414911
Jul 14 05:12:59 venus caddy[833]: github.com/ss098/certmagic-s3.(*S3).Lock(0xc0000b2630, 0x1d500d0, 0xc0002f55c0, 0xc000740cf0, 0x30, 0x2, 0x30)
Jul 14 05:12:59 venus caddy[833]: github.com/ss098/[email protected]/s3.go:108 +0x1f6 fp=0xc002a177f8 sp=0xc002a17750 pc=0x16a1716
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.acquireLock(0x1d500d0, 0xc0002f55c0, 0x1d5f478, 0xc0000b2630, 0xc000740cf0, 0x30, 0x30, 0x27fc840)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/[email protected]/storage.go:236 +0x62 fp=0xc002a17850 sp=0xc002a177f8 pc=0x8b1e22
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*Config).renewCert(0xc0004de5a0, 0x1d500d0, 0xc0002f55c0, 0xc001612f90, 0x25, 0x0, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/[email protected]/config.go:662 +0x25e fp=0xc002a17a80 sp=0xc002a17850 pc=0x8919fe
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*Config).RenewCertAsync(…)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/[email protected]/config.go:639
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*Cache).queueRenewalTask.func1(0xc003585018, 0xc00086f180)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/[email protected]/maintain.go:254 +0x1ba fp=0xc002a17ea0 sp=0xc002a17a80 pc=0x8b80da
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*jobManager).worker(0x27ffb40)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/[email protected]/async.go:73 +0x103 fp=0xc002a17fd8 sp=0xc002a17ea0 pc=0x888083
Jul 14 05:12:59 venus caddy[833]: runtime.goexit()
Jul 14 05:12:59 venus caddy[833]: runtime/asm_amd64.s:1371 +0x1 fp=0xc002a17fe0 sp=0xc002a17fd8 pc=0x472841
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/certmagic.(*jobManager).Submit
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/[email protected]/async.go:50 +0x12d
Jul 14 05:12:59 venus caddy[833]: goroutine 1 [select (no cases), 1278 minutes]:
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2/cmd.cmdRun(0xc0001057a0, 0x0, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/[email protected]/cmd/commandfuncs.go:276 +0x1395
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2/cmd.Main()
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/[email protected]/cmd/main.go:85 +0x25b
Jul 14 05:12:59 venus caddy[833]: main.main()
Jul 14 05:12:59 venus caddy[833]: caddy/main.go:12 +0x25
Jul 14 05:12:59 venus caddy[833]: goroutine 17 [select, 1280 minutes]:
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*RingBufferRateLimiter).permit(0xc00009c190)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/[email protected]/ratelimiter.go:216 +0xb2
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*RingBufferRateLimiter).loop(0xc00009c190)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/[email protected]/ratelimiter.go:89 +0xa8
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/certmagic.NewRateLimiter
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/[email protected]/ratelimiter.go:45 +0x148
Jul 14 05:12:59 venus caddy[833]: goroutine 13 [chan receive, 1280 minutes]:
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2.trapSignalsCrossPlatform.func1()
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/[email protected]/sigtrap.go:42 +0x129
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/caddy/v2.trapSignalsCrossPlatform
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/[email protected]/sigtrap.go:37 +0x35
Jul 14 05:12:59 venus caddy[833]: goroutine 14 [chan receive, 1280 minutes]:
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2.trapSignalsPosix.func1()
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/[email protected]/sigtrap_posix.go:34 +0x139
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/caddy/v2.trapSignalsPosix
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/[email protected]/sigtrap_posix.go:30 +0x35
Jul 14 05:12:59 venus caddy[833]: goroutine 19 [syscall, 1280 minutes]:
Jul 14 05:12:59 venus caddy[833]: os/signal.signal_recv(0x0)
Jul 14 05:12:59 venus caddy[833]: runtime/sigqueue.go:168 +0xa5
Jul 14 05:12:59 venus caddy[833]: os/signal.loop()
Jul 14 05:12:59 venus caddy[833]: os/signal/signal_unix.go:23 +0x25
Jul 14 05:12:59 venus caddy[833]: created by os/signal.Notify.func1.1
Jul 14 05:12:59 venus caddy[833]: os/signal/signal.go:151 +0x45
Jul 14 05:12:59 venus caddy[833]: goroutine 67 [IO wait, 1280 minutes]:
Jul 14 05:12:59 venus caddy[833]: internal/poll.runtime_pollWait(0x7fe15eace7d8, 0x72, 0x0)
Jul 14 05:12:59 venus caddy[833]: runtime/netpoll.go:222 +0x55
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).wait(0xc000470c98, 0x72, 0x0, 0x0, 0x1a12a0f)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:87 +0x45
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).waitRead(…)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:92
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*FD).Accept(0xc000470c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_unix.go:401 +0x212
Jul 14 05:12:59 venus caddy[833]: net.(*netFD).accept(0xc000470c80, 0x50, 0xc0000f1cd0, 0x40de72)
Jul 14 05:12:59 venus caddy[833]: net/fd_unix.go:172 +0x45
Jul 14 05:12:59 venus caddy[833]: net.(*TCPListener).accept(0xc0006c4138, 0xc000484820, 0x7fe15e7d3768, 0x30)
Jul 14 05:12:59 venus caddy[833]: net/tcpsock_posix.go:139 +0x32
Jul 14 05:12:59 venus caddy[833]: net.(*TCPListener).Accept(0xc0006c4138, 0x282c1a0, 0xc000382d80, 0xc0000f1d00, 0x40e398)
Jul 14 05:12:59 venus caddy[833]: net/tcpsock.go:261 +0x65
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2.(*fakeCloseListener).Accept(0xc0006c0680, 0x1902040, 0xc0006c2d50, 0x17dab00, 0x27f59c0)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/[email protected]/listeners.go:122 +0x43
Jul 14 05:12:59 venus caddy[833]: net/http.(*Server).Serve(0xc00036a2a0, 0x1d48f90, 0xc0006c0680, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: net/http/server.go:2981 +0x285
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2.replaceLocalAdminServer.func2(0x1d48f90, 0xc0006c0680, 0xc0004847d0)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/[email protected]/admin.go:343 +0x5c
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/caddy/v2.replaceLocalAdminServer
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/[email protected]/admin.go:342 +0x455
Jul 14 05:12:59 venus caddy[833]: goroutine 68 [select]:
Jul 14 05:12:59 venus caddy[833]: github.com/go-redis/redis/v8/internal/pool.(*ConnPool).reaper(0xc0004de0a0, 0xdf8475800)
Jul 14 05:12:59 venus caddy[833]: github.com/go-redis/redis/[email protected]/internal/pool/pool.go:455 +0xd7
Jul 14 05:12:59 venus caddy[833]: created by github.com/go-redis/redis/v8/internal/pool.NewConnPool
Jul 14 05:12:59 venus caddy[833]: github.com/go-redis/redis/[email protected]/internal/pool/pool.go:110 +0x1fc
Jul 14 05:12:59 venus caddy[833]: goroutine 69 [select]:
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*Cache).maintainAssets(0xc000188380, 0x0)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/[email protected]/maintain.go:71 +0x1ab
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/certmagic.NewCache
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/[email protected]/cache.go:121 +0x172
Jul 14 05:12:59 venus caddy[833]: goroutine 77 [IO wait, 1 minutes]:
Jul 14 05:12:59 venus caddy[833]: internal/poll.runtime_pollWait(0x7fe15eace608, 0x72, 0x0)
Jul 14 05:12:59 venus caddy[833]: runtime/netpoll.go:222 +0x55
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).wait(0xc00016b098, 0x72, 0x0, 0x0, 0x1a12a0f)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:87 +0x45
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).waitRead(…)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:92
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*FD).Accept(0xc00016b080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_unix.go:401 +0x212
Jul 14 05:12:59 venus caddy[833]: net.(*netFD).accept(0xc00016b080, 0x203000, 0x203000, 0x0)
Jul 14 05:12:59 venus caddy[833]: net/fd_unix.go:172 +0x45
similar lines removed for brevity
Jul 14 05:12:59 venus caddy[833]: goroutine 1161462 [IO wait, 527 minutes]:
Jul 14 05:12:59 venus caddy[833]: internal/poll.runtime_pollWait(0x7fe15e5a5fc8, 0x72, 0xffffffffffffffff)
Jul 14 05:12:59 venus caddy[833]: runtime/netpoll.go:222 +0x55
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).wait(0xc004545998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:87 +0x45
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).waitRead(…)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:92
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*FD).Read(0xc004545980, 0xc000f72000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_unix.go:166 +0x1d5
Jul 14 05:13:00 venus caddy[833]: net.(*netFD).Read(0xc004545980, 0xc000f72000, 0x1000, 0x1000, 0x101, 0xc001324a80, 0xc002a9e6a0)
Jul 14 05:13:00 venus caddy[833]: net/fd_posix.go:55 +0x4f
Jul 14 05:13:00 venus caddy[833]: net.(*conn).Read(0xc0009a80c0, 0xc000f72000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: net/net.go:183 +0x91
Jul 14 05:13:00 venus caddy[833]: net/http.(*connReader).Read(0xc003e60510, 0xc000f72000, 0x1000, 0x1000, 0x1ba, 0x4149a8, 0xc002a9e7c8)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:800 +0x1b9
Jul 14 05:13:00 venus caddy[833]: bufio.(*Reader).fill(0xc003b04e40)
Jul 14 05:13:00 venus caddy[833]: bufio/bufio.go:101 +0x108
Jul 14 05:13:00 venus caddy[833]: bufio.(*Reader).ReadSlice(0xc003b04e40, 0x418e0a, 0x7fe15e8292d8, 0x0, 0xc003e60090, 0x1a124ac, 0x100000dd2)
Jul 14 05:13:00 venus caddy[833]: bufio/bufio.go:360 +0x3d
Jul 14 05:13:00 venus caddy[833]: bufio.(*Reader).ReadLine(0xc003b04e40, 0x203000, 0x203000, 0x756ea1fb2446cc, 0x203001, 0x203001, 0x203001)
Jul 14 05:13:00 venus caddy[833]: bufio/bufio.go:389 +0x34
Jul 14 05:13:00 venus caddy[833]: net/textproto.(*Reader).readLineSlice(0xc004c4c330, 0xc0034c9800, 0x15, 0x7fe185793108, 0x18, 0xc005e74b28)
Jul 14 05:13:00 venus caddy[833]: net/textproto/reader.go:57 +0xd6
Jul 14 05:13:00 venus caddy[833]: net/textproto.(*Reader).ReadLine(…)
Jul 14 05:13:00 venus caddy[833]: net/textproto/reader.go:38
Jul 14 05:13:00 venus caddy[833]: net/http.readRequest(0xc003b04e40, 0x0, 0xc0034c9800, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: net/http/request.go:1027 +0xaa
Jul 14 05:13:00 venus caddy[833]: net/http.(*conn).readRequest(0xc000add860, 0x1d500d0, 0xc001324a40, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:986 +0x19d
Jul 14 05:13:00 venus caddy[833]: net/http.(*conn).serve(0xc000add860, 0x1d50178, 0xc001324a40)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:1878 +0x705
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*Server).Serve
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:3013 +0x39b
Jul 14 05:13:00 venus caddy[833]: goroutine 2492 [select, 1279 minutes]:
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7.Client.GetObject.func1(0xc0015e5920, 0xc0015e5980, 0xc0015e59e0, 0xc00158b920, 0xc0019a8dc0, 0xc0019ae210, 0x1d50108, 0xc00003e050, 0xc0001fb950, 0xa, …)
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/[email protected]/api-get-object.go:74 +0x148
Jul 14 05:13:00 venus caddy[833]: created by github.com/minio/minio-go/v7.Client.GetObject
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/[email protected]/api-get-object.go:65 +0x397
similar lines removed for brevity
Jul 14 05:13:00 venus caddy[833]: goroutine 2340058 [select, 1 minutes]:
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2serverConn).serve(0xc001e80780)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4428 +0x579
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2Server).ServeConn(0xc0000c9d80, 0x1d5d4d8, 0xc00171c000, 0xc000ec3b68)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4038 +0x725
Jul 14 05:13:00 venus caddy[833]: net/http.http2ConfigureServer.func1(0xc00036a1c0, 0xc00171c000, 0x1d33800, 0xc002be7540)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:3864 +0xf0
Jul 14 05:13:00 venus caddy[833]: net/http.(*conn).serve(0xc001e7a5a0, 0x1d50178, 0xc0003cacc0)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:1861 +0x1535
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*Server).Serve
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:3013 +0x39b
Jul 14 05:13:00 venus caddy[833]: goroutine 2956 [select, 1279 minutes]:
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7.Client.GetObject.func1(0xc000a988a0, 0xc000a98900, 0xc000a98a80, 0xc0017fde40, 0xc001149140, 0xc0019aedc0, 0x1d50108, 0xc00003e050, 0xc0001fb950, 0xa, …)
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/[email protected]/api-get-object.go:74 +0x148
similar lines removed for brevity
Jul 14 05:13:00 venus caddy[833]: goroutine 2340626 [IO wait, 1 minutes]:
Jul 14 05:13:00 venus caddy[833]: internal/poll.runtime_pollWait(0x7fe15e5f3d78, 0x72, 0xffffffffffffffff)
Jul 14 05:13:00 venus caddy[833]: runtime/netpoll.go:222 +0x55
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*pollDesc).wait(0xc0038fc798, 0x72, 0x200, 0x205, 0xffffffffffffffff)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_poll_runtime.go:87 +0x45
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*pollDesc).waitRead(…)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_poll_runtime.go:92
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*FD).Read(0xc0038fc780, 0xc002c72900, 0x205, 0x205, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_unix.go:166 +0x1d5
Jul 14 05:13:00 venus caddy[833]: net.(*netFD).Read(0xc0038fc780, 0xc002c72900, 0x205, 0x205, 0x200, 0xc002c72900, 0x5)
Jul 14 05:13:00 venus caddy[833]: net/fd_posix.go:55 +0x4f
Jul 14 05:13:00 venus caddy[833]: net.(*conn).Read(0xc0009a80e0, 0xc002c72900, 0x205, 0x205, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: net/net.go:183 +0x91
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*atLeastReader).Read(0xc0006ce288, 0xc002c72900, 0x205, 0x205, 0x200, 0xc0020a0000, 0x0)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:776 +0x63
Jul 14 05:13:00 venus caddy[833]: bytes.(*Buffer).ReadFrom(0xc000b09778, 0x1d2d7c0, 0xc0006ce288, 0x40b9c5, 0x1818a00, 0x198d260)
Jul 14 05:13:00 venus caddy[833]: bytes/buffer.go:204 +0xbe
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readFromUntil(0xc000b09500, 0x1d31080, 0xc0009a80e0, 0x5, 0xc0009a80e0, 0x11)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:798 +0xf3
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readRecordOrCCS(0xc000b09500, 0x0, 0x0, 0xc00323dd38)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:605 +0x115
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readRecord(…)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:573
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).Read(0xc000b09500, 0xc0015da1f8, 0x9, 0x9, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:1276 +0x165
Jul 14 05:13:00 venus caddy[833]: io.ReadAtLeast(0x7fe15e6c89a0, 0xc000b09500, 0xc0015da1f8, 0x9, 0x9, 0x9, 0x101006dccc5, 0x0, 0x2)
Jul 14 05:13:00 venus caddy[833]: io/io.go:328 +0x87
Jul 14 05:13:00 venus caddy[833]: io.ReadFull(…)
Jul 14 05:13:00 venus caddy[833]: io/io.go:347
Jul 14 05:13:00 venus caddy[833]: net/http.http2readFrameHeader(0xc0015da1f8, 0x9, 0x9, 0x7fe15e6c89a0, 0xc000b09500, 0x0, 0xc000000000, 0xc0016daf00, 0xc00323df70)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:1477 +0x89
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2Framer).ReadFrame(0xc0015da1c0, 0xc00323df38, 0x0, 0x0, 0x2)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:1735 +0xa5
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2serverConn).readFrames(0xc001e80900)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4314 +0xa9
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*http2serverConn).serve
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4420 +0x366
Jul 14 05:13:00 venus caddy[833]: goroutine 3073 [select, 1279 minutes]:
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7.Client.GetObject.func1(0xc001832720, 0xc001832780, 0xc0018327e0, 0xc00172da90, 0xc001610c40, 0xc00189c0b0, 0x1d50108, 0xc00003e050, 0xc0001fb950, 0xa, …)
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/[email protected]/api-get-object.go:74 +0x148
similar lines removed for brevity
Jul 14 05:13:00 venus caddy[833]: goroutine 2341476 [select]:
Jul 14 05:13:00 venus caddy[833]: net/http.(*persistConn).writeLoop(0xc00066b440)
Jul 14 05:13:00 venus caddy[833]: net/http/transport.go:2382 +0xf7
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*Transport).dialConn
Jul 14 05:13:00 venus caddy[833]: net/http/transport.go:1744 +0xc9c
Jul 14 05:13:00 venus caddy[833]: goroutine 3219 [select, 1278 minutes]:
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7.Client.GetObject.func1(0xc00144fbc0, 0xc00144fc20, 0xc00144fc80, 0xc0015a8c60, 0xc0018a1b80, 0xc00144c210, 0x1d50108, 0xc00003e050, 0xc0001fb950, 0xa, …)
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/[email protected]/api-get-object.go:74 +0x148
similar lines removed for brevity
Jul 14 05:13:00 venus caddy[833]: net/http.(*conn).serve(0xc002493400, 0x1d50178, 0xc001bf1880)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:1977 +0xa47
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*Server).Serve
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:3013 +0x39b
Jul 14 05:13:00 venus caddy[833]: io/io.go:328 +0x87
Jul 14 05:13:00 venus caddy[833]: io.ReadFull(…)
Jul 14 05:13:00 venus caddy[833]: io/io.go:347
Jul 14 05:13:00 venus caddy[833]: net/http.http2readFrameHeader(0xc001f21df8, 0x9, 0x9, 0x7fe15e6c89a0, 0xc002f54380, 0x0, 0xc000000000, 0xc0050d4300, 0xc000cdaf70)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:1477 +0x89
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2Framer).ReadFrame(0xc001f21dc0, 0xc000cdaf38, 0x0, 0x0, 0x2)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:1735 +0xa5
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2serverConn).readFrames(0xc003d28780)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4314 +0xa9
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*http2serverConn).serve
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4420 +0x366
Jul 14 05:13:00 venus caddy[833]: goroutine 2341491 [select]:
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7.Client.GetObject.func1(0xc003bf2300, 0xc003bf23c0, 0xc003bf2420, 0xc003df8360, 0xc001e7d340, 0xc001e54210, 0x1d50108, 0xc00003e050, 0xc0001fb950, 0xa, …)
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/[email protected]/api-get-object.go:74 +0x148
Jul 14 05:13:00 venus caddy[833]: created by github.com/minio/minio-go/v7.Client.GetObject
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/[email protected]/api-get-object.go:65 +0x397
Jul 14 05:13:00 venus caddy[833]: goroutine 2339504 [IO wait, 2 minutes]:
Jul 14 05:13:00 venus caddy[833]: internal/poll.runtime_pollWait(0x7fe15e305b80, 0x72, 0xffffffffffffffff)
Jul 14 05:13:00 venus caddy[833]: runtime/netpoll.go:222 +0x55
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*pollDesc).wait(0xc0043c4998, 0x72, 0x200, 0x205, 0xffffffffffffffff)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_poll_runtime.go:87 +0x45
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*pollDesc).waitRead(…)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_poll_runtime.go:92
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*FD).Read(0xc0043c4980, 0xc000453b00, 0x205, 0x205, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_unix.go:166 +0x1d5
Jul 14 05:13:00 venus caddy[833]: net.(*netFD).Read(0xc0043c4980, 0xc000453b00, 0x205, 0x205, 0x0, 0x3ed, 0xc0032296d8)
Jul 14 05:13:00 venus caddy[833]: net/fd_posix.go:55 +0x4f
Jul 14 05:13:00 venus caddy[833]: net.(*conn).Read(0xc001392110, 0xc000453b00, 0x205, 0x205, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: net/net.go:183 +0x91
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*atLeastReader).Read(0xc00137ddd0, 0xc000453b00, 0x205, 0x205, 0x80, 0xc00059c000, 0x0)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:776 +0x63
Jul 14 05:13:00 venus caddy[833]: bytes.(*Buffer).ReadFrom(0xc002490278, 0x1d2d7c0, 0xc00137ddd0, 0x40b9c5, 0x1818a00, 0x198d260)
Jul 14 05:13:00 venus caddy[833]: bytes/buffer.go:204 +0xbe
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readFromUntil(0xc002490000, 0x1d31080, 0xc001392110, 0x5, 0xc001392110, 0x0)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:798 +0xf3
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readRecordOrCCS(0xc002490000, 0x0, 0x0, 0x8000000000000000)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:605 +0x115
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readRecord(…)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:573
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).Read(0xc002490000, 0xc00228e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:1276 +0x165
Jul 14 05:13:00 venus caddy[833]: net/http.(*connReader).Read(0xc00131dda0, 0xc00228e000, 0x1000, 0x1000, 0xc00077fb90, 0x64d971, 0xc001392110)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:800 +0x1b9
Jul 14 05:13:00 venus caddy[833]: bufio.(*Reader).fill(0xc002af10e0)
Jul 14 05:13:00 venus caddy[833]: bufio/bufio.go:101 +0x108
Jul 14 05:13:00 venus caddy[833]: bufio.(*Reader).Peek(0xc002af10e0, 0x4, 0x45ff5bf68c45, 0x282b400, 0x0, 0x0, 0x282b400)
Jul 14 05:13:00 venus caddy[833]: bufio/bufio.go:139 +0x4f
Jul 14 05:13:00 venus caddy[833]: net/http.(*conn).serve(0xc002493400, 0x1d50178, 0xc001bf1880)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:1977 +0xa47
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*Server).Serve
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:3013 +0x39b
Jul 14 05:13:00 venus systemd[1]: caddy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 14 05:13:00 venus systemd[1]: caddy.service: Failed with result ‘exit-code’.

systemd automatically restarted Caddy and it was fine, but this is the 3rd time it's happened over the last couple of weeks.

Note this didn't start happening till Caddy was upgraded to v 2.4.3