The `want` command won’t work if LE limit is reached for *any* domains about acmetool

[hlandau]

This appears to be another case of hlandau/acme.t#4
Can you count the certificates for belfalas.eu in /var/lib/acme/certs? There should only be one, but if it's this bug it might be another.

I've been unable to reproduce this, so if you could come up with a series of steps that causes the same certificate to be requested multiple times (e.g. using the staging server, which has very high rate limits), that would be really useful. Maybe pass --xlog.severity=debug while you're doing that.

[Flink]

Yes I only have one certificate per domain in /var/lib/acme/certs. And I didn’t see this earlier but the certificates are correctly generated but with this error I thought it wasn’t the case.
Also I’m writing a plugin for Dokku and with this error the script halts instead of completing correctly.

[hlandau]

I think the live service currently has a limit of one certificate per domain, so that doesn't necessarily mean that acmetool isn't erroneously trying to acquire more. Replicating this using the staging server might still be useful, in which case the presence of multiple certificates would be evidentiary.

[Flink]

At the moment you can request 5 certificates per week for a given domain (with or without subdomains). I often request the root domain with a subdomain, it was working without problem before reaching the limit.

[hlandau]

I don't think LE cares about domains vs. subdomains; presumably it just limits the number of certificates per account and per hostname?

[Flink]

If I understood correctly, it limits certificates requests per domain. If you request only belfalas.eu then it counts as one request but then you can request belfalas.eu www.belfalas.eu blog.belfalas.eu it will counts as another request. When you reach 5 requests on this domain you’ll have to wait one week.

[hlandau]

Not sure what the issue is here, unless acmetool is requesting certificates it doesn't need as in the case of the previous unconfirmed bug. This is an error from the server.

[Flink]

Well it seems indeed that acmetool is requesting something about belfalas.eu (previous existing certificates I think) since there is a reconcile in the error. This error happened as I was requesting another domain unrelated to belfalas.eu. I requested it also with the official client, no error. And since the certificates are still created with acmetool (as I was saying previously), it seems to be really a little bug (want shouldn’t call reconcile or something like that maybe?)

[hlandau]

The point of reconcile is to be idempotent. It shouldn't request certificates it already has.

Please create a new state directory using the staging server and try to reproduce this. It seems like it's trying to request certificates it already has, which is a bug. If you can get multiple certificates for the same name in the certs directory, this is a bug.

[Flink]

Well acmetool seems to not behave exactly the same way when using the staging server. I have errors but on validation and it proposes me to use DNS record for verification (never had this with production). But it generated all the requested certificates anyway. I generated 4 certificates for 4 subdomains of belfalas.eu with the root domain for each of them. It generated 4 certificates as expected.
Also the limits aren’t the same on the staging server.

[hlandau]

That is most confounding. You did ensure to use a fresh state directory, correct? Otherwise you'd now have a state directory mixing live and staging certificates.

It only asks for DNS verification if the normal challenges fail. It's essentially an admission of failure.

If acmetool is working correctly, subsequent executions of reconcile should be a no-op.

Hmm...

[Flink]

Yes yes I’m using another state directory, I didn’t want to have a problem with real certificates :)

[hlandau]

v0.0.18 adds more debug logging which illustrates the decision processes of the reconcile command. Please run it on your live state with --xlog.severity=debug. Builds will be available in ~10m.

[Flink]

Ok I think I found the problem:

20151209171031 [DEBUG] acme.storage: certificate Certificate(e353w5gbbvgxaau5g5tmbv4paco3edzu5nme7qlts6rxcsso6ovq) cannot satisfy Target(paste.belfalas.eu,belfalas.eu;;0) because required hostname "paste.belfalas.eu" is not listed on it:
 []string{"blog.belfalas.eu", "belfalas.eu"}
20151209171031 [DEBUG] acme.storage: certificate Certificate(ljtzuht63n7b6dhcyykvi7bamstlryp6ahntewjs3wjtxtoppj6a) cannot satisfy Target(paste.belfalas.eu,belfalas.eu;;0) because required hostname "paste.belfalas.eu" is not listed on it:
 []string{"cozy.gatitac.eu", "gatitac.eu"}
20151209171031 [DEBUG] acme.storage: certificate Certificate(mj3mdjvrnd5iuif7mnwog5e5ve3d4vffwmwko5ob7wr6um3z2asa) cannot satisfy Target(paste.belfalas.eu,belfalas.eu;;0) because required hostname "paste.belfalas.eu" is not listed on it:
 []string{"rss.belfalas.eu", "belfalas.eu"}
20151209171031 [DEBUG] acme.storage: certificate Certificate(onqpgbu4qyanhomnjfv3jnzv4kdbpfsemon7crfcvvxnxxg4tbna) cannot satisfy Target(paste.belfalas.eu,belfalas.eu;;0) because required hostname "paste.belfalas.eu" is not listed on it:
 []string{"cozy.belfalas.eu"}
20151209171031 [DEBUG] acme.storage: certificate Certificate(t3wacgfosf5e3gkz4q7gnlu5yyqkcn3kz3qfjrv7kdkshg2uclkq) cannot satisfy Target(paste.belfalas.eu,belfalas.eu;;0) because required hostname "paste.belfalas.eu" is not listed on it:
 []string{"webmail.belfalas.eu", "belfalas.eu"}
20151209171031 [DEBUG] acme.storage: certificate Certificate(xriaglxhzdf6wfdvp4zz36vysshwx2rvqrfba3prf2d4lrmgkgka) cannot satisfy Target(paste.belfalas.eu,belfalas.eu;;0) because required hostname "paste.belfalas.eu" is not listed on it:
 []string{"issues.spicatto.fr", "spicatto.fr"}
20151209171031 [DEBUG] acme.storage: certificate Certificate(wtumawfemdfseyuixqxhrngwum6vrx6qzv5l4aiwd45teqlfqrtq) cannot satisfy Target(paste.belfalas.eu,belfalas.eu;;0) because we do not have a key for it
20151209171031 [DEBUG] acme.storage: certificate Certificate(4sopf6bbtanpbggvlv6kkqa7i76en5m7pnweom27zm2k5owwaeva) cannot satisfy Target(paste.belfalas.eu,belfalas.eu;;0) because required hostname "paste.belfalas.eu" is not listed on it:
 []string{"baikal.belfalas.eu", "belfalas.eu"}
20151209171031 [DEBUG] acme.storage: best certificate satisfying Target(paste.belfalas.eu,belfalas.eu;;0) is <nil>, err=no certificate satisifes this target
20151209171031 [DEBUG] acme.storage: requesting certificate for target Target(paste.belfalas.eu,belfalas.eu;;0)
20151209171031 [DEBUG] acme.api: request: https://acme-v01.api.letsencrypt.org/directory
20151209171031 [DEBUG] acme.api: response: &{200 OK 200 HTTP/1.1 1 1 map[X-Frame-Options:[DENY] Strict-Transport-Security:[max-age=604800] Cache-Control:[max-age=0, no-cache, no-store] Date:[Wed, 09 Dec 2015 16:10:27 GMT] Connection:[kee
p-alive] Server:[nginx] Content-Type:[application/json] Expires:[Wed, 09 Dec 2015 16:10:27 GMT] Pragma:[no-cache] Content-Length:[263] Replay-Nonce:[2psncEqEY3BdcRamDw4jjjXMdRx5iy7qFvJXRkGGgzQ]] 0xc8202f0440 263 [] false map[] 0xc8203966
20 0xc8203a62c0} <nil>
20151209171031 [DEBUG] acme.api: request: https://acme-v01.api.letsencrypt.org/acme/new-reg
20151209171031 [DEBUG] acme.api: response: &{409 Conflict 409 HTTP/1.1 1 1 map[Server:[nginx] Content-Type:[application/problem+json] Content-Length:[94] Location:[https://acme-v01.api.letsencrypt.org/acme/reg/55582] Cache-Control:[max-a
ge=0, no-cache, no-store] Replay-Nonce:[ERPEM-1zGRyyfKAuhJoHkHJVfnGLOlxgMcE-tEvfOhg] Expires:[Wed, 09 Dec 2015 16:10:27 GMT] Pragma:[no-cache] Date:[Wed, 09 Dec 2015 16:10:27 GMT]] 0xc8202cebc0 94 [] true map[] 0xc820556380 0xc8203a62c0}
 <nil>
20151209171031 [DEBUG] acme.api: request: https://acme-v01.api.letsencrypt.org/acme/reg/55582
20151209171031 [DEBUG] acme.api: response: &{202 Accepted 202 HTTP/1.1 1 1 map[Cache-Control:[max-age=0, no-cache, no-store] Server:[nginx] Replay-Nonce:[FRrYA4-w15fL-9HFY8G_f5JGsCzZUiHiApSRvZXKX8A] Expires:[Wed, 09 Dec 2015 16:10:27 GMT
] Pragma:[no-cache] Date:[Wed, 09 Dec 2015 16:10:27 GMT] Connection:[keep-alive] Content-Type:[application/json] Content-Length:[578] Link:[<https://acme-v01.api.letsencrypt.org/acme/new-authz>;rel="next" <https://letsencrypt.org/documen
ts/LE-SA-v1.0.1-July-27-2015.pdf>;rel="terms-of-service"]] 0xc8204c4080 578 [] false map[] 0xc820587500 0xc8202a0370} <nil>
20151209171032 [DEBUG] acme.storage: requesting certificate for Target(paste.belfalas.eu,belfalas.eu;;0)
20151209171032 [DEBUG] acme.api: request: https://acme-v01.api.letsencrypt.org/acme/new-cert
20151209171032 [DEBUG] acme.api: response: &{429 Unknown 429 HTTP/1.1 1 1 map[Cache-Control:[max-age=0, no-cache, no-store] Date:[Wed, 09 Dec 2015 16:10:28 GMT] Server:[nginx] Content-Type:[application/problem+json] Content-Length:[142] 
Replay-Nonce:[n9NZ4P6bjNma3qb4RZ0QgRio80GVAUZ54y657ow8mRc] Expires:[Wed, 09 Dec 2015 16:10:28 GMT] Pragma:[no-cache]] 0xc82041c100 142 [] true map[] 0xc820639b20 0xc8202a0370} <nil>
20151209171032 [ERROR] acme.storage: could not request certificate: HTTP error: 429 Unknown
map[Server:[nginx] Content-Type:[application/problem+json] Content-Length:[142] Replay-Nonce:[n9NZ4P6bjNma3qb4RZ0QgRio80GVAUZ54y657ow8mRc] Expires:[Wed, 09 Dec 2015 16:10:28 GMT] Pragma:[no-cache] Cache-Control:[max-age=0, no-cache, no-s
tore] Date:[Wed, 09 Dec 2015 16:10:28 GMT]]
{"type":"urn:acme:error:rateLimited","detail":"Error creating new cert :: Too many certificates already issued for: belfalas.eu","status":429}
20151209171032 [ERROR] acme.storage: failed to request certificate for target %vTarget(paste.belfalas.eu,belfalas.eu;;0): HTTP error: 429 Unknown

I have a certificate not generated for paste.belfalas.eu since I hit the limit of 5 certificates. And acmetool tries to generate it everytime.
Just tested another thing: if I request a certificate that can’t be validated (for whatever reason), the next want command will try to request it again and it will fail (seems to not generate new certificates here).

And so, removing the correct wanted certificate in the desired directory make things work again :)

[hlandau]

Yep, this is by design. acmetool will keep trying and trying until it gets to its desired state. :)

[Flink]

Yes but it breaks completely the tool when errors are present. So I don’t think you should close this issue. Those errors should be handled properly to avoid a completely broken installation, what do you think?

[hlandau]

Hm, not sure what you mean. Is the issue that failure to acquire a give certificate blocks the acquisition of other certificates?

[Flink]

It seems so. I made a test with a domain where I didn’t setup anything and so it couldn’t be validated. If I try to generate a certificate for another domain where everything is ok, acmetool stops on the non working domain.
The errors displayed even if everything happens as expected are misleading.

Thanks anyway for your time and your responsiveness :)

[hlandau]

Essentially there are two ways forward. Either acmetool is changed to collect a list of errors that occur and present them as one, rather than bailing on the first error, or it implements temporary self-signed certificates, or possibly both.

As I outlined in the state schema specification, one thing I had in mind was generating temporary self-signed certificates if a certificate couldn't be obtained, under the premise that webservers will generally be happier with getting any certificate, even if it is invalid. Whereas a missing certificate may prevent a webserver from starting. This is #37, and I think I'll now consider it rather higher priority...

[hlandau]

v0.0.20 now tries all targets, then reports any errors that occur in a list at the end of the reconciliation process. Closing this.

[Flink]

ok nice :)