System DNS resolver caching fails DNS-01 challenges about pebble

[shred]

If it is any help: I run integration tests against Pebble in my project, and my dns-01 tests succeed. So generally, it seems to work. However I am using a tiny and very simple DNS server, nothing sophisticated.

[munnerz]

Is that DNS server also authoritative for your hostname under test?

I'm using system default recursive nameservers during my tests, and updating a real cloudflare dns zone in order to perform validations. I suspect this is the issue, as I'll be using some random recursive DNS resolvers that are honouring the NXDOMAIN ttl (as the domain I am using is a randomly generated subdomain).

[shred]

I run Pebble in a docker container, and my test DNS server in a second docker container. The /etc/resolv.conf of Pebble's container points to that DNS server's IP, so it is the only way for Pebble to resolve domains. This way I can also run http-01 validations for fake domains like example.com.

The server only does a minimal job. It reacts to A and TXT queries, and sends the responses that I have previously set. No TTL, no recursions.

I only mention this so you know that Pebble's dns-01 validation is not generally broken, but I don't want to rule out that there might be an issue.

[cpu]

I'm using system default recursive nameservers during my tests, and updating a real cloudflare dns zone in order to perform validations. I suspect this is the issue, as I'll be using some random recursive DNS resolvers that are honouring the NXDOMAIN ttl (as the domain I am using is a randomly generated subdomain).

Yup, that's the issue :-) I think we should do a better job of documenting this Pebble gotcha.

Boulder and the Let's Encrypt prod/staging stack use an Unbound instance to do the heavy lifting for DNS. We run a configuration (basically identical to this) that sets a very low max TTL to avoid caching problems for those environments. Boulder's test environment uses a fake recursive resolver that returns fibs. In both cases Boulder uses miekg/dns to talk to the specifically configured resolver (The fake one or the Unbound instance).

Ideally Pebble could be changed to do similar: config would point Pebble's DNS requests to a fake or otherwise customized recursive DNS server. @shred and I chatted about that way back in Jul 2017 in #33. Unfortunately my conclusion at the time was that it would mean pulling in miekg/dns to Pebble and doing a lot more custom DNS code. Presently (as you noted) Pebble uses net.LookupTXT from the stdlib and Go uses the system DNS resolver unconditionally.

I think a solution like what @shred arrived at where you find a way to configure the system DNS for your integration tests unobtrusively with ✨ Container Magic ✨ is the best path forward. (edit: at least for the short term until there's time for more involved Pebble DNS rework).

[cpu]

I put out a PR to clarify some of Pebble's limitations, including this system DNS resolver "gotcha": #123

I'm going to close this issue for now since the problem is a known limitation with Pebble. I'll leave #33 open for tracking more intensive work to integrate more complex DNS handling.

Thanks!