Cannot complete ACME challenge

[ArturFortunato]

I'm running Caddy to manage wildcard certificates using route53 as a DNS provider.
This is running inside an AWS ECS task, where the task role has all the permissions on Route53 for all resources.

I keep getting this error when libdns tries to write a temporary record:

{
"level":"error",
"ts":1711561234.8623207,
"logger":"tls.obtain",
"msg":"could not get certificate from issuer",
"identifier":"*.mysite.dev",
"issuer":"acme-v02.api.letsencrypt.org-directory",
"error":"[*.mysite.dev] solving challenges: presenting for challenge: adding temporary record for zone \"mysite.dev.\": not found, ResolveEndpointV2 (order=https://acme-v02.api.letsencrypt.org/acme/order/XXXXXXX/XXXXXX) (ca=https://acme-v02.api.letsencrypt.org/directory)"
}

Seems the error comes from here, which calls https://github.com/libdns/route53/blob/master/provider.go#L42.

I see a trailing dot in the domain, but seems like this is not a problem.
Is this a bug or am I'm doing something wrong?

[elihunter]

I think I'm experiencing a similar issue trying to use the dynamic dns caddy plugin with route53. It looks like it may be related to this.

aws/aws-sdk-go-v2#2370 (comment)

[winterqt]

I made a fork that applies the fix from the linked comment above. Given a recent version of xcaddy (v0.4.2 or newer), you can build a version of Caddy with the fix very easily:

$ xcaddy build --with github.com/caddy-dns/route53 --replace github.com/libdns/route53=github.com/winterqt/libdns-route53@upgrade-aws-sdk

[kdevan]

I was getting this exact same error without using a wildcard domain, just a plain individual domain. I tried winterqt's fix and it is now working. Looks like there is already PRs for this and they just need to me merged.

Edit: Just tested with wildcard and that's working with this fix as well.

[vinczedani]

Would this issue be solved by merging the open PRs from dependabot? I think yes, based on the commit on @winterqt fork
Would be nice to get this solved in this repo, so ppl could use cady 2.8+ without replacing this module

Are there any blockers to upgrade the aws deps?

[forest-code42]

I made a fork that applies the fix from the linked comment above. Given a recent version of xcaddy (v0.4.2 or newer), you can build a version of Caddy with the fix very easily:

$ xcaddy build --with github.com/caddy-dns/route53 --replace github.com/libdns/route53=github.com/winterqt/libdns-route53@upgrade-aws-sdk

I tried this and it appears to be failing silently -- it jumps directly from

trying to solve challenge

to

DNS problem: NXDOMAIN looking up TXT for _acme-challenge.<my-domain> - check that a DNS record exists for this domain

Where in the past it would fail with

...trying to solve challenge...
...cleaning up solver...
...no memory of presenting a DNS record for \"_acme-challenge.<my-domain>\" (usually OK if presenting also failed)
solving challenges: presenting for challenge: adding temporary record for zone \"<my-zone>.\": not found, ResolveEndpointV2

I checked CloudTrail and I saw that it would create the TXT record, then 1 or 2 seconds later it would delete it.

[kdevan]

@forest-code42 I ran into that as well.

The solution here is what worked for me: caddy-dns/route53#43

The maintainer mentions the wait_for_propagation fix here: caddy-dns/route53#42 (comment)

[forest-code42]

Oh interesting. I thought wait_for_propagation would not work because if you look here its not parsed out of the config:

https://github.com/caddy-dns/route53/blob/master/route53.go#L56-L101

I missed your comment and I went on my own adventure to fork it and fix it myself. I got it working.

These changes were just to point to the forked module: caddy-dns/route53@master...forest-code42:route53:forest

These changes fixed the issue: winterqt/libdns-route53@master...forest-code42:libdns-route53:master

• always WaitForPropagation
• Treat timeout error (when waiting) as not a blocking error

I built it with

GOOS=linux GOARCH=amd64 ~/go/bin/xcaddy build --with github.com/forest-code42/route53@5f7726361f1b66e7bebdaa2397b42d5e9d0728a3

[nebez]

Thanks @forest-code42 ! This indeed fixes the issue I've been experiencing in 2.8.4, as well as the same issue I was facing with the winterqt/libdns-route53 fork.

For those who use a simple Dockerfile to manage Caddy, the below should work (replacing GOARCH with your arch of course)

FROM caddy:2.8.4-builder AS builder

ENV GOOS=linux
ENV GOARCH=arm64
RUN xcaddy build --with github.com/forest-code42/route53@5f7726361f1b66e7bebdaa2397b42d5e9d0728a3
FROM caddy:2.8.4

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

docker run --cap-add=NET_ADMIN -p 80:80 -p 443:443 -p 443:443/udp \
    -v $PWD/Caddyfile:/etc/caddy/Caddyfile \
    -v caddy_data:/data \
    -v caddy_config:/config \
    <above-docker-image>

[baowei]

Maintainer please update ASAP, Thanks. @aymanbagabas

[yami-no-ryuu]

@forest-code42 Can you please make a PR?
And huge thanks!
This issue was bonkers

[aymanbagabas]

Oh interesting. I thought wait_for_propagation would not work because if you look here its not parsed out of the config:

https://github.com/caddy-dns/route53/blob/master/route53.go#L56-L101

This should be fixed in caddy-dns/route53#50. It was only affecting Caddyfiles, JSON config are fine. EDIT: released in caddy-dns/[email protected]

@baowei @yami-no-ryuu @forest-code42 You could use wait_for_propagation and max_wait_dur to make caddy wait until the records are propagated and insync. The default values for wait_for_propagation and max_wait_dur are false and 60 seconds, respectively.

As you know, this module is also used for other things than ACME challenges, and making it specific to ACME challenges is not the goal here.