[berlin] certbot renewal appears to be broken

Open

Details

4 participants

Attila Lendvai
Giovanni Biscuolo
Ludovic Courtès
Maxim Cournoyer

Owner: unassigned

Submitted by: Maxim Cournoyer

Severity: normal

Merged with

56678

Maxim Cournoyer wrote on 27 Mar 2023 23:05

Recipients:(name . bug-guix)(address . bug-guix@gnu.org)(name . guix-sysadmin)(address . guix-sysadmin@gnu.org)

Message-ID:87cz4tq501.fsf@gmail.com

Hi,

The TLS cert of https://disarchive.guix.gnu.org/expired today. Looking

at /var/log/mcron.log on Berlin, we see that the last certbot renew job

failed like so:

Toggle snippet (134 lines)2023-03-24 00:30:00 127768 certbot renew --webroot --webroot-path /var/www: running...
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Saving debug log to /var/log/letsencrypt/letsencrypt.log
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/bootstrappable.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/ci.guix.gnu.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/disarchive.guix.gnu.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:32:54 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for disarchive.guix.gnu.org
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   Domain: disarchive.guix.gnu.org
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://disarchive.guix.gnu.org/.well-known/acme-challenge/O1kK3tsJtH0r9RwvbCIFhHagJhBwewV3Ka0NPW86nAI: 404
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate disarchive.guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/dump.guix.gnu.org.conf
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/guix.gnu.org.conf
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:10 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for guix.gnu.org
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   Domain: guix.gnu.org
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 2a0c:e300::58: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/_PlXq5i2BRw23Ui1Yl4rLtyB2aSDnUNMZXurCWBwH-k: 404
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/guix.info.conf
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:19 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for guix.info and www.guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Domain: guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/O6y6aqSvLdjdS77MgaEhh7sN7Q75OQX3Jz69xnT4qnY: 404
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Domain: www.guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/lCioloihdJF6xwwTBg6cSNFjRearp4EBZBWcjkznrUE: 404
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate guix.info with error: Some challenges have failed.
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/issues.guix.gnu.org.conf
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/issues.guix.info.conf
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:26 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for issues.guix.info and 3 more domains
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   Domain: guix.info
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/Yv4KpoYC95LzGsM5IPTE68vf6lLfNHVK5kMUocSuDW0: 404
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate issues.guix.info with error: Some challenges have failed.
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/monitor.guix.gnu.org.conf
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for monitor.guix.gnu.org
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   Domain: monitor.guix.gnu.org
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   Type:   unauthorized
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   Detail: 141.80.181.40: Invalid response from https://monitor.guix.gnu.org/.well-known/acme-challenge/_wxH92e9QQag7TEYdqsA4-C-5pE5DnUd6pzMvQWzWNU: 400
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate monitor.guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/www.guixwl.org-0001.conf
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/www.guixwl.org.conf
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: The following certificates are not due for renewal yet:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/bootstrappable.org/fullchain.pem expires on 2023-05-14 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/ci.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/dump.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/issues.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/www.guixwl.org-0001/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/www.guixwl.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: All renewals failed. The following certificates could not be renewed:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/disarchive.guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/guix.info/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/issues.guix.info/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   /etc/letsencrypt/live/monitor.guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 5 renew failure(s), 0 parse failure(s)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: failed after 234.635s with: (misc-error #f unclean exit status ~S (1) #f)--8<---------------cut here---------------end--------------->8---

I removed the certbot file name prefix
(/gnu/store/jnp0166xw62dafd2zgxdmvjb6yq8ak32-certbot-1.28.0/bin/) in the
above output to improve readability.

-- 
Thanks,
Maxim

Maxim Cournoyer wrote on 29 Mar 2023 02:42

control message for bug #62491

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87355o75hc.fsf@gmail.com

merge 62491 56678

quit

Attila Lendvai wrote on 4 May 2023 16:37

(No Subject)

Recipients:(name . 62491@debbugs.gnu.org)(address . 62491@debbugs.gnu.org)(name . clement@lassieur.org)(address . clement@lassieur.org)

Message-ID:xUfl58WwIGDQakb2wFTlATboSCRB4-uR1eu3HS0G6Mo1IdzgYsOsA2D4YmBt_TgLWFrMlmJFi2a2yykmDNZJuUCRHoENEmnvrhWdSYC8DSA=@lendvai.name

i don't think this is the same issue as #56678.

or at least what i'm seeing on my server is that the wrong certbot cmd line is generated, which then results in saving the challenge at the wrong path.

this is the mcron that gets generated:

[...]/certbot certonly -n --agree-tos --webroot -w /srv/http/ --cert-name dwim.hu -d dwim.hu --email attila@lendvai.name

and this what worked when i fixed the -w arg:

[...]/certbot certonly -n --agree-tos --webroot -w /srv/http/dwim.hu --cert-name dwim.hu -d dwim.hu --email attila@lendvai.name

i.e. the -w parameter should point to the webroot of the virtual domain, but the guix config structure does not allow setting the webroot for each <certificate-configuration>, only at their parent, i.e. in the <certbot-configuration>.

this all seems to me as if the certbot service code was assuming that the certbot script will append the domain names (specified with -d) to the webroot path, but it does not.

from the certbot log (i.e. challenge is saved at the wrong path):

"Removing /srv/http/.well-known/acme-challenge/[hash]"

the relevant code is from 2018, so certbot's behavior may very well have changed since then:

https://git.savannah.gnu.org/cgit/guix.git/commit/gnu/services/certbot.scm?id=c3215d2f9d8fa4b890e3a41ceb4404b76a7c5c49

it seems to me that the webroot field should be moved down into <certificate-configuration>.

am i right? if so i may try to patch this up.

- attila

PGP: 5D5F 45C7 DFCD 0A39

• attila lendvai

• PGP: 963F 5D5F 45C7 DFCD 0A39

“State is the name of the coldest of all cold monsters. Coldly it lies; and this lie slips from its mouth: "I, the state, am the people."”

— Friedrich Nietzsche (1844–1900), 'Thus Spoke Zarathustra' (1885), http://j.mp/1k6pbwS

Giovanni Biscuolo wrote on 22 Nov 2023 18:37

bug#62491: [berlin] certbot renewal appears to be broken

Recipients:

Message-ID:87sf4x6653.fsf@xelera.eu

Hello Attila,

I'm starting using certbot on a new Guix System server of mine: I've not

much experience with this Guix service but I'm using certbot on other

machines so I hope I can help here.

Attila Lendvai <attila@lendvai.name> writes:

Toggle quote (2 lines)

> i don't think this is the same issue as #56678.

AFAIU actually #56678 is (was?) caused by a duplicate certbot account:

Toggle snippet (7 lines)

Please choose an account

Choices: ['guix-hpc.bordeaux.inria.fr@2017-09-04T08:51:13Z (48c5)',

'localhost@2016-12-03T21:08:38Z (00bc)']

on bayfront, probably caused by some "manual" certbot invocation (I'm

guessing, I cannot have a look to /etc/letsenctypt)

Lodo' please: has that issue (#56678) been solved and how?

The problem on berlin (#62491) is (was) due to a failed challenge:

Toggle snippet (11 lines)

2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Hint: The

Certificate Authority failed to download the temporary challenge files created by Certbot.

Ensure that the listed domains serve their content from the provided --webroot-path/-w and

that files created there can be downloaded from the internet.

2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:

2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew

certificate disarchive.guix.gnu.org with error: Some challenges have failed.

Maxim please: has that issue (#62491) been solved and how?

[...]

Toggle quote (3 lines)

> this is the mcron that gets generated:

> [...]/certbot certonly -n --agree-tos --webroot -w /srv/http/ --cert-name dwim.hu -d dwim.hu --email attila@lendvai.name

Did you specify a different webroot? The default one defined in

"certbot-configuration" is "/var/www".

This is my certbot service config:

Toggle snippet (11 lines)

(service certbot-service-type

(certbot-configuration

(email "giovanni@biscuolo.net")

(certificates

(list

(certificate-configuration

(domains '("mx01.biscuolo.net")))))))

This is the certbot command that gets generated (and is scheduled in my

mcron):

Toggle snippet (7 lines)

#!/gnu/store/x4m56h5qkim0pnvx6vgvp541mrdwdrah-guile-3.0.9/bin/guile --no-auto-compile

(begin (use-modules (ice-9 match)) (let ((code 0)) (for-each (match-lambda ((name . command) (begin (format #t "Acquiring or renewing certificate: ~a~%" name) (set! code (or (apply system* command) code))))) (quote (("mx01.biscuolo.net" "/gnu/store/8vs33jaqpjkr5mzpz8syxvz2w472s5w7-certbot-2.3.0/bin/certbot" "certonly" "-n" "--agree-tos" "--webroot" "-w" "/var/www" "--cert-name" "mx01.biscuolo.net" "-d" "mx01.biscuolo.net" "--email" "giovanni@biscuolo.net")))) code))

Also, this is the "server" config for the generated nginx configuration:

Toggle snippet (20 lines)

server {

listen 80;

listen [::]:80;

server_name mx01.biscuolo.net ;

root /srv/http;

index index.html ;

server_tokens off;

location /.well-known {

root /var/www;

}

location / {

return 301 https://$host$request_uri;

}

Toggle quote (2 lines)

> and this what worked when i fixed the -w arg:

What was the error before you fixed the -w arg?

How was the nginx service configured?

Toggle quote (5 lines)

> [...]/certbot certonly -n --agree-tos --webroot -w /srv/http/dwim.hu --cert-name dwim.hu -d dwim.hu --email attila@lendvai.name

> i.e. the -w parameter should point to the webroot of the virtual

> domain,

No: that webroot is the directory from which to serve the Let’s Encrypt

challenge/response files, it have nothing do do with the webroot of the

corresponding virtual domain served by *another* nginx service (or other

service using the certificate)

Toggle quote (4 lines)

> but the guix config structure does not allow setting the webroot for

> each <certificate-configuration>, only at their parent, i.e. in the

> <certbot-configuration>.

AFAIU there is no need to set a certbot webroot for each certificate:

one webroot can serve all the challenge/response files needed for each

certificate, since certbot creates a unique subfolder in /.well-known

for each of them.

[...]

Toggle quote (4 lines)

> from the certbot log (i.e. challenge is saved at the wrong path):

> "Removing /srv/http/.well-known/acme-challenge/[hash]"

Why do you say that challenge is in the wrong path?

It works that way :-)

[...]

WDYT?

Happy hacking! Gio'

Giovanni Biscuolo

Xelera IT Infrastructures

Attila Lendvai wrote on 22 Nov 2023 19:05

Recipients:(name . Giovanni Biscuolo)(address . g@xelera.eu)

Message-ID:vvzWc4OLtIgDeH6vblpu15xH0Ka3R8tyi4EAcZ4i7vxRXDIn9-9Pt09kWC3v9-OIRpyFVuF3tX0VzOLS-QmyLRWRU_gLPOLELrzpB-zG60U=@lendvai.name

hi Giovanni,

it's been a long time, i don't remember much anymore.

but let's run a quick assert:

my server is serving multiple virtual domains (dwim.hu and lendvai.name) from completely different webroot directories. that's why i assumed that certbot needs to generate two different certificates for the two domains, and then be able to download them by accessing the same ip address through two separate domain names, and nginx serving the certificates corresponding to the domain name in the request.

did you write your answer with this in mind?

if yes, then i'll need to get back in context to answer properly.

• attila lendvai

• PGP: 963F 5D5F 45C7 DFCD 0A39

“Not to discuss with a man worthy of conversation is to waste the man. To discuss with a man not worthy of conversation is to waste words. The wise waste neither men nor words.”

— Confucius (551–479 BC), 'The Analects'

Maxim Cournoyer wrote on 23 Nov 2023 05:17

Recipients:(name . Giovanni Biscuolo)(address . g@xelera.eu)

Message-ID:87msv585nj.fsf@gmail.com

Hi Giovanni,

Giovanni Biscuolo <g@xelera.eu> writes:

Toggle quote (37 lines)> Hello Attila,
>
> I'm starting using certbot on a new Guix System server of mine: I've not
> much experience with this Guix service but I'm using certbot on other
> machines so I hope I can help here.
>
> Attila Lendvai <attila@lendvai.name> writes:
>
>> i don't think this is the same issue as #56678.
>
> AFAIU actually #56678 is (was?) caused by a duplicate certbot account:
>
>
> Please choose an account
> Choices: ['guix-hpc.bordeaux.inria.fr@2017-09-04T08:51:13Z (48c5)',
> 'localhost@2016-12-03T21:08:38Z (00bc)']
>
>
> on bayfront, probably caused by some "manual" certbot invocation (I'm
> guessing, I cannot have a look to /etc/letsenctypt)
>
> Lodo' please: has that issue (#56678) been solved and how?
>
> The problem on berlin (#62491) is (was) due to a failed challenge:
>
>
> 2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Hint: The
> Certificate Authority failed to download the temporary challenge files created by Certbot.
> Ensure that the listed domains serve their content from the provided --webroot-path/-w and
> that files created there can be downloaded from the internet.
> 2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
> 2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew
> certificate disarchive.guix.gnu.org with error: Some challenges have failed.
>
>
> Maxim please: has that issue (#62491) been solved and how?

I don't think it was truly resolved.  The problem keeps coming and
someone (usually Ludovic) has to manually run some commands get it to
cooperate (IIUC).  I've never investigated certbot nor configured such a
setup myself, so I'm not knowledgeable about it.

-- 
Thanks,
Maxim

Giovanni Biscuolo wrote on 23 Nov 2023 08:23

Recipients:(name . Attila Lendvai)(address . attila@lendvai.name)

Message-ID:87pm0153wh.fsf@xelera.eu

Hi Attila,

Attila Lendvai <attila@lendvai.name> writes:

[...]

Toggle quote (2 lines)

> if yes, then i'll need to get back in context to answer properly.

In this thread I'd like to understand what is (was?) the real nature of

the bugs described, I'm just trying to collect more information

I feel we should discuss how the certbot service works in a different

thread, to stay focused on the bug report

If you need further discussion, please feel free to open a new thread on

guix-devel and Cc: me! :-)

Thanks! Gio'

Giovanni Biscuolo

Xelera IT Infrastructures

Giovanni Biscuolo wrote on 23 Nov 2023 08:42

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)

Message-ID:87msv46hlk.fsf@xelera.eu

Hi Maxim,

thank you for your feedback.

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

Toggle quote (2 lines)

>> AFAIU actually #56678 is (was?) caused by a duplicate certbot account:

[...]

Toggle quote (2 lines)

>> The problem on berlin (#62491) is (was) due to a failed challenge:

I'm almost sure those are different bugs and I'm almost sure the bugs

are caused by _state_ (/etc/letsencrypt/[accounts|renewal])

[...]

Toggle quote (4 lines)

> I don't think it was truly resolved. The problem keeps coming and

> someone (usually Ludovic) has to manually run some commands get it to

> cooperate (IIUC).

Bugs like this are very difficult to reproduce and to investigate if we

wait the certs expiration and are forced to find a quick "workaround";

we should force a renewal (via CLI) before the expiration date and share

the logs to see what's happening.

I'd like to help but I'm not a sysadmin on bayfront nor on berlin.

I think this kind "statefulness issues" are affecting other users.

Happy hacking! Gio'

[...]

Giovanni Biscuolo

Xelera IT Infrastructures

Ludovic Courtès wrote on 23 Nov 2023 09:46

Recipients:(name . Giovanni Biscuolo)(address . g@xelera.eu)

Message-ID:87o7fkg8lb.fsf@inria.fr

Hi,

Giovanni Biscuolo <g@xelera.eu> skribis:

Toggle quote (13 lines)> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> [...]
>
>>> AFAIU actually #56678 is (was?) caused by a duplicate certbot account:
>
> [...]
>
>>> The problem on berlin (#62491) is (was) due to a failed challenge:
>
> I'm almost sure those are different bugs and I'm almost sure the bugs
> are caused by _state_ (/etc/letsencrypt/[accounts|renewal])

Indeed, that’s part of the problem.

Another example: our cerbot service offers a ‘deploy-hook’, but the

/gnu/store/… file name of that hook gets recorded somewhere in

/etc/letsencrypt and thus becomes invalid once the hook has been GC’d or

the system has been reconfigured.

Toggle quote (13 lines)>> I don't think it was truly resolved.  The problem keeps coming and
>> someone (usually Ludovic) has to manually run some commands get it to
>> cooperate (IIUC).
>
> Bugs like this are very difficult to reproduce and to investigate if we
> wait the certs expiration and are forced to find a quick "workaround";
> we should force a renewal (via CLI) before the expiration date and share
> the logs to see what's happening.
>
> I'd like to help but I'm not a sysadmin on bayfront nor on berlin.
>
> I think this kind "statefulness issues" are affecting other users.

Yeah, I think anyone running a web server on Guix System gets hit by

this issue. I’m not super knowledgeable about certbot either so I tend

to just hack around to get things to work, which is not great.

Ludo’.

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 62491@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date