[berlin] certbot renewal appears to be broken

  • Open
  • quality assurance status badge
Details
4 participants
  • Attila Lendvai
  • Giovanni Biscuolo
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
normal
Merged with
M
M
Maxim Cournoyer wrote on 27 Mar 2023 23:05
(name . bug-guix)(address . bug-guix@gnu.org)(name . guix-sysadmin)(address . guix-sysadmin@gnu.org)
87cz4tq501.fsf@gmail.com
Hi,

The TLS cert of https://disarchive.guix.gnu.org/expired today. Looking
at /var/log/mcron.log on Berlin, we see that the last certbot renew job
failed like so:

Toggle snippet (134 lines)
2023-03-24 00:30:00 127768 certbot renew --webroot --webroot-path /var/www: running...
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Saving debug log to /var/log/letsencrypt/letsencrypt.log
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/bootstrappable.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/ci.guix.gnu.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/disarchive.guix.gnu.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:32:54 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for disarchive.guix.gnu.org
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Domain: disarchive.guix.gnu.org
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Type: unauthorized
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Detail: 141.80.181.40: Invalid response from https://disarchive.guix.gnu.org/.well-known/acme-challenge/O1kK3tsJtH0r9RwvbCIFhHagJhBwewV3Ka0NPW86nAI: 404
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate disarchive.guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/dump.guix.gnu.org.conf
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/guix.gnu.org.conf
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:10 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for guix.gnu.org
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Domain: guix.gnu.org
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Type: unauthorized
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Detail: 2a0c:e300::58: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/_PlXq5i2BRw23Ui1Yl4rLtyB2aSDnUNMZXurCWBwH-k: 404
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/guix.info.conf
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:19 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for guix.info and www.guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Domain: guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Type: unauthorized
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/O6y6aqSvLdjdS77MgaEhh7sN7Q75OQX3Jz69xnT4qnY: 404
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Domain: www.guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Type: unauthorized
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/lCioloihdJF6xwwTBg6cSNFjRearp4EBZBWcjkznrUE: 404
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate guix.info with error: Some challenges have failed.
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/issues.guix.gnu.org.conf
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/issues.guix.info.conf
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:26 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for issues.guix.info and 3 more domains
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Domain: guix.info
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Type: unauthorized
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Detail: 141.80.181.40: Invalid response from https://guix.gnu.org/.well-known/acme-challenge/Yv4KpoYC95LzGsM5IPTE68vf6lLfNHVK5kMUocSuDW0: 404
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate issues.guix.info with error: Some challenges have failed.
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/monitor.guix.gnu.org.conf
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: Renewing an existing certificate for monitor.guix.gnu.org
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Domain: monitor.guix.gnu.org
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Type: unauthorized
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Detail: 141.80.181.40: Invalid response from https://monitor.guix.gnu.org/.well-known/acme-challenge/_wxH92e9QQag7TEYdqsA4-C-5pE5DnUd6pzMvQWzWNU: 400
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew certificate monitor.guix.gnu.org with error: Some challenges have failed.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/www.guixwl.org-0001.conf
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Processing /etc/letsencrypt/renewal/www.guixwl.org.conf
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Certificate not yet due for renewal
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: The following certificates are not due for renewal yet:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: /etc/letsencrypt/live/bootstrappable.org/fullchain.pem expires on 2023-05-14 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: /etc/letsencrypt/live/ci.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: /etc/letsencrypt/live/dump.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: /etc/letsencrypt/live/issues.guix.gnu.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: /etc/letsencrypt/live/www.guixwl.org-0001/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: /etc/letsencrypt/live/www.guixwl.org/fullchain.pem expires on 2023-06-04 (skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: All renewals failed. The following certificates could not be renewed:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: /etc/letsencrypt/live/disarchive.guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: /etc/letsencrypt/live/guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: /etc/letsencrypt/live/guix.info/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: /etc/letsencrypt/live/issues.guix.info/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: /etc/letsencrypt/live/monitor.guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 5 renew failure(s), 0 parse failure(s)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: failed after 234.635s with: (misc-error #f unclean exit status ~S (1) #f)--8<---------------cut here---------------end--------------->8---

I removed the certbot file name prefix
(/gnu/store/jnp0166xw62dafd2zgxdmvjb6yq8ak32-certbot-1.28.0/bin/) in the
above output to improve readability.

--
Thanks,
Maxim
M
M
Maxim Cournoyer wrote on 29 Mar 2023 02:42
control message for bug #62491
(address . control@debbugs.gnu.org)
87355o75hc.fsf@gmail.com
merge 62491 56678
quit
A
A
Attila Lendvai wrote on 4 May 2023 16:37
(No Subject)
(name . 62491@debbugs.gnu.org)(address . 62491@debbugs.gnu.org)(name . clement@lassieur.org)(address . clement@lassieur.org)
xUfl58WwIGDQakb2wFTlATboSCRB4-uR1eu3HS0G6Mo1IdzgYsOsA2D4YmBt_TgLWFrMlmJFi2a2yykmDNZJuUCRHoENEmnvrhWdSYC8DSA=@lendvai.name
i don't think this is the same issue as #56678.

or at least what i'm seeing on my server is that the wrong certbot cmd line is generated, which then results in saving the challenge at the wrong path.

this is the mcron that gets generated:
[...]/certbot certonly -n --agree-tos --webroot -w /srv/http/ --cert-name dwim.hu -d dwim.hu --email attila@lendvai.name

and this what worked when i fixed the -w arg:

[...]/certbot certonly -n --agree-tos --webroot -w /srv/http/dwim.hu --cert-name dwim.hu -d dwim.hu --email attila@lendvai.name

i.e. the -w parameter should point to the webroot of the virtual domain, but the guix config structure does not allow setting the webroot for each <certificate-configuration>, only at their parent, i.e. in the <certbot-configuration>.

this all seems to me as if the certbot service code was assuming that the certbot script will append the domain names (specified with -d) to the webroot path, but it does not.

from the certbot log (i.e. challenge is saved at the wrong path):

"Removing /srv/http/.well-known/acme-challenge/[hash]"

the relevant code is from 2018, so certbot's behavior may very well have changed since then:


it seems to me that the webroot field should be moved down into <certificate-configuration>.

am i right? if so i may try to patch this up.

--
- attila
PGP: 5D5F 45C7 DFCD 0A39
--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“State is the name of the coldest of all cold monsters. Coldly it lies; and this lie slips from its mouth: "I, the state, am the people."”
— Friedrich Nietzsche (1844–1900), 'Thus Spoke Zarathustra' (1885), http://j.mp/1k6pbwS
G
G
Giovanni Biscuolo wrote on 22 Nov 2023 18:37
bug#62491: [berlin] certbot renewal appears to be broken
87sf4x6653.fsf@xelera.eu
Hello Attila,

I'm starting using certbot on a new Guix System server of mine: I've not
much experience with this Guix service but I'm using certbot on other
machines so I hope I can help here.

Attila Lendvai <attila@lendvai.name> writes:

Toggle quote (2 lines)
> i don't think this is the same issue as #56678.

AFAIU actually #56678 is (was?) caused by a duplicate certbot account:

Toggle snippet (7 lines)
Please choose an account
Choices: ['guix-hpc.bordeaux.inria.fr@2017-09-04T08:51:13Z (48c5)',
'localhost@2016-12-03T21:08:38Z (00bc)']


on bayfront, probably caused by some "manual" certbot invocation (I'm
guessing, I cannot have a look to /etc/letsenctypt)

Lodo' please: has that issue (#56678) been solved and how?

The problem on berlin (#62491) is (was) due to a failed challenge:

Toggle snippet (11 lines)
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Hint: The
Certificate Authority failed to download the temporary challenge files created by Certbot.
Ensure that the listed domains serve their content from the provided --webroot-path/-w and
that files created there can be downloaded from the internet.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew
certificate disarchive.guix.gnu.org with error: Some challenges have failed.


Maxim please: has that issue (#62491) been solved and how?

[...]

Toggle quote (3 lines)
> this is the mcron that gets generated:
> [...]/certbot certonly -n --agree-tos --webroot -w /srv/http/ --cert-name dwim.hu -d dwim.hu --email attila@lendvai.name

Did you specify a different webroot? The default one defined in
"certbot-configuration" is "/var/www".

This is my certbot service config:

Toggle snippet (11 lines)
(service certbot-service-type
(certbot-configuration
(email "giovanni@biscuolo.net")
(certificates
(list
(certificate-configuration
(domains '("mx01.biscuolo.net")))))))


This is the certbot command that gets generated (and is scheduled in my
mcron):

Toggle snippet (7 lines)
#!/gnu/store/x4m56h5qkim0pnvx6vgvp541mrdwdrah-guile-3.0.9/bin/guile --no-auto-compile
!#
(begin (use-modules (ice-9 match)) (let ((code 0)) (for-each (match-lambda ((name . command) (begin (format #t "Acquiring or renewing certificate: ~a~%" name) (set! code (or (apply system* command) code))))) (quote (("mx01.biscuolo.net" "/gnu/store/8vs33jaqpjkr5mzpz8syxvz2w472s5w7-certbot-2.3.0/bin/certbot" "certonly" "-n" "--agree-tos" "--webroot" "-w" "/var/www" "--cert-name" "mx01.biscuolo.net" "-d" "mx01.biscuolo.net" "--email" "giovanni@biscuolo.net")))) code))


Also, this is the "server" config for the generated nginx configuration:

Toggle snippet (20 lines)
server {
listen 80;
listen [::]:80;
server_name mx01.biscuolo.net ;
root /srv/http;
index index.html ;
server_tokens off;

location /.well-known {
root /var/www;
}
location / {
return 301 https://$host$request_uri;
}

}


Toggle quote (2 lines)
> and this what worked when i fixed the -w arg:

What was the error before you fixed the -w arg?

How was the nginx service configured?

Toggle quote (5 lines)
> [...]/certbot certonly -n --agree-tos --webroot -w /srv/http/dwim.hu --cert-name dwim.hu -d dwim.hu --email attila@lendvai.name
>
> i.e. the -w parameter should point to the webroot of the virtual
> domain,

No: that webroot is the directory from which to serve the Let’s Encrypt
challenge/response files, it have nothing do do with the webroot of the
corresponding virtual domain served by *another* nginx service (or other
service using the certificate)

Toggle quote (4 lines)
> but the guix config structure does not allow setting the webroot for
> each <certificate-configuration>, only at their parent, i.e. in the
> <certbot-configuration>.

AFAIU there is no need to set a certbot webroot for each certificate:
one webroot can serve all the challenge/response files needed for each
certificate, since certbot creates a unique subfolder in /.well-known
for each of them.

[...]

Toggle quote (4 lines)
> from the certbot log (i.e. challenge is saved at the wrong path):
>
> "Removing /srv/http/.well-known/acme-challenge/[hash]"

Why do you say that challenge is in the wrong path?

It works that way :-)

[...]

WDYT?

Happy hacking! Gio'

--
Giovanni Biscuolo

Xelera IT Infrastructures
-----BEGIN PGP SIGNATURE-----

iQJABAEBCgAqFiEERcxjuFJYydVfNLI5030Op87MORIFAmVePGgMHGdAeGVsZXJh
LmV1AAoJENN9DqfOzDkSgp4P/RUROCBqxGlPqo2qiNRrZ7dLgNzxcKj3Mo9y9htI
WcHInP5QDGL36slGYu/+3Bpo56WK+pRY5IXHwlWTlRZOimyOWoFhZGQ32OnamoFB
PUglZvmVtvUPkY0jPALd0oiX9VcO7R7+vUi8JYmpMNhn1XKdQpYyNi3yac6mGj02
87e0ZXW9YnAvRy3PuqEih6+V/OyCysPh8VX80M7GqCgiVuEbqDEFbpmUVRM6TyOh
5zuFLlf66zfXUVg2JbGMt/kcCCpSERF453dYZDyClY/YkSjwHC7Spg82A7OZ1t8O
o25PZalYw7LvkSk20SrhdavC2jWQsQu0O1F1NAAVzzqyCNd8F76g6j0CdzKPKLYh
meSFy9S7fRVVrV6szjJPPJPT0kA3ggBtAO7wuGyNAo3gPEGF6cCuybSI7UjhJdxj
yaU3+xeunlbc8EzpN+WFlYVb6zvOYIxPXKhBfxtxsMHd3I6z0YKr36kS0Z03Z9mP
e/KRojJD5BJmZlJiQE2erZh9o1EfU4TSvlljcBI9ZVQM7AQqwlv2DCADiQO4Vmby
064eRJu6lOWCccxa0H5wLymZVbAg9rJ1ti0M5BOsJj9u8oGH9t2y/sa4ChB7pJxi
KMS/t9CDUx75NIcDhSg50lrH9uQs27+o9ExDQBe1DqrIfOgLHX5iLzXQ389SjrXn
IGfL
=7aif
-----END PGP SIGNATURE-----

A
A
Attila Lendvai wrote on 22 Nov 2023 19:05
(name . Giovanni Biscuolo)(address . g@xelera.eu)
vvzWc4OLtIgDeH6vblpu15xH0Ka3R8tyi4EAcZ4i7vxRXDIn9-9Pt09kWC3v9-OIRpyFVuF3tX0VzOLS-QmyLRWRU_gLPOLELrzpB-zG60U=@lendvai.name
hi Giovanni,

it's been a long time, i don't remember much anymore.

but let's run a quick assert:

my server is serving multiple virtual domains (dwim.hu and lendvai.name) from completely different webroot directories. that's why i assumed that certbot needs to generate two different certificates for the two domains, and then be able to download them by accessing the same ip address through two separate domain names, and nginx serving the certificates corresponding to the domain name in the request.

did you write your answer with this in mind?

if yes, then i'll need to get back in context to answer properly.

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Not to discuss with a man worthy of conversation is to waste the man. To discuss with a man not worthy of conversation is to waste words. The wise waste neither men nor words.”
— Confucius (551–479 BC), 'The Analects'
M
M
Maxim Cournoyer wrote on 23 Nov 2023 05:17
(name . Giovanni Biscuolo)(address . g@xelera.eu)
87msv585nj.fsf@gmail.com
Hi Giovanni,

Giovanni Biscuolo <g@xelera.eu> writes:

Toggle quote (37 lines)
> Hello Attila,
>
> I'm starting using certbot on a new Guix System server of mine: I've not
> much experience with this Guix service but I'm using certbot on other
> machines so I hope I can help here.
>
> Attila Lendvai <attila@lendvai.name> writes:
>
>> i don't think this is the same issue as #56678.
>
> AFAIU actually #56678 is (was?) caused by a duplicate certbot account:
>
>
> Please choose an account
> Choices: ['guix-hpc.bordeaux.inria.fr@2017-09-04T08:51:13Z (48c5)',
> 'localhost@2016-12-03T21:08:38Z (00bc)']
>
>
> on bayfront, probably caused by some "manual" certbot invocation (I'm
> guessing, I cannot have a look to /etc/letsenctypt)
>
> Lodo' please: has that issue (#56678) been solved and how?
>
> The problem on berlin (#62491) is (was) due to a failed challenge:
>
>
> 2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Hint: The
> Certificate Authority failed to download the temporary challenge files created by Certbot.
> Ensure that the listed domains serve their content from the provided --webroot-path/-w and
> that files created there can be downloaded from the internet.
> 2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:
> 2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: Failed to renew
> certificate disarchive.guix.gnu.org with error: Some challenges have failed.
>
>
> Maxim please: has that issue (#62491) been solved and how?

I don't think it was truly resolved. The problem keeps coming and
someone (usually Ludovic) has to manually run some commands get it to
cooperate (IIUC). I've never investigated certbot nor configured such a
setup myself, so I'm not knowledgeable about it.

--
Thanks,
Maxim
G
G
Giovanni Biscuolo wrote on 23 Nov 2023 08:23
(name . Attila Lendvai)(address . attila@lendvai.name)
87pm0153wh.fsf@xelera.eu
Hi Attila,

Attila Lendvai <attila@lendvai.name> writes:

[...]

Toggle quote (2 lines)
> if yes, then i'll need to get back in context to answer properly.

In this thread I'd like to understand what is (was?) the real nature of
the bugs described, I'm just trying to collect more information

I feel we should discuss how the certbot service works in a different
thread, to stay focused on the bug report

If you need further discussion, please feel free to open a new thread on
guix-devel and Cc: me! :-)

Thanks! Gio'

--
Giovanni Biscuolo

Xelera IT Infrastructures
-----BEGIN PGP SIGNATURE-----

iQJABAEBCgAqFiEERcxjuFJYydVfNLI5030Op87MORIFAmVe/f8MHGdAeGVsZXJh
LmV1AAoJENN9DqfOzDkSavcP/1QUxvHcdLv0KhAled+68zp42CBx9tFFCel/PSOw
kdxRNM4Un5hEk7CQPoULvJfhQr9yk5teFl6KxA0T5BnRTfYXTQ4ShuhGvl67Jf9a
w2F1KMPiNntEiM58qy9zlv8V4zudXgXGLR3870KMnyg2tCwDfJ807goKO3uzfwBs
Ys3qCrCZlTlZwAbtuHOGCWMS/IyXLzIbaRD+Tu98S2Mu7xiPbjybGBf8TY56MpWZ
YJm7rCTrKrL2QbCshXhWZMRPhz2qGvv63YlsN5G0suyYu39d9GLcofVeMRBm42TY
9sZlfd/BEiW+MyNNSLRY0dYLQcwsVts+K2rQt2ga+mi5Yt3op8ZAVcLYSyKUHDKc
ekNq3af/EltpBdMHV15kjpxtz1ysWsTIvCaePnWFXQ9hTICmzfXv5DCSQ+/MSwZN
CAJuOVfKIzTi23xTqe1q4VrD+QOHChu4St5xpTpPMxzBiYx5JxePcwr3Edz5MR46
1dPrVUVM64/oyeS1P+FAp1Bj2D8GX8LFfrb6W6uTanVfZrDleOk9nUZ+5NHzwUsv
KiUHpzASoZygsSV9608Yvfn7vW5+twSI34mPx/cpk9EmTLS2Hl0pvKliWSJREZXd
d0iIOmO+j9SLoLR9545RX9g+OlURVHKvJn7YV9GenVLWNsKJvSQxT3FA456IYaHK
no7C
=u0sh
-----END PGP SIGNATURE-----

G
G
Giovanni Biscuolo wrote on 23 Nov 2023 08:42
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87msv46hlk.fsf@xelera.eu
Hi Maxim,

thank you for your feedback.

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

Toggle quote (2 lines)
>> AFAIU actually #56678 is (was?) caused by a duplicate certbot account:

[...]

Toggle quote (2 lines)
>> The problem on berlin (#62491) is (was) due to a failed challenge:

I'm almost sure those are different bugs and I'm almost sure the bugs
are caused by _state_ (/etc/letsencrypt/[accounts|renewal])

[...]

Toggle quote (4 lines)
> I don't think it was truly resolved. The problem keeps coming and
> someone (usually Ludovic) has to manually run some commands get it to
> cooperate (IIUC).

Bugs like this are very difficult to reproduce and to investigate if we
wait the certs expiration and are forced to find a quick "workaround";
we should force a renewal (via CLI) before the expiration date and share
the logs to see what's happening.

I'd like to help but I'm not a sysadmin on bayfront nor on berlin.

I think this kind "statefulness issues" are affecting other users.

Happy hacking! Gio'

[...]

--
Giovanni Biscuolo

Xelera IT Infrastructures
-----BEGIN PGP SIGNATURE-----

iQJABAEBCgAqFiEERcxjuFJYydVfNLI5030Op87MORIFAmVfAmcMHGdAeGVsZXJh
LmV1AAoJENN9DqfOzDkSzHAP/0JouWJlRwZEX83kz4JSFw3M9kP1qY+rKirg5eyb
ZnJ32QccLdSjGd4V7xDppXJT+CFacfUZXj6hB1bcaZ/Qn57YFTKC5zgpE+B8bjVe
7xRBgCEaZw0aiHP1u/EHDQ6gySxib8UsO1TX/O7Gddg/LD9g3bxHGqaFk4GhaK3x
QusIdVsxMjuWl51Cn4ExpCUgCNLcmhoZa+ZUcjkHQNcH6++U1t1l1771UQWVWCrF
uO5tsClptYlO97KGV5oBNP4b/RGy7vuyeT987tR+uFk7ExWRar8ICc3OUugSIxPj
tb2M7McJvgT55uyIM0Pe2QdSjqWk+BKKm0UGoLiBwYqLLzRjlSDLdfA7SYgRT1TX
DY9C4SlWidc0az5bhsksZqwAbADXGR2pLAU8dL1aCD+wC+BdQhLvqwT2cA76iSBQ
iHeggv75zm2yZox2S8vDh+nn6EaNgFO5RAQjHlJ72O9O4XkQH1h6JLuLEqAFZsi4
Q5hPSTZilwixC/HyEvpcXGOUHAZl3VK/X67Ccw8xwrC7dpWNm8ul4bp85XH3qo98
ygbs972njGimO1mHz/1MHuXfQHHyGrKWEOKn72ZgGC/bMmLUTsfsBo4+m95R7pWO
kRoYe3Skj1WJfl5kc9UFDedLy/8wz47NBGsc+3PxjEJdUhBg6Ts8/p2AN3HO4zl6
g1Dn
=EhxT
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 23 Nov 2023 09:46
(name . Giovanni Biscuolo)(address . g@xelera.eu)
87o7fkg8lb.fsf@inria.fr
Hi,

Giovanni Biscuolo <g@xelera.eu> skribis:

Toggle quote (13 lines)
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> [...]
>
>>> AFAIU actually #56678 is (was?) caused by a duplicate certbot account:
>
> [...]
>
>>> The problem on berlin (#62491) is (was) due to a failed challenge:
>
> I'm almost sure those are different bugs and I'm almost sure the bugs
> are caused by _state_ (/etc/letsencrypt/[accounts|renewal])

Indeed, that’s part of the problem.

Another example: our cerbot service offers a ‘deploy-hook’, but the
/gnu/store/… file name of that hook gets recorded somewhere in
/etc/letsencrypt and thus becomes invalid once the hook has been GC’d or
the system has been reconfigured.

Toggle quote (13 lines)
>> I don't think it was truly resolved. The problem keeps coming and
>> someone (usually Ludovic) has to manually run some commands get it to
>> cooperate (IIUC).
>
> Bugs like this are very difficult to reproduce and to investigate if we
> wait the certs expiration and are forced to find a quick "workaround";
> we should force a renewal (via CLI) before the expiration date and share
> the logs to see what's happening.
>
> I'd like to help but I'm not a sysadmin on bayfront nor on berlin.
>
> I think this kind "statefulness issues" are affecting other users.

Yeah, I think anyone running a web server on Guix System gets hit by
this issue. I’m not super knowledgeable about certbot either so I tend
to just hack around to get things to work, which is not great.

Ludo’.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 62491@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 62491
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch