nginx/certbot interaction doesn't work as documented

  • Open
  • quality assurance status badge
Details
3 participants
  • Alex Sassmannshausen
  • Andreas Enge
  • Robert Vollmert
Owner
unassigned
Submitted by
Robert Vollmert
Severity
normal
R
R
Robert Vollmert wrote on 26 Jun 2019 10:39
(address . bug-guix@gnu.org)
249AC56B-BE05-4162-B65D-618490163CB0@vllmrt.net
I’ve tried setting up nginx with certbot on guix. Two immediate issues:

- certbot extends the nginx service to serve challenge files. It appears
that this nginx service extension conflicts (silently) with an independently
configured nginx service. I.e., I had nginx previously configured, and
after adding certbot, my previous nginx kept running with the previous
configuration (even after herd restart nginx), while there was an additional
nginx config in the gnu store with the certbot-specific fragments. certbot
activation called nginx to test that fragment, but apparently never started
nginx (successfully?). There were no errors.

After removing the stand-alone nginx service and restarting nginx, it started
with the certbot configuration.

- After this, /var/lib/certbot/renew worked successfully to register a
certificate, but then failed when calling the nginx deploy hook that I’d
copied from the guix certbot documentation, because /var/run/nginx/pid
doesn’t exist. That might be a bug in the nginx package, not sure. I can’t
find an nginx pid file anywhere, and no other errors related to it either,
even though the config file includes
pid /var/run/nginx/pid;
A
A
Alex Sassmannshausen wrote on 26 Jun 2019 11:31
(address . bug-guix@gnu.org)(address . 36389@debbugs.gnu.org)
875zos3d6a.fsf@gmail.com
Hi Robert,

Robert Vollmert <rob@vllmrt.net> writes:

Toggle quote (14 lines)
> I’ve tried setting up nginx with certbot on guix. Two immediate issues:
>
> - certbot extends the nginx service to serve challenge files. It appears
> that this nginx service extension conflicts (silently) with an independently
> configured nginx service. I.e., I had nginx previously configured, and
> after adding certbot, my previous nginx kept running with the previous
> configuration (even after herd restart nginx), while there was an additional
> nginx config in the gnu store with the certbot-specific fragments. certbot
> activation called nginx to test that fragment, but apparently never started
> nginx (successfully?). There were no errors.
>
> After removing the stand-alone nginx service and restarting nginx, it started
> with the certbot configuration.

This sounds odd, and I don't recall having this issue on my servers with
nginx SSL server configuration extended with certbot service.

Toggle quote (9 lines)
>
> - After this, /var/lib/certbot/renew worked successfully to register a
> certificate, but then failed when calling the nginx deploy hook that I’d
> copied from the guix certbot documentation, because /var/run/nginx/pid
> doesn’t exist. That might be a bug in the nginx package, not sure. I can’t
> find an nginx pid file anywhere, and no other errors related to it either,
> even though the config file includes
> pid /var/run/nginx/pid;

The pid exists on my servers running an SSL nginx server config
configuration extended with certbot.

I've found the certbot & nginx services, overall, work very well
together. But there are a couple of gotchas in my experience:

- The certbot service includes a redirect from port 80 to 443 for all
except .well-known location. By itself this may cause no problems for
you.

- If deploying on a server that hitherto has no SSL certificate you have
a chicken and egg problem: you will want your site to be configured to
use the letsencrypt cert directories, to serve ssl (the redirect means
any non-ssl deployments won't work anyway), but those directories
don't yet exist as you haven't generated certs with certbot yet.

Here's a journey that should work:
- run system configuration with just the certbot service
- use certbot to generate your initial certificates
- reconfigure with additional nginx server configuration, pointing to
the SSL certificates created by certbot

If the above is not helpful, perhaps you could share the nginx
configuration generated when you have both certbot & your custom server
running?

Can't promise anything, but we might be able to spot what's happening.

Best wishes,

Alex
R
R
Robert Vollmert wrote on 26 Jun 2019 20:21
odd
(address . 36389@debbugs.gnu.org)
828259BC-1B2F-4741-975F-7E9E842AB750@vllmrt.net
I agree that it sounds odd, and some of my original diagnostic
must be skewed. After several configuration changes and
system reconfigurations and nginx restarts, I do appear to
have a sensible state currently, and I can’t reliably
reproduce the problems I had before. I’m also pretty sure I
didn’t imagine it all, though.


Here’s something else I ran into while getting there:

At some point, nginx was running, even after calling

# herd stop nginx

and herd did list it as stopped. That nginx instance that got
away from shepherd might have been involved in the earlier
trouble. (Is it ok for shepherd to lose track of a child like
that?)

Another thing was that I got a failed nginx configuration test
that didn’t make sense. Notably, it complained that

(a) the user directive `user nginx nginx;` is ineffective when
when not running as root and
(b) it didn’t have permission to access the letsencrypt keys.

Both of these indicate that the configuration test was not run
as root. I don’t see any reason in the code why that would
happen…


I’ll keep an eye on it and see if something similar occurs
again.
A
A
Andreas Enge wrote on 20 Dec 2021 17:17
Nginx and certbot
(address . 36389@debbugs.gnu.org)
YcCssR4t+wHOxsUu@jurong
Hello,

I am also experiencing problems with setting up nginx and certbot, but I
think it is more nginx that is to blame. After reconfiguring and restarting
nginx, it is still running with the old configuration. Only rebooting solves
the problem for me.

Here is what it looks like (everything as root):
$ ps -ef | grep nginx
root 2821 1 0 17:03 ? 00:00:00 nginx: master process /gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -c /gnu/store/q7bwm828r8y88sfs395n04bi8s6b7zwl-nginx.conf -p /var/run/nginx

$ guix system reconfigure ...
nginx: configuration file /gnu/store/clq2yshkq3gxpcqa6d54m8qif8i37kl9-nginx.conf test is successful

$ herd restart nginx; ps -ef | grep nginx
root 2835 1 0 17:12 ? 00:00:00 nginx: master process /gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -c /gnu/store/q7bwm828r8y88sfs395n04bi8s6b7zwl-nginx.conf -p /var/run/nginx

Notice that it is still running the old, q7b... config file!

$ reboot
$ ps -ef | grep nginx
root 188 1 0 17:13 ? 00:00:00 nginx: master process /gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -c /gnu/store/clq2yshkq3gxpcqa6d54m8qif8i37kl9-nginx.conf -p /var/run/nginx

Now the new, clq... config file is used!

So somehow nginx appears to memorise its previous configuration file even
when the service is stopped.

The problem can be solved by rebooting after each web server reconfiguration,
but this is of course not very comfortable.

Andreas
A
A
Andreas Enge wrote on 20 Dec 2021 17:46
(address . 36389@debbugs.gnu.org)
YcCzbGVQNfc/vts5@jurong
Actually this seems to be a thing of the service, not nginx itself.

When I stop the service with the old configuration file, manually run
/gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -p /var/run/nginx -c new_configuration_file
kill the process, and "herd restart nginx",
the herd service uses the old configuration file again.

Andreas
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 36389@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 36389
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch