nginx/certbot interaction doesn't work as documented

I’ve tried setting up nginx with certbot on guix. Two immediate issues:

- certbot extends the nginx service to serve challenge files. It appears

that this nginx service extension conflicts (silently) with an independently

configured nginx service. I.e., I had nginx previously configured, and

after adding certbot, my previous nginx kept running with the previous

configuration (even after herd restart nginx), while there was an additional

nginx config in the gnu store with the certbot-specific fragments. certbot

activation called nginx to test that fragment, but apparently never started

nginx (successfully?). There were no errors.

After removing the stand-alone nginx service and restarting nginx, it started

with the certbot configuration.

- After this, /var/lib/certbot/renew worked successfully to register a

certificate, but then failed when calling the nginx deploy hook that I’d

copied from the guix certbot documentation, because /var/run/nginx/pid

doesn’t exist. That might be a bug in the nginx package, not sure. I can’t

find an nginx pid file anywhere, and no other errors related to it either,

even though the config file includes

pid /var/run/nginx/pid;

Hi Robert,

Robert Vollmert <rob@vllmrt.net> writes:

This sounds odd, and I don't recall having this issue on my servers with

nginx SSL server configuration extended with certbot service.

The pid exists on my servers running an SSL nginx server config

configuration extended with certbot.

I've found the certbot & nginx services, overall, work very well

together. But there are a couple of gotchas in my experience:

- The certbot service includes a redirect from port 80 to 443 for all

except .well-known location. By itself this may cause no problems for

you.

- If deploying on a server that hitherto has no SSL certificate you have

a chicken and egg problem: you will want your site to be configured to

use the letsencrypt cert directories, to serve ssl (the redirect means

any non-ssl deployments won't work anyway), but those directories

don't yet exist as you haven't generated certs with certbot yet.

Here's a journey that should work:

- run system configuration with just the certbot service

- use certbot to generate your initial certificates

- reconfigure with additional nginx server configuration, pointing to

the SSL certificates created by certbot

If the above is not helpful, perhaps you could share the nginx

configuration generated when you have both certbot & your custom server

running?

Can't promise anything, but we might be able to spot what's happening.

Best wishes,

Alex

I agree that it sounds odd, and some of my original diagnostic

must be skewed. After several configuration changes and

system reconfigurations and nginx restarts, I do appear to

have a sensible state currently, and I can’t reliably

reproduce the problems I had before. I’m also pretty sure I

didn’t imagine it all, though.

Here’s something else I ran into while getting there:

At some point, nginx was running, even after calling

# herd stop nginx

and herd did list it as stopped. That nginx instance that got

away from shepherd might have been involved in the earlier

trouble. (Is it ok for shepherd to lose track of a child like

that?)

Another thing was that I got a failed nginx configuration test

that didn’t make sense. Notably, it complained that

(a) the user directive `user nginx nginx;` is ineffective when

when not running as root and

(b) it didn’t have permission to access the letsencrypt keys.

Both of these indicate that the configuration test was not run

as root. I don’t see any reason in the code why that would

happen…

I’ll keep an eye on it and see if something similar occurs

again.

Hello,

I am also experiencing problems with setting up nginx and certbot, but I

think it is more nginx that is to blame. After reconfiguring and restarting

nginx, it is still running with the old configuration. Only rebooting solves

the problem for me.

Here is what it looks like (everything as root):

$ ps -ef | grep nginx

root 2821 1 0 17:03 ? 00:00:00 nginx: master process /gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -c /gnu/store/q7bwm828r8y88sfs395n04bi8s6b7zwl-nginx.conf -p /var/run/nginx

$ guix system reconfigure ...

nginx: configuration file /gnu/store/clq2yshkq3gxpcqa6d54m8qif8i37kl9-nginx.conf test is successful

$ herd restart nginx; ps -ef | grep nginx

root 2835 1 0 17:12 ? 00:00:00 nginx: master process /gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -c /gnu/store/q7bwm828r8y88sfs395n04bi8s6b7zwl-nginx.conf -p /var/run/nginx

Notice that it is still running the old, q7b... config file!

$ reboot

$ ps -ef | grep nginx

root 188 1 0 17:13 ? 00:00:00 nginx: master process /gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -c /gnu/store/clq2yshkq3gxpcqa6d54m8qif8i37kl9-nginx.conf -p /var/run/nginx

Now the new, clq... config file is used!

So somehow nginx appears to memorise its previous configuration file even

when the service is stopped.

The problem can be solved by rebooting after each web server reconfiguration,

but this is of course not very comfortable.

Andreas

Actually this seems to be a thing of the service, not nginx itself.

When I stop the service with the old configuration file, manually run

/gnu/store/bdhfqs7sx3mal6pzz8z00hw4cpn5dj7x-nginx-1.21.4/sbin/nginx -p /var/run/nginx -c new_configuration_file

kill the process, and "herd restart nginx",

the herd service uses the old configuration file again.

Andreas