ca-certificate-bundle fails to build

Hi,

I’ve seen this multiple times now, most recently with commit

8ef38cd1bf7ec17b8d1bc1c0bcb42ac30ac30406 on a foreign system: Some

operation, here a `guix pack` tries to build ca-certificate-bundle and

it fails the first time, but rerunning the same operation succeeds:

---snip---

-builder for `/gnu/store/iqm2zwq8jrr9gaw5jp4vs42j887aglmk-ca-certificate-bundle.drv' failed with exit code 1

Erstellung von /gnu/store/iqm2zwq8jrr9gaw5jp4vs42j887aglmk-ca-certificate-bundle.drv fehlgeschlagen

Das Erstellungsprotokoll kann unter „/var/log/guix/drvs/iq/m2zwq8jrr9gaw5jp4vs42j887aglmk-ca-certificate-bundle.drv.bz2“ eingesehen werden.

cannot build derivation `/gnu/store/riw7sfkl8a5rapyvf6vvh07cwv3nrwil-profile.drv': 1 dependencies couldn't be built

cannot build derivation `/gnu/store/l8zdf69zk0pg599gifcyxbd80h676yp1-python-jupyterlab-bash-coreutils-docker-pack.tar.zst.drv': 1 dependencies couldn't be built

guix pack: Fehler: build of `/gnu/store/l8zdf69zk0pg599gifcyxbd80h676yp1-python-jupyterlab-bash-coreutils-docker-pack.tar.zst.drv' failed

---snap---

The backtrace looks like this:

---snip---

Backtrace:

2 (primitive-load "/gnu/store/74i539y9b0bvslk3bk0kc27q76p?")

In ice-9/eval.scm:

619:8 1 (_ #f)

In unknown file:

0 (setlocale 6 "en_US.utf8")

ERROR: In procedure setlocale:

In procedure setlocale: Invalid argument

---snap---

Curiously the output path mentioned in the .drv

(/gnu/store/zjbpik3bfhmcamqzixzm1892mqbknnj9-ca-certificate-bundle) does

not exist, even after successfully running the command. Manually

building the driver with `guix build

/gnu/store/iqm2zwq8jrr9gaw5jp4vs42j887aglmk-ca-certificate-bundle.drv`

yields the same error as above, so I’m assuming the profile hook is

simply broken, while `guix pack` does not actually depend on it.

Cheers,

Lars

Derive([("out","/gnu/store/zjbpik3bfhmcamqzixzm1892mqbknnj9-ca-certificate-bundle","","")],[("/gnu/store/107rn94lvmjzsrakvd06bd93d47hyd2x-findutils-4.7.0.drv",["out"]),("/gnu/store/1blcyyk3ccdjprg204zapq8qjc5ibr3x-python-attrs-21.2.0.drv",["out"]),("/gnu/store/4626bkx4rbgn2c7bsgwwi5q1kvkv20mk-python-pyzmq-22.1.0.drv",["out"]),("/gnu/store/5ld8a3299gm11gi5phhzyh24l0p5jn3h-guile-3.0.2.drv",["out"]),("/gnu/store/70y6fmwpfdg6w7c9kgbdj12vs6qfll19-python-jupyter-core-4.7.1.drv",["out"]),("/gnu/store/8ic55yjz9vzlscwvi0cah69rf3srg88c-module-import-compiled.drv",["out"]),("/gnu/store/97pdzby1i6ljr719848df9j84v8diy5v-python-zope-interface-5.1.0.drv",["out"]),("/gnu/store/adicmdfi4jrqg9kyndy12sgqh6rcw5c3-coreutils-8.32.drv",["out"]),("/gnu/store/cgqvxwlwvdybxlnj7j9a1nbyw98apnpw-python-ipython-genutils-0.1.0.drv",["out"]),("/gnu/store/d3qqhr08h3cg9ic68qrnj3jfp3jmq26q-python-importlib-metadata-1.5.0.drv",["out"]),("/gnu/store/dl99sx2bqksmmywk0pvxil3a1i6shykv-python-six-1.14.0.drv",["out"]),("/gnu/store/fvfnw45mzc5n9kz0slw53jd33zy9sp7y-python-traitlets-4.3.3.drv",["out"]),("/gnu/store/g1n6xsvsihljvakwrb6j6rlxvzcnc407-python-nbformat-5.1.3.drv",["out"]),("/gnu/store/g5lwnsbbfy6zkgyf7jb6pd9svlg1hv6c-node-10.24.0.drv",["out"]),("/gnu/store/h87hgphx6jnfhph21h886rh8kbajmpz1-python-zipp-1.0.0.drv",["out"]),("/gnu/store/hh1r1lfa0r01vd74y4hmdhkiv1yrxxj8-python-twisted-19.7.0.drv",["out"]),("/gnu/store/i4m0kg16hwvqqb8z6cslnsq0dfmsrndq-python-pandocfilters-1.4.3.drv",["out"]),("/gnu/store/jskrmcik3i55xlgi8rbn13683ksfrk23-python-pygments-2.7.4.drv",["out"]),("/gnu/store/kllhbk944sjis1bys3jw3x926s44j6mh-python-more-itertools-8.2.0.drv",["out"]),("/gnu/store/m2v3cqiadb16w06pgnfz53rd7ram0rjz-python-jupyterlab-pygments-0.1.2.drv",["out"]),("/gnu/store/mfh071vb88x859b0my59zir3gwldalm6-python-send2trash-1.5.0.drv",["out"]),("/gnu/store/ngh9h4s31my5q5h83bn1rrjlykj8r2aq-python-jsonschema-3.2.0.drv",["out"]),("/gnu/store/q899l6i4l0piy4v88l0s9csgl4w8b7xg-bash-5.0.16.drv",["out"]),("/gnu/store/qxd9ngc6sbassv1wgyh557h6r6dci4ry-python-idna-2.10.drv",["out"]),("/gnu/store/rpi7nq5q7m2isfzc5mc6gypwpsa1sypq-python-terminado-0.10.0.drv",["out"]),("/gnu/store/rs55kbs4fvkrmygrffmrgywvlvn36wn7-python-testpath-0.4.4.drv",["out"]),("/gnu/store/rsw5dcynczjm5dr2x324bngc3gvfykqz-python-tornado-6.1.drv",["out"]),("/gnu/store/sf249r9rsgj3xvnrwn728bs46pb9iia7-python-ptyprocess-0.5.2.drv",["out"]),("/gnu/store/vlhmdjkiwxrqzizalwvg8jn17j6mmm06-python-constantly-15.1.0.drv",["out"]),("/gnu/store/w0rblvjfhz49a084ihmd6k4plwh727c4-python-automat-20.2.0.drv",["out"]),("/gnu/store/w9wc5939py5fp40mpd205yzifqx6zxv3-python-incremental-17.5.0.drv",["out"]),("/gnu/store/wkba5dds9kgwrd1qsjxf5lxlpz3ddp12-python-pyhamcrest-1.9.0-0.25fdc5f.drv",["out"]),("/gnu/store/wkym6kznwkbs74x6n4fvl90299a12afv-python-pyrsistent-0.16.0.drv",["out"]),("/gnu/store/xmzsqzaigpzb85zni80db5nycrhc8par-python-prometheus-client-0.7.1.drv",["out"]),("/gnu/store/yz94jmahv05p2gjwx5c57660dbn7nbiq-python-decorator-4.3.0.drv",["out"]),("/gnu/store/ziz5svq4vfm1l2x767k2ywb8svkwdw0s-python-hyperlink-19.0.0.drv",["out"])],["/gnu/store/74i539y9b0bvslk3bk0kc27q76p6zigb-ca-certificate-bundle-builder","/gnu/store/xv5ylv9hxvs1wraw375b5g9jwy57vs8p-module-import"],"x86_64-linux","/gnu/store/0m0vd873jp61lcm4xa3ljdgx381qa782-guile-3.0.2/bin/guile",["--no-auto-compile","-L","/gnu/store/xv5ylv9hxvs1wraw375b5g9jwy57vs8p-module-import","-C","/gnu/store/08da6i38fzn9g36gyq443k03f84nk8rk-module-import-compiled","/gnu/store/74i539y9b0bvslk3bk0kc27q76p6zigb-ca-certificate-bundle-builder"],[("allowSubstitutes","0"),("guix properties","((type . profile-hook) (hook . ca-certificate-bundle))"),("out","/gnu/store/zjbpik3bfhmcamqzixzm1892mqbknnj9-ca-certificate-bundle"),("preferLocalBuild","1")])

Hi,

Lars-Dominik Braun <lars@6xq.net> skribis:

Is this the backtrace found in the build log (under /var/log/guix/drvs)?

I tried and failed to reproduce it like this:

Could you find a way to reproduce the issue?

Also, could you run ‘guix build glibc-utf8-locales --check’, so make

sure that store item is not corrupt?

I have:

TIA,

Ludo’.

Hi Ludo,

thanks for looking into this!

Yes, it is printed on stdout, as well as part of the build log.

Alright, let’s see. The command I have been using is

guix pack -L . -C 'zstd' -f docker -S /bin=bin python-jupyterlab bash coreutils findutils

with . being a checkout of guix-science (same applies to `guix

time-machine` though). The first time it’ll fail, but the second time

it succeeds. Running `guix gc` makes it fail again the first time.

The docker-pack.tar.zst.drv used for the first build is

different from the second one (different hash prefix). For me it’s

/gnu/store/r096cm3np7hbdn853ih35h1a5l39in4s-python-jupyterlab-bash-coreutils-docker-pack.tar.zst.drv

the first time and

/gnu/store/dywspxjshfjhc07i17hkcyrlq8kn7m07-python-jupyterlab-bash-coreutils-docker-pack.tar.zst.drv

the second time. YMMV.

Looking at ca-certificate-bundle.drv, the first one lacks any form of

glibc-utf8-locales as an input (neither in the .drv, nor via `guix gc

--references <.drv> | grep glibc-`), so it’s clear it must fail. I’m

not quite sure why though, since the actual builder still has a reference

to the locales and sets LOCPATH. So I’m a little puzzled.

Exactly the same for me, so we have the same data at least. `guix gc`

with the repair,contents options also does not show any corrupted

items. I’m thus assuming my store is intact.

Any ideas?

Thanks,

Lars

For what it's worth, this same thing just happened to me when updating

my profile on my laptop. The first time it failed exactly as described,

and the second time it worked.

Kyle

On Mon Sep 6, 2021 at 6:18 PM CDT, Kyle Andrews wrote:

I've also hit this twice in the last couple weeks. I just ran the

updates again and they worked both times.

Exact same issue trying to update my Guix Laptop.

guix pull - runs fine

sudo guix reconfigure - fails

fails to build ca-bundle with the above mentioned "invalid argument in

setlocale" error

--

Best regards

Vlad Kozin

Hi,

Lars-Dominik Braun <lars@6xq.net> skribis:

Wait, you run the same command twice and it leads a different .drv?

The expected behavior is that the .drv is always the same:

Could you send the contents of this “broken” ca-certificate-bundle.drv

as well as the ca-certificate-bundle-builder it refers to?

Thanks,

Ludo’.

On Fri, 10 Sep 2021, Ludovic Courtès wrote:

Just to note that I've seen this two, where I get a different derivation

the next time that works. Apologies for not reporting it at the time. It

always seems to happen when I'm busy with something else, and since it's

not reproducible… If I can get it to happen to me again, I'll collect more

information as well.

Best,

Jack

Hi Ludo,

yes, but only if I run `guix gc` before the first one, i.e. it’ll

build/downloads lots of packages before actually building the .drv.

This is with commit 8b2b027df9198cb5b22505813ada0c5e199929ed for the same

`guix pack` I mentioned earlier. I’m attaching:

Failure:

/gnu/store/kxkxji6m101ylgpa82jmxvmy5szpvk1k-ca-certificate-bundle.drv

/gnu/store/wahy9p8985vbcibwwjk3p73hxh82f90v-ca-certificate-bundle-builder

Success:

/gnu/store/rx6pqcpgwcyl6spfi8acgjzksrpqhac1-ca-certificate-bundle.drv

/gnu/store/wahy9p8985vbcibwwjk3p73hxh82f90v-ca-certificate-bundle-builder

(Yes, exactly the same builder input hash. I double-checked.)

Meanwhile a colleague of mine has seen the same problem during building a

`guix environment` too.

Cheers,

Lars

Derive([("out","/gnu/store/2df7jsi5h7ckdzalccrv7fgxgq5mnp2b-ca-certificate-bundle","","")],[("/gnu/store/107rn94lvmjzsrakvd06bd93d47hyd2x-findutils-4.7.0.drv",["out"]),("/gnu/store/4kc352jqrgldjx5jkdv144kd6032bks8-python-send2trash-1.5.0.drv",["out"]),("/gnu/store/5ld8a3299gm11gi5phhzyh24l0p5jn3h-guile-3.0.2.drv",["out"]),("/gnu/store/69g8wmbi635kynqxgfnrkrlih7n7dw9x-python-terminado-0.10.0.drv",["out"]),("/gnu/store/6v547balkz4s9n9qrnncawzk32i9zm55-python-twisted-19.7.0.drv",["out"]),("/gnu/store/8ic55yjz9vzlscwvi0cah69rf3srg88c-module-import-compiled.drv",["out"]),("/gnu/store/97pdzby1i6ljr719848df9j84v8diy5v-python-zope-interface-5.1.0.drv",["out"]),("/gnu/store/adicmdfi4jrqg9kyndy12sgqh6rcw5c3-coreutils-8.32.drv",["out"]),("/gnu/store/cgqvxwlwvdybxlnj7j9a1nbyw98apnpw-python-ipython-genutils-0.1.0.drv",["out"]),("/gnu/store/dl99sx2bqksmmywk0pvxil3a1i6shykv-python-six-1.14.0.drv",["out"]),("/gnu/store/ib0ji4g0096qskdhgj6fdmlwyd18i0nh-python-prometheus-client-0.7.1.drv",["out"]),("/gnu/store/p3b21hd5ykwgfnbaqvvsmiglyqqc7m94-python-automat-20.2.0.drv",["out"]),("/gnu/store/pmxvg4xymw2mh4yxlzlbhbkri6bd9cdd-python-pyzmq-22.1.0.drv",["out"]),("/gnu/store/q899l6i4l0piy4v88l0s9csgl4w8b7xg-bash-5.0.16.drv",["out"]),("/gnu/store/qr3i59afbd5hrc4y5j9d23pgs7vm37zi-python-attrs-21.2.0.drv",["out"]),("/gnu/store/qxd9ngc6sbassv1wgyh557h6r6dci4ry-python-idna-2.10.drv",["out"]),("/gnu/store/qybsdmi94cyh42kqjxa5aym4sfbx190j-python-pyhamcrest-1.9.0-0.25fdc5f.drv",["out"]),("/gnu/store/r61c3am7jikwgbqh7hmn0mrlq94l7hyd-python-traitlets-4.3.3.drv",["out"]),("/gnu/store/rsw5dcynczjm5dr2x324bngc3gvfykqz-python-tornado-6.1.drv",["out"]),("/gnu/store/sf249r9rsgj3xvnrwn728bs46pb9iia7-python-ptyprocess-0.5.2.drv",["out"]),("/gnu/store/sqrdm4hmyr8y9pn88b3ncm254cn7zn97-node-10.24.0.drv",["out"]),("/gnu/store/vlhmdjkiwxrqzizalwvg8jn17j6mmm06-python-constantly-15.1.0.drv",["out"]),("/gnu/store/w9wc5939py5fp40mpd205yzifqx6zxv3-python-incremental-17.5.0.drv",["out"]),("/gnu/store/yz94jmahv05p2gjwx5c57660dbn7nbiq-python-decorator-4.3.0.drv",["out"]),("/gnu/store/ziz5svq4vfm1l2x767k2ywb8svkwdw0s-python-hyperlink-19.0.0.drv",["out"])],["/gnu/store/wahy9p8985vbcibwwjk3p73hxh82f90v-ca-certificate-bundle-builder","/gnu/store/xv5ylv9hxvs1wraw375b5g9jwy57vs8p-module-import"],"x86_64-linux","/gnu/store/0m0vd873jp61lcm4xa3ljdgx381qa782-guile-3.0.2/bin/guile",["--no-auto-compile","-L","/gnu/store/xv5ylv9hxvs1wraw375b5g9jwy57vs8p-module-import","-C","/gnu/store/08da6i38fzn9g36gyq443k03f84nk8rk-module-import-compiled","/gnu/store/wahy9p8985vbcibwwjk3p73hxh82f90v-ca-certificate-bundle-builder"],[("allowSubstitutes","0"),("guix properties","((type . profile-hook) (hook . ca-certificate-bundle))"),("out","/gnu/store/2df7jsi5h7ckdzalccrv7fgxgq5mnp2b-ca-certificate-bundle"),("preferLocalBuild","1")])

severity 50264 important

quit

Hi,

Lars-Dominik Braun <lars@6xq.net> skribis:

As discussed on IRC, I stumbled upon this bug while running:

./pre-inst-env guix system vm gnu/system/examples/desktop.tmpl

on commit 580984f2417853379d98ea927fd95c0f0fbe2c97.

The first attempt downloaded lots of substitutes and produced, as in

your case, a broken ca-certificate-bundle.drv: the ‘-builder’ script is

correct, but there are only ~80 derivation inputs instead of ~230.

The second run right after that produced the correct

ca-certificate-bundle.drv: same ‘-builder’ script, but all the

derivation inputs were there, including glibc-utf8-locales.drv.

I’ve tried to reproduce it running GC during or before the command, to

no avail so far. If someone has an easy and reliable way to reproduce

it, please share!

I don’t have a good hypothesis so far as to where things go wrong…

Ludo’.

On Tue, Sep 14, 2021 at 03:08:19PM +0200, Ludovic Courtès wrote:

I wonder if the problem began after the introduction of

bordeaux.guix.gnu.org, and if everyone who experiences the bug is using

both substitute servers.

I can't imagine what the actual problem is, but it does sound like the

dependency graph has somehow diverged...

Hi Leo,

I started the Guix daemon with only the CI substitute server enabled

explicitly, disabled local discovery, ran a `guix gc` and tried again. It

still fails with exactly the same issue.

Lars

Hi,

Lars-Dominik Braun <lars@6xq.net> skribis:

I’m wondering whether this could be due to

fa81971cbae85b39183ccf8f51e8d96ac88fb4ac somehow.

If you have a reliable way to test this hypothesis, that’d be great.

Thanks,

Ludo’.

Hi!

Ludovic Courtès <ludo@gnu.org> skribis:

Confirmed! In essence, ‘map/accumulate-builds’ would return the tail of

its ‘lst’ argument unprocessed when cutoff was reached (ouch!). Fixed:

https://git.savannah.gnu.org/cgit/guix.git/commit/?id=f72f4b48c6777da9465ab17baa6762476d6cb270

Let me know if you still experience fishy behavior!

Ludo’.

And for posterity, here’s the script I used to reproduce the problem:

it’d pick 10 packages at random and call ‘ca-certificate-bundle’ on them.

Since this bug depends on what’s in the store, I’d run it on my laptop,

which only contains a fraction of the 18K packages in Guix, so it would

reproduce the bug after a couple of iterations.

That, together with the inevitable ‘pk’ calls plus a bit of chance, voilà!

Ludo’.

And for posterity, here’s the script I used to reproduce the problem:

it’d pick 10 packages at random and call ‘ca-certificate-bundle’ on them.

Since this bug depends on what’s in the store, I’d run it on my laptop,

which only contains a fraction of the 18K packages in Guix, so it would

reproduce the bug after a couple of iterations.

That, together with the inevitable ‘pk’ calls plus a bit of chance, voilà!

Ludo’.

severity 50264 serious

quit