'guix substitute' creates files with incorrect names when not running in a UTF-8 locale

  • Done
  • quality assurance status badge
Details
3 participants
  • Brett Gilio
  • Ludovic Courtès
  • maxim.cournoyer
Owner
unassigned
Submitted by
Brett Gilio
Severity
important
B
B
Brett Gilio wrote on 3 Dec 2018 20:58
Invalid hash for NSS-Certs
(address . bug-guix@gnu.org)
87pnui8jrq.fsf@posteo.net
Generation 10 Dec 03 2018 11:42:41 (current)
guix 4f03aa2
branch: master
commit: 4f03aa23e805bd653de774e1d74ed2f50826899b

nss-certs-3.39 145KiB 417KiB/s 00:00 [##################] 100.0%

sha256 hash mismatch for /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39:
expected hash: 101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
actual hash: 08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s
substitution of /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 failed
L
L
Ludovic Courtès wrote on 5 Dec 2018 12:47
(name . Brett Gilio)(address . brettg@posteo.net)(address . 33603@debbugs.gnu.org)
87mupkyz2p.fsf@gnu.org
Hi Brett,

Brett Gilio <brettg@posteo.net> skribis:

Toggle quote (14 lines)
> Generation 10 Dec 03 2018 11:42:41 (current)
> guix 4f03aa2
> repository URL: https://git.savannah.gnu.org/git/guix.git
> branch: master
> commit: 4f03aa23e805bd653de774e1d74ed2f50826899b
>
> downloading from https://mirror.hydra.gnu.org/guix/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39...
> nss-certs-3.39 145KiB 417KiB/s 00:00 [##################] 100.0%
>
> sha256 hash mismatch for /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39:
> expected hash: 101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
> actual hash: 08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s
> substitution of /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 failed

The problem seems to be gone because I’m seeing the right hash here:

Toggle snippet (6 lines)
$ wget -q -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix hash -
101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
$ wget -q -O - https://mirror.hydra.gnu.org/xbj4fhad0lnz0ziflwi90gyqbls8ains.narinfo |grep Hash
NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla

Could you try again?

However berlin.guixsd.org is publishing the one with hash
08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s, and the difference
is an encoding bug:

Toggle snippet (7 lines)
$ diff -ru /tmp/nss-certs.{hydra,berlin}
Only in /tmp/nss-certs.hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
Only in /tmp/nss-certs.berlin/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
Only in /tmp/nss-certs.hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tanúsítvány:2.6.73.65.44.228.0.16.pem
Only in /tmp/nss-certs.berlin/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem


Ludo’.
M
M
maxim.cournoyer wrote on 8 Jan 2019 17:24
(name . Ludovic Courtès)(address . ludo@gnu.org)
874lajnmlp.fsf@kwak.i-did-not-set--mail-host-address--so-tickle-me
ludo@gnu.org (Ludovic Courtès) writes:

Toggle quote (25 lines)
> Hi Brett,
>
> Brett Gilio <brettg@posteo.net> skribis:
>
>> Generation 10 Dec 03 2018 11:42:41 (current)
>> guix 4f03aa2
>> repository URL: https://git.savannah.gnu.org/git/guix.git
>> branch: master
>> commit: 4f03aa23e805bd653de774e1d74ed2f50826899b
>>
>> downloading from https://mirror.hydra.gnu.org/guix/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39...
>> nss-certs-3.39 145KiB 417KiB/s 00:00 [##################] 100.0%
>>
>> sha256 hash mismatch for /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39:
>> expected hash: 101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
>> actual hash: 08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s
>> substitution of /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 failed
>
> The problem seems to be gone because I’m seeing the right hash here:
>
> $ wget -q -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix hash -
> 101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
> $ wget -q -O - https://mirror.hydra.gnu.org/xbj4fhad0lnz0ziflwi90gyqbls8ains.narinfo |grep Hash
> NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla

I got the failure while trying to reconfigure:

Toggle snippet (10 lines)
downloading from https://mirror.hydra.gnu.org/guix/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39...
nss-certs-3.39 145KiB 608KiB/s 00:00 [##################] 100.0%

sha256 hash mismatch for /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39:
expected hash: 101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
actual hash: 08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s
substitution of
/gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 failed

Maxim
L
L
Ludovic Courtès wrote on 11 Jan 2019 09:19
(address . maxim.cournoyer@gmail.com)
87d0p3a9oi.fsf@gnu.org
maxim.cournoyer@gmail.com skribis:

Toggle quote (38 lines)
> ludo@gnu.org (Ludovic Courtès) writes:
>
>> Hi Brett,
>>
>> Brett Gilio <brettg@posteo.net> skribis:
>>
>>> Generation 10 Dec 03 2018 11:42:41 (current)
>>> guix 4f03aa2
>>> repository URL: https://git.savannah.gnu.org/git/guix.git
>>> branch: master
>>> commit: 4f03aa23e805bd653de774e1d74ed2f50826899b
>>>
>>> downloading from https://mirror.hydra.gnu.org/guix/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39...
>>> nss-certs-3.39 145KiB 417KiB/s 00:00 [##################] 100.0%
>>>
>>> sha256 hash mismatch for /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39:
>>> expected hash: 101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
>>> actual hash: 08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s
>>> substitution of /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 failed
>>
>> The problem seems to be gone because I’m seeing the right hash here:
>>
>> $ wget -q -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 |gunzip -c |guix hash -
>> 101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
>> $ wget -q -O - https://mirror.hydra.gnu.org/xbj4fhad0lnz0ziflwi90gyqbls8ains.narinfo |grep Hash
>> NarHash: sha256:101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
>
> I got the failure while trying to reconfigure:
>
> downloading from https://mirror.hydra.gnu.org/guix/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39...
> nss-certs-3.39 145KiB 608KiB/s 00:00 [##################] 100.0%
>
> sha256 hash mismatch for /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39:
> expected hash: 101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
> actual hash: 08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s
> substitution of
> /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 failed

The wget commands above still give me the correct result, with hash
101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla.

Are you running Guix on a foreign distro? If so, could it be that
guix-daemon is effectively running in the C locale?

Thanks,
Ludo’.
M
M
Maxim Cournoyer wrote on 14 Jan 2019 03:33
(name . Ludovic Courtès)(address . ludo@gnu.org)
87muo4vuhg.fsf@gmail.com
Hello!

Ludovic Courtès <ludo@gnu.org> writes:
[...]
Toggle quote (6 lines)
> The wget commands above still give me the correct result, with hash
> 101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla.
>
> Are you running Guix on a foreign distro? If so, could it be that
> guix-daemon is effectively running in the C locale?

This is a good guess, and we've seen this very issue before. I am using
GuixSD. I had to use --fallback to work around it.

I've digged a little bit:
Toggle snippet (20 lines)
$ wget -q -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 | gunzip | guix archive -x /tmp/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39_from-hydra

$ guix hash -r /tmp/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39_from-hydra
101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla

$ guix build nss-certs
/gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39

$ guix hash -r /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39
08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s

$ diff -r /tmp/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39_from-hydra /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39
Only in /tmp/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39_from-hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
Only in /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
Only in /tmp/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39_from-hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tanúsítvány:2.6.73.65.44.228.0.16.pem
Only in
/gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39/etc/ssl/certs:
NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem

It's a rather old install (late 2016 -- but kept up-to-date, of course
:-)) so there might be remnants from the past? How could I verify in
which locale the guix-daemon is running?

Thanks!

Maxim
L
L
Ludovic Courtès wrote on 14 Jan 2019 09:48
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
878szny69n.fsf@gnu.org
Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (36 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
> [...]
>> The wget commands above still give me the correct result, with hash
>> 101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla.
>>
>> Are you running Guix on a foreign distro? If so, could it be that
>> guix-daemon is effectively running in the C locale?
>
> This is a good guess, and we've seen this very issue before. I am using
> GuixSD. I had to use --fallback to work around it.
>
> I've digged a little bit:
>
> $ wget -q -O - https://mirror.hydra.gnu.org/nar/gzip/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39 | gunzip | guix archive -x /tmp/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39_from-hydra
>
> $ guix hash -r /tmp/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39_from-hydra
> 101v69xp1qzw9v6pgmbhw7gfdaic8vvs4v5l567lx7f2mjp25rla
>
> $ guix build nss-certs
> /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39
>
> $ guix hash -r /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39
> 08ziz714diyfq2klxy1nc0nhr5wa2vd356n9vizlq913a7an9a9s
>
> $ diff -r /tmp/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39_from-hydra /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39
> Only in /tmp/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39_from-hydra/etc/ssl/certs: AC_Raíz_Certicámara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
> Only in /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39/etc/ssl/certs: AC_Ra?z_Certic?mara_S.A.:2.15.7.126.82.147.123.224.21.227.87.240.105.140.203.236.12.pem
> Only in /tmp/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39_from-hydra/etc/ssl/certs: NetLock_Arany_=Class_Gold=_F?tanúsítvány:2.6.73.65.44.228.0.16.pem
> Only in
> /gnu/store/xbj4fhad0lnz0ziflwi90gyqbls8ains-nss-certs-3.39/etc/ssl/certs:
> NetLock_Arany_=Class_Gold=_F?tan?s?tv?ny:2.6.73.65.44.228.0.16.pem
>
> It's a rather old install (late 2016 -- but kept up-to-date, of course
> :-)) so there might be remnants from the past? How could I verify in
> which locale the guix-daemon is running?

You could check /proc/$(pidof guix-daemon)/environ for variables like
‘LC_ALL’. And of course, you can see if ‘guix substitute’ emits the
infamous “can’t install locale” message. :-)

Regardless, I think ‘guix substitute’ should ideally be
locale-insensitive, or it should error out rather than produce files
with the wrong names.

HTH,
Ludo’.
L
L
Ludovic Courtès wrote on 14 Jan 2019 09:49
control message for bug #33603
(address . control@debbugs.gnu.org)
877ef7y67k.fsf@gnu.org
retitle 33603 'guix substitute' creates files with incorrect names when not running in a UTF-8 locale
L
L
Ludovic Courtès wrote on 14 Jan 2019 09:49
(address . control@debbugs.gnu.org)
875zury67g.fsf@gnu.org
severity 33603 important
M
M
Maxim Cournoyer wrote on 15 Jan 2019 06:03
Re: bug#33603: Invalid hash for NSS-Certs
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 33603@debbugs.gnu.org)
87imyqqzpk.fsf@gmail.com
Hello!

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (2 lines)
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

[...]

Toggle quote (12 lines)
>> It's a rather old install (late 2016 -- but kept up-to-date, of course
>> :-)) so there might be remnants from the past? How could I verify in
>> which locale the guix-daemon is running?
>
> You could check /proc/$(pidof guix-daemon)/environ for variables like
> ‘LC_ALL’. And of course, you can see if ‘guix substitute’ emits the
> infamous “can’t install locale” message. :-)
>
> Regardless, I think ‘guix substitute’ should ideally be
> locale-insensitive, or it should error out rather than produce files
> with the wrong names.

The only environment variable(s?) defined for the guix-daemon process on
that machine is:

Toggle snippet (7 lines)
$ pidof guix-daemon
270

sudo cat /proc/270/environ
GUIX_LOCPATH=/gnu/store/94k5w17z54w25lgp90czdqfv9m4hwzhq-glibc-utf8-locales-2.28/lib/localeLC_ALL=en_US.utf8

I'm not familiar with this systemfs structure, but shouldn't there be a
newline before the LC_ALL=en_US.utf8 variable assignment?

It's the same on a 2nd GuixSD machine.

Toggle snippet (5 lines)
$ sudo guix substitute --help
# Usage: guix substitute [OPTION]...
...

No infamous locale error here.

Not sure what happened here :-/

Maxim
L
L
Ludovic Courtès wrote on 15 Jan 2019 13:57
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 33603@debbugs.gnu.org)
87d0oy13jg.fsf@gnu.org
Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (29 lines)
> Hello!
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
> [...]
>
>>> It's a rather old install (late 2016 -- but kept up-to-date, of course
>>> :-)) so there might be remnants from the past? How could I verify in
>>> which locale the guix-daemon is running?
>>
>> You could check /proc/$(pidof guix-daemon)/environ for variables like
>> ‘LC_ALL’. And of course, you can see if ‘guix substitute’ emits the
>> infamous “can’t install locale” message. :-)
>>
>> Regardless, I think ‘guix substitute’ should ideally be
>> locale-insensitive, or it should error out rather than produce files
>> with the wrong names.
>
> The only environment variable(s?) defined for the guix-daemon process on
> that machine is:
>
> $ pidof guix-daemon
> 270
>
> sudo cat /proc/270/environ
> GUIX_LOCPATH=/gnu/store/94k5w17z54w25lgp90czdqfv9m4hwzhq-glibc-utf8-locales-2.28/lib/localeLC_ALL=en_US.utf8

This is perfect (see commit 7e4bc215098f334bc2a11737f2665dd4992fc2da,
which gave you this, fixing the issue we’re talking about on GuixSD.)

So I don’t think this machine has any problem. Perhaps nss-certs was
installed before the fix above?

Toggle quote (3 lines)
> I'm not familiar with this systemfs structure, but shouldn't there be a
> newline before the LC_ALL=en_US.utf8 variable assignment?

No, there are actually newlines, try:

cat /proc/270/environ | xargs -0 echo

HTH,
Ludo’.
M
M
Maxim Cournoyer wrote on 18 Jan 2019 05:59
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 33603@debbugs.gnu.org)
87muny36jp.fsf@gmail.com
Hi Ludovic!

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (2 lines)
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

[...]

Toggle quote (15 lines)
>> The only environment variable(s?) defined for the guix-daemon process on
>> that machine is:
>>
>> $ pidof guix-daemon
>> 270
>>
>> sudo cat /proc/270/environ
>> GUIX_LOCPATH=/gnu/store/94k5w17z54w25lgp90czdqfv9m4hwzhq-glibc-utf8-locales-2.28/lib/localeLC_ALL=en_US.utf8
>
> This is perfect (see commit 7e4bc215098f334bc2a11737f2665dd4992fc2da,
> which gave you this, fixing the issue we’re talking about on GuixSD.)
>
> So I don’t think this machine has any problem. Perhaps nss-certs was
> installed before the fix above?

Yes, that is likely the cause! I think I was using a system generation
from November to cope with some network instabilities I had at the
time. These have been resolved since :-).

Toggle quote (7 lines)
>> I'm not familiar with this systemfs structure, but shouldn't there be a
>> newline before the LC_ALL=en_US.utf8 variable assignment?
>
> No, there are actually newlines, try:
>
> cat /proc/270/environ | xargs -0 echo

Indeed. Thanks for continuously helping me to refine my knowledge ^^.

Shall we close this ticket, or did you want to keep it until we make
guix substitute fail when the locale of the daemon is not set to a UTF-8
based one?

Thank you!

Maxim
L
L
Ludovic Courtès wrote on 18 Jan 2019 17:53
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 33603-done@debbugs.gnu.org)
878szhsy90.fsf@gnu.org
Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (4 lines)
> Shall we close this ticket, or did you want to keep it until we make
> guix substitute fail when the locale of the daemon is not set to a UTF-8
> based one?

Commit 9fe3f11398e858f1d06120bd046cab506efc86dc does that.
Done!

Ludo’.
Closed
M
M
Maxim Cournoyer wrote on 19 Jan 2019 04:32
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 33603-done@debbugs.gnu.org)
874la5pbk9.fsf@gmail.com
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (13 lines)
> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Shall we close this ticket, or did you want to keep it until we make
>> guix substitute fail when the locale of the daemon is not set to a UTF-8
>> based one?
>
> Commit 9fe3f11398e858f1d06120bd046cab506efc86dc does that.
> Done!
>
> Ludo’.

That was quick! Well done! :-)

Maxim
Closed
?