downloading from software archives as default even if version is wrong

  • Open
  • quality assurance status badge
Details
5 participants
  • Leo Famulari
  • Andy Tai
  • Maxime Devos
  • Tobias Geerinckx-Rice
  • raingloom
Owner
unassigned
Submitted by
Andy Tai
Severity
normal
A
A
Andy Tai wrote on 18 Dec 2021 00:40
(address . bug-guix@gnu.org)
CAJsg1E8UPfxuyErnP_32H0Qr71+eeJsvFNLRWZotw=uTFdDJsg@mail.gmail.com
I was trying to update a package. I updated the package definition:

---patch--
Toggle diff (78 lines)
diff --git a/gnu/packages/mono.scm b/gnu/packages/mono.scm
index 6a7afbafbf..9eeb46ef16 100644
--- a/gnu/packages/mono.scm
+++ b/gnu/packages/mono.scm
@@ -38,7 +38,7 @@ (define-module (gnu packages mono)
(define-public mono
(package
(name "mono")
- (version "4.4.1.0")
+ (version "4.8.1")
(source (origin
(method url-fetch)
(uri (string-append
--end of patch--

which at this time shall fail as I have not updated the hash checksum. If I do

./pre-inst-env guix build -k -K mono

that actually succeeded but it shall have failed.

This log fragment show what happened:

----log fragment--
The following derivations will be built:
/gnu/store/cdqqdalqwl3m9bhnfyb0v6nfkzvfj0x1-mono-4.8.1.drv
/gnu/store/qs8jgc33jpwwmzq7ppdiv49qll6q32sl-mono-4.8.1.tar.xz.drv
/gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv
building /gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv...

Starting download of
/gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
From http://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2...
following redirection to
`https://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2'...
download failed
"https://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2"
404 "Not Found"

Starting download of
/gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
From https://ci.guix.gnu.org/file/mono-4.8.1.tar.bz2/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspc
fbnn37cphn...
download failed
"https://ci.guix.gnu.org/file/mono-4.8.1.tar.bz2/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y
90dcjc3dkspcfbnn37cphn" 404 "Not Found"

Starting download of
/gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
From https://tarballs.nixos.org/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspcfbnn37cphn...
download failed
"https://tarballs.nixos.org/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspcfbnn37cph
n" 404 "Not Found"

Starting download of
/gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
From https://archive.softwareheritage.org/api/1/content/sha256:165e7686b5cbb1ab671b4cb2069207999c3d700
44002190b6ec84bb1fdf62b4a/raw/...
downloading from
https://archive.softwareheritage.org/api/1/content/sha256:165e7686b5cbb1ab671b4cb2069
207999c3d70044002190b6ec84bb1fdf62b4a/raw/ ...
............
successfully built
/gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv
building /gnu/store/qs8jgc33jpwwmzq7ppdiv49qll6q32sl-mono-4.8.1.tar.xz.drv...
mono-4.4.1/
mono-4.4.1/test-driver
mono-4.4.1/support/
mono-4.4.1/support/syslog.c
mono-4.4.1/support/supportw.h
....

--end of log fratgment--


as seen above, somehow the old version was downloaded from a cached
copy at softwareheritage archives, and it proceeds to build. This
should not proceed but fail for wrong checksum.
M
M
Maxime Devos wrote on 18 Dec 2021 00:58
6c18852ca39e53a40cf6cdf0f3c661354d762907.camel@telenet.be
Andy Tai schreef op vr 17-12-2021 om 15:40 [-0800]:
Toggle quote (11 lines)
> @@ -38,7 +38,7 @@ (define-module (gnu packages mono)
>  (define-public mono
>    (package
>      (name "mono")
> -    (version "4.4.1.0")
> +    (version "4.8.1")
>      (source (origin
>                (method url-fetch)
>                (uri (string-append
> --end of patch--

You forgot to update the sha256 hash. You can use
"guix download https://the-website/mono-4.8.1.tar.bz2"to determine the
hash. Or toggle a single character & look at the hash mismatch line.

Does that work for you?

[...]
Toggle quote (4 lines)
> as seen above, somehow the old version was downloaded from a cached
> copy at softwareheritage archives, and it proceeds to build. This
> should not proceed but fail for wrong checksum.

SWH isn't a cache, its an archive that keeps everything forever.

Guix has no way to determine if upstream is doing unspeakable things to
their tarballs (*) and hence needs to use SWH, or if someone forgot to
change the hash on guix. This has been noticed in the past, without
any ideas on how to somehow teach guix to determine the case.

(*) e.g. in-place modification, which changes the hash

Greetings,
Maxime.
L
L
Leo Famulari wrote on 18 Dec 2021 01:06
(name . Andy Tai)(address . lichengtai@gmail.com)(address . 52591@debbugs.gnu.org)
Yb0mFsOxRN5sBlAJ@jasmine.lan
On Fri, Dec 17, 2021 at 03:40:45PM -0800, Andy Tai wrote:
Toggle quote (4 lines)
> as seen above, somehow the old version was downloaded from a cached
> copy at softwareheritage archives, and it proceeds to build. This
> should not proceed but fail for wrong checksum.

This can happen with the Nix content-addressed archive fallback that we
use, too.

Basically, when the normal URLs fail, Guix queries these
content-addressed archives by content. That is, by their hash.

The version number is considered to be metadata, not content, and so the
download "succeeds".

Since the source origins are named by their hash, this is the correct
behaviour, although there is some room for error, as you've found.

Do you have any ideas about how to improve things?

We taught Guix to respect the GUIX_DOWNLOAD_FALLBACK_TEST [0]
environment variable, which controls how Guix uses these fallback
archives.

I recommend setting it to "none" if you are doing package development in
order to avoid this pitfall. Otherwise, these archives are helpful for
using Guix, since sources do disappear upstream, URLs change, etc, and
we'd like for old versions of Guix to be usable.

[0]
R
R
raingloom wrote on 18 Dec 2021 01:07
(name . Andy Tai)(address . lichengtai@gmail.com)(address . 52591@debbugs.gnu.org)
20211218010723.53203615@riseup.net
On Fri, 17 Dec 2021 15:40:45 -0800
Andy Tai <lichengtai@gmail.com> wrote:

Toggle quote (89 lines)
> I was trying to update a package. I updated the package definition:
>
> ---patch--
> diff --git a/gnu/packages/mono.scm b/gnu/packages/mono.scm
> index 6a7afbafbf..9eeb46ef16 100644
> --- a/gnu/packages/mono.scm
> +++ b/gnu/packages/mono.scm
> @@ -38,7 +38,7 @@ (define-module (gnu packages mono)
> (define-public mono
> (package
> (name "mono")
> - (version "4.4.1.0")
> + (version "4.8.1")
> (source (origin
> (method url-fetch)
> (uri (string-append
> --end of patch--
>
> which at this time shall fail as I have not updated the hash
> checksum. If I do
>
> ./pre-inst-env guix build -k -K mono
>
> that actually succeeded but it shall have failed.
>
> This log fragment show what happened:
>
> ----log fragment--
> The following derivations will be built:
> /gnu/store/cdqqdalqwl3m9bhnfyb0v6nfkzvfj0x1-mono-4.8.1.drv
> /gnu/store/qs8jgc33jpwwmzq7ppdiv49qll6q32sl-mono-4.8.1.tar.xz.drv
> /gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv
> building
> /gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv...
>
> Starting download of
> /gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
> From
> http://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2...
> following redirection to
> `https://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2'...
> download failed
> "https://download.mono-project.com/sources/mono/mono-4.8.1.tar.bz2"
> 404 "Not Found"
>
> Starting download of
> /gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
> From
> https://ci.guix.gnu.org/file/mono-4.8.1.tar.bz2/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspc
> fbnn37cphn... download failed
> "https://ci.guix.gnu.org/file/mono-4.8.1.tar.bz2/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y
> 90dcjc3dkspcfbnn37cphn" 404 "Not Found"
>
> Starting download of
> /gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
> From
> https://tarballs.nixos.org/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspcfbnn37cphn...
> download failed
> "https://tarballs.nixos.org/sha256/0jibyvyv2jy8dq5ij0j00iq3v74r0y90dcjc3dkspcfbnn37cph
> n" 404 "Not Found"
>
> Starting download of
> /gnu/store/l6309vrm95cwpmcwgjj5xgdddp6vfly5-mono-4.8.1.tar.bz2
> From
> https://archive.softwareheritage.org/api/1/content/sha256:165e7686b5cbb1ab671b4cb2069207999c3d700
> 44002190b6ec84bb1fdf62b4a/raw/... downloading from
> https://archive.softwareheritage.org/api/1/content/sha256:165e7686b5cbb1ab671b4cb2069
> 207999c3d70044002190b6ec84bb1fdf62b4a/raw/ ...
> ............
> successfully built
> /gnu/store/6aih7gpx3p1dpsbdhkri7mk3qqnqkk9x-mono-4.8.1.tar.bz2.drv
> building
> /gnu/store/qs8jgc33jpwwmzq7ppdiv49qll6q32sl-mono-4.8.1.tar.xz.drv...
> mono-4.4.1/ mono-4.4.1/test-driver
> mono-4.4.1/support/
> mono-4.4.1/support/syslog.c
> mono-4.4.1/support/supportw.h
> ....
>
> --end of log fratgment--
>
>
> as seen above, somehow the old version was downloaded from a cached
> copy at softwareheritage archives, and it proceeds to build. This
> should not proceed but fail for wrong checksum.
>
>
>

This is somewhat unfortunate, but also kind of expected behaviour.
Fixed output derivations don't have version information as far as I
know and part of the usefulness of cryptographic checksums on them is
that they can be downloaded from anywhere, even when their original
servers go down.
There isn't really any practical way I can see for testing if the
checksum is correct when the URL is wrong. How should Guix know what
file it should download?
I think the best workflow is to `use guix refresh -u` instead of
manually updating the package, but sadly that doesn't always work,
although I think the git based updater was merged recently, which
should improve your chances of success.
T
T
Tobias Geerinckx-Rice wrote on 18 Dec 2021 01:06
(name . Maxime Devos)(address . maximedevos@telenet.be)
874k76n55q.fsf@nckx
Maxime Devos ???
Toggle quote (2 lines)
> You forgot to update the sha256 hash.

That's it, in a nutshell!

Toggle quote (3 lines)
> which at this time shall fail as I have not updated the hash
> checksum.

No, that's the reason it *succeeds*!

Guix is mostly content-addressed, meaning that the hash identifies
a file. If you don't change the hash, you're not asking for a new
file, by definition. URLs are like directions (hence: ‘locator’),
not identifiers.

You can easily generate a bogus-but-valid hash by negating the
first digit, which (in nix-base32 at least), can only be either 0
or 1. Guix will then print the correct hash of whatever it found
at the URI you specified.

OK to close as NOTABUG?

Kind regards,

T G-R
-----BEGIN PGP SIGNATURE-----

iIIEARYKACsWIQT12iAyS4c9C3o4dnINsP+IT1VteQUCYb0nIQ0cbWVAdG9iaWFz
LmdyAAoJEA2w/4hPVW15X7YA9iWFSpcWnB2t2fBzD7efAPDBM/Zbe6YORbDi/GUx
P3sBAMQlRih69X31RVeJ8KEtuZ6uA3gbOF6oMrmakLZ9MPIA
=Hm/z
-----END PGP SIGNATURE-----

A
A
Andy Tai wrote on 18 Dec 2021 01:23
(name . Tobias Geerinckx-Rice)(address . me@tobias.gr)
CAJsg1E_qm3E6ORy=7=_n5q+Th+peVkTDUD2nLuok1Gr5ox2E2Q@mail.gmail.com
On Fri, Dec 17, 2021 at 4:10 PM Tobias Geerinckx-Rice <me@tobias.gr> wrote:
Toggle quote (1 lines)
> OK to close as NOTABUG?
sure
L
L
Leo Famulari wrote on 18 Dec 2021 02:51
(name . Andy Tai)(address . lichengtai@gmail.com)
Yb0+stbMerZz7Muy@jasmine.lan
On Fri, Dec 17, 2021 at 04:23:11PM -0800, Andy Tai wrote:
Toggle quote (4 lines)
> On Fri, Dec 17, 2021 at 4:10 PM Tobias Geerinckx-Rice <me@tobias.gr> wrote:
> > OK to close as NOTABUG?
> sure

Wait! :)

Although I agree that this behaviour is not a bug, it's something that
many Guix contributors discover independently and find confusing, or
even distressing.

And we sometimes receive patches that include the wrong source hash or
wrong URL because of similar issues; sometimes these mistakes are even
committed to guix.git.

Maybe there is *something* we can do to improve this experience, if only
a section in the manual or a tweak to the UI.

Any ideas?
T
T
Tobias Geerinckx-Rice wrote on 18 Dec 2021 02:57
(name . Leo Famulari)(address . leo@famulari.name)
87v8zmllno.fsf@nckx
Leo Famulari ???
Toggle quote (4 lines)
> Maybe there is *something* we can do to improve this experience,
> if only
> a section in the manual or a tweak to the UI.

OK! Let's keep it open for discussion.

Kind regards,

T G-R
-----BEGIN PGP SIGNATURE-----

iIMEARYKACsWIQT12iAyS4c9C3o4dnINsP+IT1VteQUCYb1AGw0cbWVAdG9iaWFz
LmdyAAoJEA2w/4hPVW15dCsA/0ldy8j4n9UeZbm885OOFM4smpQ4ZbYe4a55PSkd
tkzxAQChb3bZYa3XDLEymwrHJAslBLFRkRwA+5Zen0aIStU0CQ==
=zGDV
-----END PGP SIGNATURE-----

?