'vault-fetch' does not follow redirects, preventing downloads from SWH

  • Done
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Ludovic Courtès
  • Simon Tournier
Owner
unassigned
Submitted by
Simon Tournier
Severity
normal
S
S
Simon Tournier wrote on 11 Feb 18:21 +0100
Failure when fallback to Disarchive+SWH
(address . bug-guix@gnu.org)
87mss6apd2.fsf@gmail.com
Hi,

Here an example where something is wrong:

+ Diarsarchive specificiation is ok.
+ SWH has it

But something is failing on Guix side.

Toggle snippet (26 lines)
$ guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv
/gnu/store/58m53a8ygffj4l5kpqhyxby1ylhri9dr-xscreensaver-6.04.tar.gz

$ guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv --check
The following derivation will be built:
/gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv
building /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv...

Starting download of /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz

[...]

Trying to use Disarchive to assemble /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz...
Retrieving Disarchive spec from https://disarchive.guix.gnu.org/sha256/787014b29f0c5793ecc2d93e1109a049ff48ab0c29b851dab34f683ceef6b152 ...
Assembling the directory xscreensaver-6.04
Downloading /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz from Software Heritage...
SWH: object swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 could not be fetched from the vault
Could not resolve directory reference
failed to download "/gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz" from "https://www.jwz.org/xscreensaver/xscreensaver-6.04.tar.gz"
builder for `/gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv' failed to produce output path `/gnu/store/58m53a8ygffj4l5kpqhyxby1ylhri9dr-xscreensaver-6.04.tar.gz'
build of /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv failed
View build log at '/var/log/guix/drvs/mx/sfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv.gz'.
guix build: error: build of `/gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv' failed


Let’s do it manually:

1. fetch content and specification
2. assemble
3. compare the assembled with the one from content-addressed-mirrors


1.

Toggle snippet (8 lines)
$ wget -O /tmp/spec https://disarchive.guix.gnu.org/sha256/787014b29f0c5793ecc2d93e1109a049ff48ab0c29b851dab34f683ceef6b152
/tmp/spec [ <=> ] 3,25M 512KB/s in 6,6s

$ wget -O /tmp/content https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931/raw
/tmp/content 100%[===================>] 24,86M 530KB/s in 49s


2.

Toggle snippet (12 lines)
$ tar xf content
$ disarchive assemble swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 spec > xscreensaver-6.04.tar.gz
Assembling the directory xscreensaver-6.04
Checking xscreensaver-6.04 digest... ok
Assembling the tarball xscreensaver-6.04.tar
Checking xscreensaver-6.04.tar digest... ok
Assembling the Gzip file xscreensaver-6.04.tar.gz
Checking xscreensaver-6.04.tar.gz digest... ok
Writing result to output port


3.

Toggle snippet (7 lines)
$ disarchive assemble swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 spec 2>/dev/null | guix hash -
0lmiyvp3qs2gngd53f191jmlizs9l04i2gnrqbn96mqckyr18w3q

$ guix hash $(guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv)
0lmiyvp3qs2gngd53f191jmlizs9l04i2gnrqbn96mqckyr18w3q

Cheers,
simon
S
S
Simon Tournier wrote on 11 Feb 19:11 +0100
(address . 69058@debbugs.gnu.org)
87frxyan2h.fsf@gmail.com
Hi,

Another example using Git. It is from v1.1.0.

Toggle snippet (24 lines)
$ guix time-machine -q --commit=v1.1.0 -- build -S redkite
guile: warning: failed to install locale
The following derivation will be built:
/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv
building /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to `/gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin:/gnu/store/jh17p4sns7dvbizwz58gdh953qpic144-tar-1.32/bin'
Initialized empty Git repository in /gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout/.git/
fatal: could not read Username for 'https://gitlab.com': No such device or address
Failed to do a shallow fetch; retrying a full fetch...
fatal: could not read Username for 'https://gitlab.com': No such device or address
git-fetch: '/gnu/store/xzhkhidb2kqwvpv8b8zaqzgd3gcjs9nn-git-minimal-2.26.0/bin/git fetch origin' failed with exit code 128
Trying content-addressed mirror at berlin.guixsd.org...
Trying content-addressed mirror at berlin.guixsd.org...
Trying to download from Software Heritage...
SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
SWH: directory d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
builder for `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed to produce output path `/gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout'
build of /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv failed
View build log at '/var/log/guix/drvs/x4/hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv.gz'.
guix build: error: build of `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed


From its definition the package reads:

Toggle snippet (7 lines)
(define-public redkite
[...]
(sha256
(base32
"1747w1kg8y9jbl11xi018d85dm38xk7843pz26sh0k5fdv87a10q"))))

Then if I manually download it:

Toggle snippet (9 lines)
$ wget -O /tmp/content https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728/raw

[...]
/tmp/content 100%[===================>] 129,38K 555KB/s in 0,2s
2024-02-11 19:00:33 (555 KB/s) - ‘/tmp/content’ saved [132488/132488]

$ tar xf content

And we get the expected value:

Toggle snippet (5 lines)
$ guix hash -rx swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728
1747w1kg8y9jbl11xi018d85dm38xk7843pz26sh0k5fdv87a10q


Cheers,
simon
L
L
Ludovic Courtès wrote on 12 Feb 11:28 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 69058@debbugs.gnu.org)
87r0hif03m.fsf@gnu.org
Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (24 lines)
> Another example using Git. It is from v1.1.0.
>
> $ guix time-machine -q --commit=v1.1.0 -- build -S redkite
> guile: warning: failed to install locale
> The following derivation will be built:
> /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv
> building /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv...
> guile: warning: failed to install locale
> environment variable `PATH' set to `/gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin:/gnu/store/jh17p4sns7dvbizwz58gdh953qpic144-tar-1.32/bin'
> Initialized empty Git repository in /gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout/.git/
> fatal: could not read Username for 'https://gitlab.com': No such device or address
> Failed to do a shallow fetch; retrying a full fetch...
> fatal: could not read Username for 'https://gitlab.com': No such device or address
> git-fetch: '/gnu/store/xzhkhidb2kqwvpv8b8zaqzgd3gcjs9nn-git-minimal-2.26.0/bin/git fetch origin' failed with exit code 128
> Trying content-addressed mirror at berlin.guixsd.org...
> Trying content-addressed mirror at berlin.guixsd.org...
> Trying to download from Software Heritage...
> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
> SWH: directory d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
> builder for `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed to produce output path `/gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout'
> build of /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv failed
> View build log at '/var/log/guix/drvs/x4/hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv.gz'.
> guix build: error: build of `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed

I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:

Toggle snippet (6 lines)
scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
$5 = #f

The bug is that the Vault now returns 302 (redirect) in this case, which
is similar to the bug I fixed in https://issues.guix.gnu.org/68741.

So I’ll have to push that patch series and add a patch for ‘vault-fetch’
on top of it.

Ludo’.
L
L
Ludovic Courtès wrote on 12 Feb 11:58 +0100
control message for bug #69058
(address . control@debbugs.gnu.org)
87plx2eypn.fsf@gnu.org
retitle 69058 'vault-fetch' does not follow redirects, preventing downloads from SWH
quit
L
L
Ludovic Courtès wrote on 12 Feb 12:24 +0100
Re: bug#69058: Failure when fallback to Disarchive+SWH
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 69058-done@debbugs.gnu.org)
87h6ieexhf.fsf@gnu.org
Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

Toggle quote (10 lines)
> I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:
>
> scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
> SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
> $5 = #f
>
> The bug is that the Vault now returns 302 (redirect) in this case, which
> is similar to the bug I fixed in <https://issues.guix.gnu.org/68741>.

In fact it’s the exact same problem.

Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.

Thanks,
Ludo’.
Closed
S
S
Simon Tournier wrote on 12 Feb 13:59 +0100
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 69058-done@debbugs.gnu.org)
87plx17s9c.fsf@gmail.com
Hi Ludo,

On lun., 12 févr. 2024 at 12:24, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
Toggle quote (16 lines)
> Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
>
>> I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:
>>
>> scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
>> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
>> SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
>> $5 = #f
>>
>> The bug is that the Vault now returns 302 (redirect) in this case, which
>> is similar to the bug I fixed in <https://issues.guix.gnu.org/68741>.
>
> In fact it’s the exact same problem.
>
> Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.

Cool! Thanks.

However, it means that it stays broken for all commits before this fix,
right? Somehow, if only SWH is available, then

guix time-machine -C channels.scm -- shell -m manifest.scm

will fail for all channels older than today. Right?

For instance, redoing your ReScience paper from 2020 using only SWH as I
did [1] some months ago is now broken (because a change on SWH side).

Well, I am fine to keep this bug closed but for the posterity we need
something like:

guix swh get -C channels.scm -m manifest.scm

as we discussed elsewhere (off-list IIRC :-)).


Cheers,
simon
Closed
L
L
Ludovic Courtès wrote on 12 Feb 14:15 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 69058-done@debbugs.gnu.org)
875xytg6y3.fsf@inria.fr
Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (11 lines)
>> Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.
>
> Cool! Thanks.
>
> However, it means that it stays broken for all commits before this fix,
> right? Somehow, if only SWH is available, then
>
> guix time-machine -C channels.scm -- shell -m manifest.scm
>
> will fail for all channels older than today. Right?

Mostly, yes (“mostly” because derivations using “builtin:git-download”
will automatically be fixed once we’ve updated the ‘guix’ package, but
that only helps if you’re traveling to after September 2023 (commit
13b0cf85eb31e1b1ea674debbbfb0f35a32d1374).

Toggle quote (3 lines)
> For instance, redoing your ReScience paper from 2020 using only SWH as I
> did [1] some months ago is now broken (because a change on SWH side).

Indeed. :-/

That’s a real bummer, but that’s the nature of software: we cannot
retroactively fix bugs.

Toggle quote (5 lines)
> Well, I am fine to keep this bug closed but for the posterity we need
> something like:
>
> guix swh get -C channels.scm -m manifest.scm

Exactly! We have the demonstration of just how important that is.

Ludo’.
Closed
S
S
Simon Tournier wrote on 12 Feb 15:04 +0100
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 69058-done@debbugs.gnu.org)
CAJ3okZ1zMkCDpRhQRB=ZQegObKSyLFjCAmV7=S7ajpEA3PBJUA@mail.gmail.com
Toggle quote (3 lines)
> That’s a real bummer, but that’s the nature of software: we cannot
> retroactively fix bugs.

The nature of service and not software appears to me more correct. :-)

(Here it is not the retroactive fix of a bug but the introduction of a
non-backward compatible change.)

Cheers,
simon
Closed
?