'vault-fetch' does not follow redirects, preventing downloads from SWH

  • Done
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Ludovic Courtès
  • Simon Tournier
Owner
unassigned
Submitted by
Simon Tournier
Severity
normal
S
S
Simon Tournier wrote on 11 Feb 18:21 +0100
Failure when fallback to Disarchive+SWH
(address . bug-guix@gnu.org)
87mss6apd2.fsf@gmail.com
Hi,

Here an example where something is wrong:

+ Diarsarchive specificiation is ok.
+ SWH has it

But something is failing on Guix side.

Toggle snippet (26 lines)
$ guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv
/gnu/store/58m53a8ygffj4l5kpqhyxby1ylhri9dr-xscreensaver-6.04.tar.gz

$ guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv --check
The following derivation will be built:
/gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv
building /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv...

Starting download of /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz

[...]

Trying to use Disarchive to assemble /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz...
Retrieving Disarchive spec from https://disarchive.guix.gnu.org/sha256/787014b29f0c5793ecc2d93e1109a049ff48ab0c29b851dab34f683ceef6b152 ...
Assembling the directory xscreensaver-6.04
Downloading /gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz from Software Heritage...
SWH: object swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 could not be fetched from the vault
Could not resolve directory reference
failed to download "/gnu/store/123bm8wnld70qnajw5crdw7a7lhbpa0x-xscreensaver-6.04.tar.gz" from "https://www.jwz.org/xscreensaver/xscreensaver-6.04.tar.gz"
builder for `/gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv' failed to produce output path `/gnu/store/58m53a8ygffj4l5kpqhyxby1ylhri9dr-xscreensaver-6.04.tar.gz'
build of /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv failed
View build log at '/var/log/guix/drvs/mx/sfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv.gz'.
guix build: error: build of `/gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv' failed


Let’s do it manually:

1. fetch content and specification
2. assemble
3. compare the assembled with the one from content-addressed-mirrors


1.

Toggle snippet (8 lines)
$ wget -O /tmp/spec https://disarchive.guix.gnu.org/sha256/787014b29f0c5793ecc2d93e1109a049ff48ab0c29b851dab34f683ceef6b152
/tmp/spec [ <=> ] 3,25M 512KB/s in 6,6s

$ wget -O /tmp/content https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931/raw
/tmp/content 100%[===================>] 24,86M 530KB/s in 49s


2.

Toggle snippet (12 lines)
$ tar xf content
$ disarchive assemble swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 spec > xscreensaver-6.04.tar.gz
Assembling the directory xscreensaver-6.04
Checking xscreensaver-6.04 digest... ok
Assembling the tarball xscreensaver-6.04.tar
Checking xscreensaver-6.04.tar digest... ok
Assembling the Gzip file xscreensaver-6.04.tar.gz
Checking xscreensaver-6.04.tar.gz digest... ok
Writing result to output port


3.

Toggle snippet (7 lines)
$ disarchive assemble swh:1:dir:35e81cfbd7ca261e2ddebf372f38a4d1baa2f931 spec 2>/dev/null | guix hash -
0lmiyvp3qs2gngd53f191jmlizs9l04i2gnrqbn96mqckyr18w3q

$ guix hash $(guix build /gnu/store/mxsfblzy65v5y4gwrxqkihj7g4w5f5b1-xscreensaver-6.04.tar.gz.drv)
0lmiyvp3qs2gngd53f191jmlizs9l04i2gnrqbn96mqckyr18w3q

Cheers,
simon
S
S
Simon Tournier wrote on 11 Feb 19:11 +0100
(address . 69058@debbugs.gnu.org)
87frxyan2h.fsf@gmail.com
Hi,

Another example using Git. It is from v1.1.0.

Toggle snippet (24 lines)
$ guix time-machine -q --commit=v1.1.0 -- build -S redkite
guile: warning: failed to install locale
The following derivation will be built:
/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv
building /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to `/gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin:/gnu/store/jh17p4sns7dvbizwz58gdh953qpic144-tar-1.32/bin'
Initialized empty Git repository in /gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout/.git/
fatal: could not read Username for 'https://gitlab.com': No such device or address
Failed to do a shallow fetch; retrying a full fetch...
fatal: could not read Username for 'https://gitlab.com': No such device or address
git-fetch: '/gnu/store/xzhkhidb2kqwvpv8b8zaqzgd3gcjs9nn-git-minimal-2.26.0/bin/git fetch origin' failed with exit code 128
Trying content-addressed mirror at berlin.guixsd.org...
Trying content-addressed mirror at berlin.guixsd.org...
Trying to download from Software Heritage...
SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
SWH: directory d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
builder for `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed to produce output path `/gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout'
build of /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv failed
View build log at '/var/log/guix/drvs/x4/hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv.gz'.
guix build: error: build of `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed


From its definition the package reads:

Toggle snippet (7 lines)
(define-public redkite
[...]
(sha256
(base32
"1747w1kg8y9jbl11xi018d85dm38xk7843pz26sh0k5fdv87a10q"))))

Then if I manually download it:

Toggle snippet (9 lines)
$ wget -O /tmp/content https://archive.softwareheritage.org/api/1/vault/flat/swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728/raw

[...]
/tmp/content 100%[===================>] 129,38K 555KB/s in 0,2s
2024-02-11 19:00:33 (555 KB/s) - ‘/tmp/content’ saved [132488/132488]

$ tar xf content

And we get the expected value:

Toggle snippet (5 lines)
$ guix hash -rx swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728
1747w1kg8y9jbl11xi018d85dm38xk7843pz26sh0k5fdv87a10q


Cheers,
simon
L
L
Ludovic Courtès wrote on 12 Feb 11:28 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 69058@debbugs.gnu.org)
87r0hif03m.fsf@gnu.org
Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (24 lines)
> Another example using Git. It is from v1.1.0.
>
> $ guix time-machine -q --commit=v1.1.0 -- build -S redkite
> guile: warning: failed to install locale
> The following derivation will be built:
> /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv
> building /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv...
> guile: warning: failed to install locale
> environment variable `PATH' set to `/gnu/store/i2cdl0hvrml8hjdqplqln8didnvxkgp5-gzip-1.10/bin:/gnu/store/jh17p4sns7dvbizwz58gdh953qpic144-tar-1.32/bin'
> Initialized empty Git repository in /gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout/.git/
> fatal: could not read Username for 'https://gitlab.com': No such device or address
> Failed to do a shallow fetch; retrying a full fetch...
> fatal: could not read Username for 'https://gitlab.com': No such device or address
> git-fetch: '/gnu/store/xzhkhidb2kqwvpv8b8zaqzgd3gcjs9nn-git-minimal-2.26.0/bin/git fetch origin' failed with exit code 128
> Trying content-addressed mirror at berlin.guixsd.org...
> Trying content-addressed mirror at berlin.guixsd.org...
> Trying to download from Software Heritage...
> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
> SWH: directory d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
> builder for `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed to produce output path `/gnu/store/y7j54zw1fwhl95534zsjl6rl1rfp7z82-redkite-0.8.0-checkout'
> build of /gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv failed
> View build log at '/var/log/guix/drvs/x4/hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv.gz'.
> guix build: error: build of `/gnu/store/x4hcwpmb3p8llbhjzxlfggpkpqc7n5ng-redkite-0.8.0-checkout.drv' failed

I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:

Toggle snippet (6 lines)
scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
$5 = #f

The bug is that the Vault now returns 302 (redirect) in this case, which
is similar to the bug I fixed in https://issues.guix.gnu.org/68741.

So I’ll have to push that patch series and add a patch for ‘vault-fetch’
on top of it.

Ludo’.
L
L
Ludovic Courtès wrote on 12 Feb 11:58 +0100
control message for bug #69058
(address . control@debbugs.gnu.org)
87plx2eypn.fsf@gnu.org
retitle 69058 'vault-fetch' does not follow redirects, preventing downloads from SWH
quit
L
L
Ludovic Courtès wrote on 12 Feb 12:24 +0100
Re: bug#69058: Failure when fallback to Disarchive+SWH
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 69058-done@debbugs.gnu.org)
87h6ieexhf.fsf@gnu.org
Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

Toggle quote (10 lines)
> I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:
>
> scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
> SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
> $5 = #f
>
> The bug is that the Vault now returns 302 (redirect) in this case, which
> is similar to the bug I fixed in <https://issues.guix.gnu.org/68741>.

In fact it’s the exact same problem.

Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.

Thanks,
Ludo’.
Closed
S
S
Simon Tournier wrote on 12 Feb 13:59 +0100
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 69058-done@debbugs.gnu.org)
87plx17s9c.fsf@gmail.com
Hi Ludo,

On lun., 12 févr. 2024 at 12:24, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
Toggle quote (16 lines)
> Ludovic Courtès <ludovic.courtes@inria.fr> skribis:
>
>> I can reproduce it with c4372f7ebfac17236299346fe60c6ca125336b1f:
>>
>> scheme@(guile-user)> (swh-download "https://gitlab.com/geontime/redkite.git" "v0.8.0" "/tmp/redkite")
>> SWH: found revision 68276b9b88491198c33b03c4188dfe7d6e5dbf2a with directory at 'https://archive.softwareheritage.org/api/1/directory/d14c0eb6801bca920d1e1baac7a4b9e2a0589728/'
>> SWH: object swh:1:dir:d14c0eb6801bca920d1e1baac7a4b9e2a0589728 could not be fetched from the vault
>> $5 = #f
>>
>> The bug is that the Vault now returns 302 (redirect) in this case, which
>> is similar to the bug I fixed in <https://issues.guix.gnu.org/68741>.
>
> In fact it’s the exact same problem.
>
> Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.

Cool! Thanks.

However, it means that it stays broken for all commits before this fix,
right? Somehow, if only SWH is available, then

guix time-machine -C channels.scm -- shell -m manifest.scm

will fail for all channels older than today. Right?

For instance, redoing your ReScience paper from 2020 using only SWH as I
did [1] some months ago is now broken (because a change on SWH side).

Well, I am fine to keep this bug closed but for the posterity we need
something like:

guix swh get -C channels.scm -m manifest.scm

as we discussed elsewhere (off-list IIRC :-)).


Cheers,
simon
Closed
L
L
Ludovic Courtès wrote on 12 Feb 14:15 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 69058-done@debbugs.gnu.org)
875xytg6y3.fsf@inria.fr
Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (11 lines)
>> Fixed in 1610a632d4b3097282d18af27ff3e9e178d7dfcb.
>
> Cool! Thanks.
>
> However, it means that it stays broken for all commits before this fix,
> right? Somehow, if only SWH is available, then
>
> guix time-machine -C channels.scm -- shell -m manifest.scm
>
> will fail for all channels older than today. Right?

Mostly, yes (“mostly” because derivations using “builtin:git-download”
will automatically be fixed once we’ve updated the ‘guix’ package, but
that only helps if you’re traveling to after September 2023 (commit
13b0cf85eb31e1b1ea674debbbfb0f35a32d1374).

Toggle quote (3 lines)
> For instance, redoing your ReScience paper from 2020 using only SWH as I
> did [1] some months ago is now broken (because a change on SWH side).

Indeed. :-/

That’s a real bummer, but that’s the nature of software: we cannot
retroactively fix bugs.

Toggle quote (5 lines)
> Well, I am fine to keep this bug closed but for the posterity we need
> something like:
>
> guix swh get -C channels.scm -m manifest.scm

Exactly! We have the demonstration of just how important that is.

Ludo’.
Closed
S
S
Simon Tournier wrote on 12 Feb 15:04 +0100
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 69058-done@debbugs.gnu.org)
CAJ3okZ1zMkCDpRhQRB=ZQegObKSyLFjCAmV7=S7ajpEA3PBJUA@mail.gmail.com
Toggle quote (3 lines)
> That’s a real bummer, but that’s the nature of software: we cannot
> retroactively fix bugs.

The nature of service and not software appears to me more correct. :-)

(Here it is not the retroactive fix of a bug but the introduction of a
non-backward compatible change.)

Cheers,
simon
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 69058@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 69058
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch