SWH fallback fails (git-fetch)

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • zimoun
Owner
unassigned
Submitted by
zimoun
Severity
important
Z
Z
zimoun wrote on 9 Jul 2020 02:36
(address . bug-guix@gnu.org)
86imexcymw.fsf@gmail.com
Dear,

I think the fallback with SWH is broken. Here an example to reproduce.

Toggle snippet (8 lines)
cd /tmp/
tar -xvf $(guix build -S hello)
cd hello-2.10
git init
git add *
git commit -m 'init'

Then I push this fresh repo to my own GitHub account. Because it is
easy to switch between public and private state option, which mimics
upstream loss.

Let create the well-known "hello" recipe, which is simply a copy/paste
of (gnu packages base).

Toggle snippet (28 lines)
(define-module (hello)
#:use-module (guix packages)
#:use-module (guix build-system gnu)
#:use-module (guix git-download)
#:use-module (guix licenses))

(define-public hi
(package
(name "hi")
(version "2.10")
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/zimoun/hello-example.git")
(commit "e1eefd033b8a2c4c81babc6fde08ebb116c6abb8")))
(sha256
(base32
"1im1gglfm4k10bh4mdaqzmx3lm3kivnsmxrvl6vyvmfqqzljq75l"))))
(build-system gnu-build-system)
(synopsis "Hello, GNU world: An example GNU package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+)))

For the sanity of the test, "guix build -L . hi" builds correctly and
returns the item: /gnu/store/jn8d031zx4znxy7s5zhj4dbr6xjsfq9v-hi-2.10
with Guix b7249aa.

The upstream source is archived on SWH with:

guix lint -L . hi -c archival

see

Note that running again "lint -c archival" returns nothing, which means
that the linter finds it on SWH, I guess. Well, I have not checked what
'lookup-content' returns yet. However, I filtered the first ~2000+
'packages-with-sources' with 'lookup-content' and all the 'git-fetch'
sources are returned #f, and some 'url-fetch' not. Another story,
another thread. :-)


Let clean the previous builds and downloads:

guix gc -D $(guix gc --list-dead | grep -E '(hi-2|git-checkout)')

And now let turn off upstream and so the maybe bug:

Toggle snippet (34 lines)
guix build -L . hi
The following derivations will be built:
/gnu/store/x5vwhv8sn3l49vysq98j7h2rqqlff5zp-hi-2.10.drv
/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv
building /gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to `/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
Initialized empty Git repository in /gnu/store/884nsva9r8wkp40kbqyvpj1ad57jc5dd-git-checkout/.git/
fatal: could not read Username for 'https://github.com': No such device or address
Failed to do a shallow fetch; retrying a full fetch...
fatal: could not read Username for 'https://github.com': No such device or address
git-fetch: '/gnu/store/i2pkspl2vjvas0q85hw46y2li06rcmcg-git-minimal-2.26.2/bin/git fetch origin' failed with exit code 128
Trying content-addressed mirror at berlin.guixsd.org...
Trying content-addressed mirror at berlin.guixsd.org...
Trying to download from Software Heritage...
Backtrace:
4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
In ./guix/swh.scm:
573:13 3 (swh-download "https://github.com/zimoun/hello-example?" ?)
224:22 2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
In web/client.scm:
563:0 1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
231:6 0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)

web/client.scm:231:6: In procedure tls-wrap:
Error while printing exception.
builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1
build of /gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv failed
View build log at '/var/log/guix/drvs/jn/6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv.bz2'.
cannot build derivation `/gnu/store/x5vwhv8sn3l49vysq98j7h2rqqlff5zp-hi-2.10.drv': 1 dependencies couldn't be built
guix build: error: build of `/gnu/store/x5vwhv8sn3l49vysq98j7h2rqqlff5zp-hi-2.10.drv' failed


Let me know if moreinfo are required.

All the best,
simon
Z
Z
zimoun wrote on 9 Jul 2020 03:52
swh-download with https->http
(address . 42286@debbugs.gnu.org)
86eeplcv45.fsf@gmail.com
On Thu, 09 Jul 2020 at 02:36, zimoun <zimon.toutoune@gmail.com> wrote:

Toggle quote (3 lines)
> --8<---------------cut here---------------start------------->8---
> guix build -L . hi

[...]

Toggle quote (13 lines)
> Trying to download from Software Heritage...
> Backtrace:
> 4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
> In ./guix/swh.scm:
> 573:13 3 (swh-download "https://github.com/zimoun/hello-example?" ?)
> 224:22 2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
> In web/client.scm:
> 563:0 1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
> 231:6 0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>
> web/client.scm:231:6: In procedure tls-wrap:
> Error while printing exception.

[...]

Toggle quote (2 lines)
> --8<---------------cut here---------------end--------------->8---

To avoid, this error which should be a missing certificate, I switch to
'http:' instead of 'https:' in %swh-base-url. The error is then:

Toggle snippet (13 lines)
Trying to download from Software Heritage...
Backtrace:
2 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
In ./guix/swh.scm:
573:13 1 (swh-download "https://github.com/zimoun/hello-example?" ?)
243:16 0 (call "http://archive.softwareheritage.org/api/1/revis?" ?)

./guix/swh.scm:243:16: In procedure call:
Throw to key `swh-error' with args `("http://archive.softwareheritage.org/api/1/revision/e1eefd033b8a2c4c81babc6fde08ebb116c6abb8/" #<procedure http-get (uri #:key body port version keep-alive? headers decode-body? verify-certificate? streaming?)> #<<response> version: (1 . 1) code: 302 reason-phrase: "Found" headers: ((date . #<date nanosecond: 0 second: 59 minute: 38 hour: 1 day: 9 month: 7 year: 2020 zone-offset: 0>) (server . "Varnish") (x-varnish . "19881791") (location . #<<uri> scheme: https userinfo: #f host: "archive.softwareheritage.org" port: #f path: "/api/1/revision/e1eefd033b8a2c4c81babc6fde08ebb116c6abb8/" query: #f fragment: #f>) (content-length . 0) (connection close)) port: #<closed: file 7ffff5f1b690>>)'.
builder for `/gnu/store/9f6648zb0p9z6hxr31wz3d7fm7ndcxzx-git-checkout.drv' failed with exit code 1


Because I hit the rate limit, I need to wait at least one hour...
Follow-ups soon. :-)

Cheers,
simon
L
L
Ludovic Courtès wrote on 9 Jul 2020 16:31
control message for bug #42286
(address . control@debbugs.gnu.org)
875zawbw04.fsf@gnu.org
severity 42286 important
quit
Z
Z
zimoun wrote on 9 Jul 2020 20:20
Re: bug#42286: SWH fallback fails (git-fetch)
(address . 42286@debbugs.gnu.org)
CAJ3okZ0vpgQ9AHMsLeEbTSuENDtwhCx4CrLOupCWfePSpsqb+w@mail.gmail.com
On Thu, 9 Jul 2020 at 02:39, zimoun <zimon.toutoune@gmail.com> wrote:

Well, "swh-download" perfectly works directly from the REPL.

Toggle quote (3 lines)
> --8<---------------cut here---------------start------------->8---
> guix build -L . hi

[...]

Toggle quote (13 lines)
> Trying to download from Software Heritage...
> Backtrace:
> 4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
> In ./guix/swh.scm:
> 573:13 3 (swh-download "https://github.com/zimoun/hello-example?" ?)
> 224:22 2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
> In web/client.scm:
> 563:0 1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
> 231:6 0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>
> web/client.scm:231:6: In procedure tls-wrap:
> --8<---------------cut here---------------end--------------->8---

The error is definitively something related to TLS and the gexp. In
(guix git-download) "git-fetch", the "define build" returns a gexp and
here something is missing, even if the extension gnutls (module-ref
(resolve-interface '(gnu packages tls)) 'gnutls) is provided. Hum, I
am not sure to understand what.

BTW, if in "git-fetch" from (guix git), I add these lines:

Toggle snippet (9 lines)
(setenv "GIT_SSL_NO_VERIFY" "true")

(format #t "git-fetch~%")
(http-get "https://archive.softwareheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b/")
(format #t "ok~%")

(mkdir-p directory)

Then I also hit:

Toggle snippet (32 lines)
The following derivations will be built:
/gnu/store/wam9fca6vj3rifvqlix9c874vpwn5k82-hi-2.10.drv
/gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv
building /gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv...
guile: warning: failed to install locale
environment variable `PATH' set to
`/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin:/gnu/store/sf3rbvb6iqcphgm1afbplcs72hsywg25-tar-1.32/bin'
git-fetch
Backtrace:
4 (primitive-load "/gnu/store/gcr6v6p6c5gwr4l6xzqcy6wln33?")
In ice-9/eval.scm:
293:34 3 (_ #<directory (guile-user) 7ffff5bb8f00>)
In ./guix/build/git.scm:
44:2 2 (git-fetch "https://github.com/zimoun/hello-example.git" ?)
In web/client.scm:
563:0 1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
231:6 0 (tls-wrap #<closed: file 7ffff4c17150> _ # _)

web/client.scm:231:6: In procedure tls-wrap:
Error while printing exception.
builder for `/gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv'
failed with exit code 1
build of /gnu/store/cvp65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv failed
View build log at
'/var/log/guix/drvs/cv/p65m4wzmzd8pqdfvah4mrl4zkcw3vz-git-checkout.drv.bz2'.
cannot build derivation
`/gnu/store/wam9fca6vj3rifvqlix9c874vpwn5k82-hi-2.10.drv': 1
dependencies couldn't be built
guix build: error: build of
`/gnu/store/wam9fca6vj3rifvqlix9c874vpwn5k82-hi-2.10.drv' failed

Then I have tried to turn off the certificate verification with
"#:verify-certification #f" for example in "vault-fetch" or "call"
used by "define-query" but nothing works. Well, I am a bit
circumspect.

Therefore, I am waiting for a hint or the fix. :-)

Cheers,
simon
L
L
Ludovic Courtès wrote on 10 Jul 2020 00:24
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 42286-done@debbugs.gnu.org)
87o8oo9vjp.fsf@gnu.org
Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (14 lines)
> Trying to download from Software Heritage...
> Backtrace:
> 4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
> In ./guix/swh.scm:
> 573:13 3 (swh-download "https://github.com/zimoun/hello-example?" ?)
> 224:22 2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
> In web/client.scm:
> 563:0 1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
> 231:6 0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>
> web/client.scm:231:6: In procedure tls-wrap:
> Error while printing exception.
> builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1

Should be fixed with commit a7696b9733d4ede9817a0a0accb5ce5b85d9a2d3.
Let me know if anything’s amiss.

Thanks!

Ludo’.
Closed
Z
Z
zimoun wrote on 10 Jul 2020 05:18
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 42286-done@debbugs.gnu.org)
86sge0awhq.fsf@gmail.com
Hi,

On Fri, 10 Jul 2020 at 00:24, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (21 lines)
> Hi!
>
> zimoun <zimon.toutoune@gmail.com> skribis:
>
>> Trying to download from Software Heritage...
>> Backtrace:
>> 4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
>> In ./guix/swh.scm:
>> 573:13 3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>> 224:22 2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
>> In web/client.scm:
>> 563:0 1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>> 231:6 0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>>
>> web/client.scm:231:6: In procedure tls-wrap:
>> Error while printing exception.
>> builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1
>
> Should be fixed with commit a7696b9733d4ede9817a0a0accb5ce5b85d9a2d3.
> Let me know if anything’s amiss.

Cool! Works. :-)

I was almost there. :-) The missing trick was because the Guile bug
I was not aware and so the new "http-get*".

Is it worth to add the test in guix-build.sh?


All the best,
simon
Closed
L
L
Ludovic Courtès wrote on 10 Jul 2020 10:07
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 42286-done@debbugs.gnu.org)
87365zaj3u.fsf@gnu.org
Hi!

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (28 lines)
> On Fri, 10 Jul 2020 at 00:24, Ludovic Courtès <ludo@gnu.org> wrote:
>> Hi!
>>
>> zimoun <zimon.toutoune@gmail.com> skribis:
>>
>>> Trying to download from Software Heritage...
>>> Backtrace:
>>> 4 (primitive-load "/gnu/store/s56y8npabah6jc1bqrhsac6wqb1?")
>>> In ./guix/swh.scm:
>>> 573:13 3 (swh-download "https://github.com/zimoun/hello-example?" ?)
>>> 224:22 2 (call "https://archive.softwareheritage.org/api/1/revi?" ?)
>>> In web/client.scm:
>>> 563:0 1 (http-get "https://archive.softwareheritage.org/api/1/?" ?)
>>> 231:6 0 (tls-wrap #<closed: file 7ffff5f1b690> _ # _)
>>>
>>> web/client.scm:231:6: In procedure tls-wrap:
>>> Error while printing exception.
>>> builder for `/gnu/store/jn6f86hg9zyyhms1vn56hviv4m9yjm8j-git-checkout.drv' failed with exit code 1
>>
>> Should be fixed with commit a7696b9733d4ede9817a0a0accb5ce5b85d9a2d3.
>> Let me know if anything’s amiss.
>
> Cool! Works. :-)
>
> I was almost there. :-) The missing trick was because the Guile bug
> <https://bugs.gnu.org/40486>
> I was not aware and so the new "http-get*".

Yeah. :-)

Toggle quote (2 lines)
> Is it worth to add the test in guix-build.sh?

We don’t add tests that depend on external services, so we can’t really
do that. Or we would need to mock the original server, SWH, etc. but
that seems tricky.

Thanks,
Ludo’.
Closed
?