[PATCH 0/1] swh: Allows token from Software Heritage authentication service.

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • zimoun
Owner
unassigned
Submitted by
zimoun
Severity
normal
Z
Z
zimoun wrote on 14 Oct 2021 23:32
(address . guix-patches@gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
20211014213238.3270613-1-zimon.toutoune@gmail.com
Hi,

This patch allows to use a token from SWH and change the rate limit when
submitting via the archival checker. For instance, it reads,

GUIX_SWH_TOKEN=${TOKEN} guix lint -c archival

and will use the ${TOKEN} associated to the SWH account. Having an account
enable by default a rate limit to 1200 (instead of 120). Then, for testing
purpose I get a lot of android packages scheduled after checking many admin.

It is interesting to note lines as:

Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'

which shows that sources.json is maybe not fully working as expected.
Investigations required.

Toggle snippet (105 lines)
sitour@jack-Precision-7820-Tower$ GUIX_SWH_TOKEN=$TOKEN ./pre-inst-env guix lint -c archival
gnu/packages/accessibility.scm:100:5: brltty@6.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:4330:5: pam-mount@2.18: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:3902:12: launchmon@1.0.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:933:12: net-base@5.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2272:12: testdisk@7.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2148:12: acpica@20210930:
Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'
gnu/packages/admin.scm:3029:12: nmap@7.92: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:725:5: pies@1.6: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2116:5: dmidecode@3.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:1108:5: prips@1.1.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2939:12: munge@0.5.14: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:534:12: facter@4.0.52: scheduled Software Heritage archival
gnu/packages/admin.scm:1137:12: alive@2.0.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/admin.scm:2677:5: debops@1.1.0: scheduled Software Heritage archival
checking ansible@4.4.0 [archival]...
gnu/packages/admin.scm:3571:12: neofetch@7.1.0: scheduled Software Heritage archival
gnu/packages/aidc.scm:112:12: qrencode@4.1.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/aidc.scm:174:5: zbar@0.23: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:489:12: arb@2.21.0: scheduled Software Heritage archival
gnu/packages/algebra.scm:1053:14: eigen-for-tensorflow@3.3.5-1.fd6845384b86: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:1104:12: xtensor@0.20.10: scheduled Software Heritage archival
gnu/packages/algebra.scm:1682:11: sollya@7.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:1139:5: gap@4.11.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/algebra.scm:957:5: symengine@0.6.0: scheduled Software Heritage archival
gnu/packages/algebra.scm:1605:5: iml@1.0.5: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/android.scm:529:12: android-ext4-utils@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:659:12: fastboot@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:744:5: git-repo@2.4.1: scheduled Software Heritage archival
gnu/packages/android.scm:323:12: android-libziparchive@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:581:12: android-f2fs-utils@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:405:12: mkbootimg@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:358:12: adb@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:302:12: android-libsparse@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:248:12: android-libcutils@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:458:12: android-bionic-uapi@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:619:12: android-libutils@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/android.scm:101:5: android-googletest@1.8.0: scheduled Software Heritage archival
gnu/packages/android.scm:195:12: android-liblog@7.1.2_r36: scheduled Software Heritage archival
gnu/packages/animation.scm:67:5: rlottie@0.2: scheduled Software Heritage archival
gnu/packages/antivirus.scm:48:12: clamav@0.103.3:
Disarchive entry refers to non-existent SWH directory 'b25414c9864a270899ca1ff494e7ba4c437b166d'
gnu/packages/aspell.scm:115:12: aspell-dict-en@2020.12.07-0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:898:5: indi@1.9.1: scheduled Software Heritage archival
gnu/packages/astronomy.scm:239:5: wcslib@7.5: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:298:5: gnuastro@0.16: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:823:5: gpredict@2.2.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/astronomy.scm:518:5: stuff@1.26.0:
Disarchive entry refers to non-existent SWH directory '128bbe76a82dd0b38b725565ed703a7148257ae0'
gnu/packages/astronomy.scm:216:5: eye@1.4.1:
Disarchive entry refers to non-existent SWH directory '92625e2c6dbe3ad7c4f44a061ada24ce00637087'
gnu/packages/astronomy.scm:377:5: skymaker@3.10.5:
Disarchive entry refers to non-existent SWH directory '6000a273dfff9de62725b53e41562fff711069c1'
gnu/packages/astronomy.scm:278:5: weightwatcher@1.12:
Disarchive entry refers to non-existent SWH directory 'c68ff8714c6fd360a38158f3d8f22e555c061452'
gnu/packages/audio.scm:2941:12: supercollider@3.11.2: scheduled Software Heritage archival
gnu/packages/audio.scm:3131:12: rubberband@1.8.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:5431:6: mda-lv2@1.2.6: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3906:12: zita-alsa-pcmi@0.3.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2581:12: lilv@0.24.10: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:309:12: libgme@0.6.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:4813:5: redkite@1.3.0: scheduled Software Heritage archival
gnu/packages/audio.scm:1908:12: guitarix@0.41.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:4657:5: gnaural@20110606: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:1962:2: guitarix-lv2@0.41.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:947:12: azr3@1.2.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:4266:12: snd@20.9:
Disarchive entry refers to non-existent SWH directory 'cb52aaa9500df2b674bf7922811deeea1b766139'
gnu/packages/audio.scm:2811:12: patchage@1.0.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:5130:6: lsp-dsp-lib@0.5.11:
Disarchive entry refers to non-existent SWH directory '3e574043a04d77dd7231d23210547c4fe065a40c'
gnu/packages/audio.scm:4114:5: gsm@1.0.19:
Disarchive entry refers to non-existent SWH directory 'aa763150704fe06f34097b38e839409cee52366d'
gnu/packages/audio.scm:4956:6: libaudec@0.2.4: scheduled Software Heritage archival
gnu/packages/audio.scm:3699:5: qsynth@0.5.7:
Disarchive entry refers to non-existent SWH directory '127c0a03c7ccba74870aef7dac36019af35798cc'
gnu/packages/audio.scm:3425:5: wavpack@5.4.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3286:12: timidity++@2.15.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2176:12: jalv@1.6.4: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3802:12: zita-convolver@4.0.3: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2541:6: audio-to-midi@2020.7: scheduled Software Heritage archival
gnu/packages/audio.scm:563:12: aubio@0.4.9: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3199:12: sratom@0.6.6: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3226:12: suil@0.10.8: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3853:12: zita-resampler@1.6.2: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:2627:12: lv2@1.18.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/audio.scm:3170:12: rtmidi@4.0.0:
Disarchive entry refers to non-existent SWH directory 'd9745f29da983c6ad674871e68ac96362c4f11cc'
gnu/packages/audio.scm:528:12: alsa-modular-synth@2.2.0: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/augeas.scm:37:12: augeas@1.12.0:
Disarchive entry refers to non-existent SWH directory '7d7ed9f88ee649a90493f54d3988a062c3ddeafb'
gnu/packages/autogen.scm:38:5: autogen@5.18.16: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/autotools.scm:291:5: autoconf-archive@2021.02.19: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/backup.scm:765:12: wimlib@1.13.4: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/backup.scm:192:5: hdup@2.0.14: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/base.scm:587:11: binutils@2.33.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/batik.scm:204:6: java-xmlgraphics-commons@2.6:
Disarchive entry refers to non-existent SWH directory 'f5bd0fe7450175196c57d6f6d5aca8905393e814'
gnu/packages/benchmark.scm:57:12: fio@3.27: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/bioconductor.scm:779:12: r-txdb-hsapiens-ucsc-hg38-knowngene@3.4.6:
Disarchive entry refers to non-existent SWH directory '92bd3b93caa9a4b0840c70ddb96ac75b0684d7ec'
gnu/packages/bioconductor.scm:10755:5: r-timeseriesexperiment@1.10.1: Software Heritage rate limit reached; try again later

All the best,
simon


zimoun (1):
swh: Allows token from Software Heritage authentication service.

guix/swh.scm | 23 +++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)


base-commit: cf25ba2d773435af7af2cd315185acb16dcc6b58
--
2.32.0
Z
Z
zimoun wrote on 14 Oct 2021 23:33
[PATCH 1/1] swh: Allows token from Software Heritage authentication service.
(address . 51216@debbugs.gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
20211014213346.3270670-1-zimon.toutoune@gmail.com
The token is provided using the environment variable GUIX_SWH_TOKEN.

* guix/swh.scm (swh-token): New variable.
(http-get*, http-post*): Use it.
---
guix/swh.scm | 23 +++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)

Toggle diff (47 lines)
diff --git a/guix/swh.scm b/guix/swh.scm
index 5c41685a24..38a4af723a 100644
--- a/guix/swh.scm
+++ b/guix/swh.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Jakub K?dzio?ka <kuba@kadziolka.net>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
+;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -153,12 +154,30 @@ (define url
url
(string-append url "/")))
+;; Token from an account to the Software Heritage Authentication service
+;; <https://archive.softwareheritage.org/api/>
+(define swh-token
+ (match (getenv "GUIX_SWH_TOKEN")
+ (#f #f)
+ ((? string-null? s) #f)
+ ((? string? s) (string->symbol s))))
+
;; XXX: Work around a bug in Guile 3.0.2 where #:verify-certificate? would
;; be ignored (<https://bugs.gnu.org/40486>).
(define* (http-get* uri #:rest rest)
- (apply http-request uri #:method 'GET rest))
+ (apply http-request uri #:method 'GET
+ #:headers
+ (if swh-token
+ `((authorization . (Bearer ,swh-token)))
+ '())
+ rest))
(define* (http-post* uri #:rest rest)
- (apply http-request uri #:method 'POST rest))
+ (apply http-request uri #:method 'POST
+ #:headers
+ (if swh-token
+ `((authorization . (Bearer ,swh-token)))
+ '())
+ rest))
(define %date-regexp
;; Match strings like "2014-11-17T22:09:38+01:00" or
--
2.32.0
L
L
Ludovic Courtès wrote on 14 Oct 2021 23:44
Re: bug#51216: [PATCH 0/1] swh: Allows token from Software Heritage authentication service.
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 51216@debbugs.gnu.org)
87sfx3l1sr.fsf@gnu.org
Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (8 lines)
> This patch allows to use a token from SWH and change the rate limit when
> submitting via the archival checker. For instance, it reads,
>
> GUIX_SWH_TOKEN=${TOKEN} guix lint -c archival
>
> and will use the ${TOKEN} associated to the SWH account. Having an account
> enable by default a rate limit to 1200 (instead of 120). Then, for testing

Ooh, neat!

Toggle quote (2 lines)
> purpose I get a lot of android packages scheduled after checking many admin.

You mean the SWH admins manually scheduled them?

Toggle quote (7 lines)
> It is interesting to note lines as:
>
> Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'
>
> which shows that sources.json is maybe not fully working as expected.
> Investigations required.

Weird. I guess we should pick a few specific cases, make sure they’re
listed in sources.json and they’re not 404, and then ask the SWH team
whether they can check their logs to understand what happened.

Thanks!

Ludo’.
L
L
Ludovic Courtès wrote on 14 Oct 2021 23:46
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 51216@debbugs.gnu.org)
87o87rl1o9.fsf_-_@gnu.org
zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (5 lines)
> The token is provided using the environment variable GUIX_SWH_TOKEN.
>
> * guix/swh.scm (swh-token): New variable.
> (http-get*, http-post*): Use it.

[...]

Toggle quote (8 lines)
> +;; Token from an account to the Software Heritage Authentication service
> +;; <https://archive.softwareheritage.org/api/>
> +(define swh-token
> + (match (getenv "GUIX_SWH_TOKEN")
> + (#f #f)
> + ((? string-null? s) #f)
> + ((? string? s) (string->symbol s))))

Could you turn it into a SRFI-39 parameter?

Toggle quote (19 lines)
> ;; XXX: Work around a bug in Guile 3.0.2 where #:verify-certificate? would
> ;; be ignored (<https://bugs.gnu.org/40486>).
> (define* (http-get* uri #:rest rest)
> - (apply http-request uri #:method 'GET rest))
> + (apply http-request uri #:method 'GET
> + #:headers
> + (if swh-token
> + `((authorization . (Bearer ,swh-token)))
> + '())
> + rest))
> (define* (http-post* uri #:rest rest)
> - (apply http-request uri #:method 'POST rest))
> + (apply http-request uri #:method 'POST
> + #:headers
> + (if swh-token
> + `((authorization . (Bearer ,swh-token)))
> + '())
> + rest))

These two procedures are meant to go away as soon as Guile > 3.0.2 is
required.

Could you instead pass #:headers in the ‘call’ procedure, and maybe in
‘vault-fetch’ too?

Thanks!

Ludo’.
Z
Z
zimoun wrote on 14 Oct 2021 23:52
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 51216@debbugs.gnu.org)
86a6jbp941.fsf@gmail.com
Re,

On Thu, 14 Oct 2021 at 23:44, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (4 lines)
>> purpose I get a lot of android packages scheduled after checking many admin.
>
> You mean the SWH admins manually scheduled them?

No no, sorry my bad wording. I mean “guix lint -c archival” checked all
the 260+ packages in gnu/packages/admin.scm then it checked
gnu/packages/android.scm, found some unarchived and scheduled them.


Toggle quote (11 lines)
>> It is interesting to note lines as:
>>
>> Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'
>>
>> which shows that sources.json is maybe not fully working as expected.
>> Investigations required.
>
> Weird. I guess we should pick a few specific cases, make sure they’re
> listed in sources.json and they’re not 404, and then ask the SWH team
> whether they can check their logs to understand what happened.

Yeah… investigations required. :-)


Cheers,
simon
Z
Z
zimoun wrote on 15 Oct 2021 00:14
[PATCH v2] swh: Allows token from Software Heritage authentication service.
(address . 51216@debbugs.gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
20211014221454.3272836-1-zimon.toutoune@gmail.com
The token is provided using the environment variable GUIX_SWH_TOKEN.

* guix/swh.scm (%swh-token): New variable.
(call): Use it.
---
guix/swh.scm | 11 +++++++++++
1 file changed, 11 insertions(+)

Toggle diff (40 lines)
diff --git a/guix/swh.scm b/guix/swh.scm
index 5c41685a24..c7c1c873a2 100644
--- a/guix/swh.scm
+++ b/guix/swh.scm
@@ -2,6 +2,7 @@
;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org>
;;; Copyright © 2020 Jakub K?dzio?ka <kuba@kadziolka.net>
;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>
+;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -136,6 +137,12 @@ (define %verify-swh-certificate?
;; Whether to verify the X.509 HTTPS certificate for %SWH-BASE-URL.
(make-parameter #t))
+;; Token from an account to the Software Heritage Authentication service
+;; <https://archive.softwareheritage.org/api/>
+(define %swh-token
+ (make-parameter (and=> (getenv "GUIX_SWH_TOKEN")
+ string->symbol)))
+
(define (swh-url path . rest)
;; URLs returned by the API may be relative or absolute. This has changed
;; without notice before. Handle both cases by detecting whether the path
@@ -246,6 +253,10 @@ (define* (call url decode #:optional (method http-get*)
(and ((%allow-request?) url method)
(let*-values (((response port)
(method url #:streaming? #t
+ #:headers
+ (if (%swh-token)
+ `((authorization . (Bearer ,(%swh-token))))
+ '())
#:verify-certificate?
(%verify-swh-certificate?))))
;; See <https://archive.softwareheritage.org/api/#rate-limiting>.

base-commit: cf25ba2d773435af7af2cd315185acb16dcc6b58
--
2.32.0
L
L
Ludovic Courtès wrote on 15 Oct 2021 22:54
Re: bug#51216: [PATCH 0/1] swh: Allows token from Software Heritage authentication service.
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 51216-done@debbugs.gnu.org)
87lf2udn5c.fsf_-_@gnu.org
zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (5 lines)
> The token is provided using the environment variable GUIX_SWH_TOKEN.
>
> * guix/swh.scm (%swh-token): New variable.
> (call): Use it.

Applied, thanks! :-)

Ludo'.
Closed
?