[PATCH 0/1] swh: Allows token from Software Heritage authentication service.

DoneSubmitted by zimoun.
Details
2 participants
  • Ludovic Courtès
  • zimoun
Owner
unassigned
Severity
normal
Z
Z
zimoun wrote on 14 Oct 23:32 +0200
(address . guix-patches@gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
20211014213238.3270613-1-zimon.toutoune@gmail.com
Hi,
This patch allows to use a token from SWH and change the rate limit whensubmitting via the archival checker. For instance, it reads,
GUIX_SWH_TOKEN=${TOKEN} guix lint -c archival
and will use the ${TOKEN} associated to the SWH account. Having an accountenable by default a rate limit to 1200 (instead of 120). Then, for testingpurpose I get a lot of android packages scheduled after checking many admin.
It is interesting to note lines as:
Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'
which shows that sources.json is maybe not fully working as expected.Investigations required.
Toggle snippet (105 lines)sitour@jack-Precision-7820-Tower$ GUIX_SWH_TOKEN=$TOKEN ./pre-inst-env guix lint -c archivalgnu/packages/accessibility.scm:100:5: brltty@6.2: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/admin.scm:4330:5: pam-mount@2.18: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/admin.scm:3902:12: launchmon@1.0.2: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/admin.scm:933:12: net-base@5.3: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/admin.scm:2272:12: testdisk@7.1: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/admin.scm:2148:12: acpica@20210930:Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'gnu/packages/admin.scm:3029:12: nmap@7.92: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/admin.scm:725:5: pies@1.6: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/admin.scm:2116:5: dmidecode@3.3: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/admin.scm:1108:5: prips@1.1.1: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/admin.scm:2939:12: munge@0.5.14: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/admin.scm:534:12: facter@4.0.52: scheduled Software Heritage archivalgnu/packages/admin.scm:1137:12: alive@2.0.3: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/admin.scm:2677:5: debops@1.1.0: scheduled Software Heritage archivalchecking ansible@4.4.0 [archival]...gnu/packages/admin.scm:3571:12: neofetch@7.1.0: scheduled Software Heritage archivalgnu/packages/aidc.scm:112:12: qrencode@4.1.1: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/aidc.scm:174:5: zbar@0.23: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/algebra.scm:489:12: arb@2.21.0: scheduled Software Heritage archivalgnu/packages/algebra.scm:1053:14: eigen-for-tensorflow@3.3.5-1.fd6845384b86: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/algebra.scm:1104:12: xtensor@0.20.10: scheduled Software Heritage archivalgnu/packages/algebra.scm:1682:11: sollya@7.0: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/algebra.scm:1139:5: gap@4.11.0: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/algebra.scm:957:5: symengine@0.6.0: scheduled Software Heritage archivalgnu/packages/algebra.scm:1605:5: iml@1.0.5: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/android.scm:529:12: android-ext4-utils@7.1.2_r36: scheduled Software Heritage archivalgnu/packages/android.scm:659:12: fastboot@7.1.2_r36: scheduled Software Heritage archivalgnu/packages/android.scm:744:5: git-repo@2.4.1: scheduled Software Heritage archivalgnu/packages/android.scm:323:12: android-libziparchive@7.1.2_r36: scheduled Software Heritage archivalgnu/packages/android.scm:581:12: android-f2fs-utils@7.1.2_r36: scheduled Software Heritage archivalgnu/packages/android.scm:405:12: mkbootimg@7.1.2_r36: scheduled Software Heritage archivalgnu/packages/android.scm:358:12: adb@7.1.2_r36: scheduled Software Heritage archivalgnu/packages/android.scm:302:12: android-libsparse@7.1.2_r36: scheduled Software Heritage archivalgnu/packages/android.scm:248:12: android-libcutils@7.1.2_r36: scheduled Software Heritage archivalgnu/packages/android.scm:458:12: android-bionic-uapi@7.1.2_r36: scheduled Software Heritage archivalgnu/packages/android.scm:619:12: android-libutils@7.1.2_r36: scheduled Software Heritage archivalgnu/packages/android.scm:101:5: android-googletest@1.8.0: scheduled Software Heritage archivalgnu/packages/android.scm:195:12: android-liblog@7.1.2_r36: scheduled Software Heritage archivalgnu/packages/animation.scm:67:5: rlottie@0.2: scheduled Software Heritage archivalgnu/packages/antivirus.scm:48:12: clamav@0.103.3:Disarchive entry refers to non-existent SWH directory 'b25414c9864a270899ca1ff494e7ba4c437b166d'gnu/packages/aspell.scm:115:12: aspell-dict-en@2020.12.07-0: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/astronomy.scm:898:5: indi@1.9.1: scheduled Software Heritage archivalgnu/packages/astronomy.scm:239:5: wcslib@7.5: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/astronomy.scm:298:5: gnuastro@0.16: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/astronomy.scm:823:5: gpredict@2.2.1: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/astronomy.scm:518:5: stuff@1.26.0:Disarchive entry refers to non-existent SWH directory '128bbe76a82dd0b38b725565ed703a7148257ae0'gnu/packages/astronomy.scm:216:5: eye@1.4.1:Disarchive entry refers to non-existent SWH directory '92625e2c6dbe3ad7c4f44a061ada24ce00637087'gnu/packages/astronomy.scm:377:5: skymaker@3.10.5:Disarchive entry refers to non-existent SWH directory '6000a273dfff9de62725b53e41562fff711069c1'gnu/packages/astronomy.scm:278:5: weightwatcher@1.12:Disarchive entry refers to non-existent SWH directory 'c68ff8714c6fd360a38158f3d8f22e555c061452'gnu/packages/audio.scm:2941:12: supercollider@3.11.2: scheduled Software Heritage archivalgnu/packages/audio.scm:3131:12: rubberband@1.8.2: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:5431:6: mda-lv2@1.2.6: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:3906:12: zita-alsa-pcmi@0.3.2: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:2581:12: lilv@0.24.10: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:309:12: libgme@0.6.3: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:4813:5: redkite@1.3.0: scheduled Software Heritage archivalgnu/packages/audio.scm:1908:12: guitarix@0.41.0: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:4657:5: gnaural@20110606: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:1962:2: guitarix-lv2@0.41.0: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:947:12: azr3@1.2.3: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:4266:12: snd@20.9:Disarchive entry refers to non-existent SWH directory 'cb52aaa9500df2b674bf7922811deeea1b766139'gnu/packages/audio.scm:2811:12: patchage@1.0.2: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:5130:6: lsp-dsp-lib@0.5.11:Disarchive entry refers to non-existent SWH directory '3e574043a04d77dd7231d23210547c4fe065a40c'gnu/packages/audio.scm:4114:5: gsm@1.0.19:Disarchive entry refers to non-existent SWH directory 'aa763150704fe06f34097b38e839409cee52366d'gnu/packages/audio.scm:4956:6: libaudec@0.2.4: scheduled Software Heritage archivalgnu/packages/audio.scm:3699:5: qsynth@0.5.7:Disarchive entry refers to non-existent SWH directory '127c0a03c7ccba74870aef7dac36019af35798cc'gnu/packages/audio.scm:3425:5: wavpack@5.4.0: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:3286:12: timidity++@2.15.0: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:2176:12: jalv@1.6.4: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:3802:12: zita-convolver@4.0.3: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:2541:6: audio-to-midi@2020.7: scheduled Software Heritage archivalgnu/packages/audio.scm:563:12: aubio@0.4.9: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:3199:12: sratom@0.6.6: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:3226:12: suil@0.10.8: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:3853:12: zita-resampler@1.6.2: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:2627:12: lv2@1.18.0: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/audio.scm:3170:12: rtmidi@4.0.0:Disarchive entry refers to non-existent SWH directory 'd9745f29da983c6ad674871e68ac96362c4f11cc'gnu/packages/audio.scm:528:12: alsa-modular-synth@2.2.0: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/augeas.scm:37:12: augeas@1.12.0:Disarchive entry refers to non-existent SWH directory '7d7ed9f88ee649a90493f54d3988a062c3ddeafb'gnu/packages/autogen.scm:38:5: autogen@5.18.16: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/autotools.scm:291:5: autoconf-archive@2021.02.19: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/backup.scm:765:12: wimlib@1.13.4: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/backup.scm:192:5: hdup@2.0.14: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/base.scm:587:11: binutils@2.33.1: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/batik.scm:204:6: java-xmlgraphics-commons@2.6:Disarchive entry refers to non-existent SWH directory 'f5bd0fe7450175196c57d6f6d5aca8905393e814'gnu/packages/benchmark.scm:57:12: fio@3.27: source not archived on Software Heritage and missing from the Disarchive databasegnu/packages/bioconductor.scm:779:12: r-txdb-hsapiens-ucsc-hg38-knowngene@3.4.6:Disarchive entry refers to non-existent SWH directory '92bd3b93caa9a4b0840c70ddb96ac75b0684d7ec'gnu/packages/bioconductor.scm:10755:5: r-timeseriesexperiment@1.10.1: Software Heritage rate limit reached; try again later
All the best,simon

zimoun (1): swh: Allows token from Software Heritage authentication service.
guix/swh.scm | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)

base-commit: cf25ba2d773435af7af2cd315185acb16dcc6b58-- 2.32.0
Z
Z
zimoun wrote on 14 Oct 23:33 +0200
[PATCH 1/1] swh: Allows token from Software Heritage authentication service.
(address . 51216@debbugs.gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
20211014213346.3270670-1-zimon.toutoune@gmail.com
The token is provided using the environment variable GUIX_SWH_TOKEN.
* guix/swh.scm (swh-token): New variable.(http-get*, http-post*): Use it.--- guix/swh.scm | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-)
Toggle diff (47 lines)diff --git a/guix/swh.scm b/guix/swh.scmindex 5c41685a24..38a4af723a 100644--- a/guix/swh.scm+++ b/guix/swh.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net> ;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>+;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com> ;;; ;;; This file is part of GNU Guix. ;;;@@ -153,12 +154,30 @@ (define url url (string-append url "/"))) +;; Token from an account to the Software Heritage Authentication service+;; <https://archive.softwareheritage.org/api/>+(define swh-token+ (match (getenv "GUIX_SWH_TOKEN")+ (#f #f)+ ((? string-null? s) #f)+ ((? string? s) (string->symbol s))))+ ;; XXX: Work around a bug in Guile 3.0.2 where #:verify-certificate? would ;; be ignored (<https://bugs.gnu.org/40486>). (define* (http-get* uri #:rest rest)- (apply http-request uri #:method 'GET rest))+ (apply http-request uri #:method 'GET+ #:headers+ (if swh-token+ `((authorization . (Bearer ,swh-token)))+ '())+ rest)) (define* (http-post* uri #:rest rest)- (apply http-request uri #:method 'POST rest))+ (apply http-request uri #:method 'POST+ #:headers+ (if swh-token+ `((authorization . (Bearer ,swh-token)))+ '())+ rest)) (define %date-regexp ;; Match strings like "2014-11-17T22:09:38+01:00" or-- 2.32.0
L
L
Ludovic Courtès wrote on 14 Oct 23:44 +0200
Re: bug#51216: [PATCH 0/1] swh: Allows token from Software Heritage authentication service.
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 51216@debbugs.gnu.org)
87sfx3l1sr.fsf@gnu.org
Hi,
zimoun <zimon.toutoune@gmail.com> skribis:
Toggle quote (8 lines)> This patch allows to use a token from SWH and change the rate limit when> submitting via the archival checker. For instance, it reads,>> GUIX_SWH_TOKEN=${TOKEN} guix lint -c archival>> and will use the ${TOKEN} associated to the SWH account. Having an account> enable by default a rate limit to 1200 (instead of 120). Then, for testing
Ooh, neat!
Toggle quote (2 lines)> purpose I get a lot of android packages scheduled after checking many admin.
You mean the SWH admins manually scheduled them?
Toggle quote (7 lines)> It is interesting to note lines as:>> Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'>> which shows that sources.json is maybe not fully working as expected.> Investigations required.
Weird. I guess we should pick a few specific cases, make sure they’relisted in sources.json and they’re not 404, and then ask the SWH teamwhether they can check their logs to understand what happened.
Thanks!
Ludo’.
L
L
Ludovic Courtès wrote on 14 Oct 23:46 +0200
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 51216@debbugs.gnu.org)
87o87rl1o9.fsf_-_@gnu.org
zimoun <zimon.toutoune@gmail.com> skribis:
Toggle quote (5 lines)> The token is provided using the environment variable GUIX_SWH_TOKEN.>> * guix/swh.scm (swh-token): New variable.> (http-get*, http-post*): Use it.
[...]
Toggle quote (8 lines)> +;; Token from an account to the Software Heritage Authentication service> +;; <https://archive.softwareheritage.org/api/>> +(define swh-token> + (match (getenv "GUIX_SWH_TOKEN")> + (#f #f)> + ((? string-null? s) #f)> + ((? string? s) (string->symbol s))))
Could you turn it into a SRFI-39 parameter?
Toggle quote (19 lines)> ;; XXX: Work around a bug in Guile 3.0.2 where #:verify-certificate? would> ;; be ignored (<https://bugs.gnu.org/40486>).> (define* (http-get* uri #:rest rest)> - (apply http-request uri #:method 'GET rest))> + (apply http-request uri #:method 'GET> + #:headers> + (if swh-token> + `((authorization . (Bearer ,swh-token)))> + '())> + rest))> (define* (http-post* uri #:rest rest)> - (apply http-request uri #:method 'POST rest))> + (apply http-request uri #:method 'POST> + #:headers> + (if swh-token> + `((authorization . (Bearer ,swh-token)))> + '())> + rest))
These two procedures are meant to go away as soon as Guile > 3.0.2 isrequired.
Could you instead pass #:headers in the ‘call’ procedure, and maybe in‘vault-fetch’ too?
Thanks!
Ludo’.
Z
Z
zimoun wrote on 14 Oct 23:52 +0200
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 51216@debbugs.gnu.org)
86a6jbp941.fsf@gmail.com
Re,
On Thu, 14 Oct 2021 at 23:44, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (4 lines)>> purpose I get a lot of android packages scheduled after checking many admin.>> You mean the SWH admins manually scheduled them?
No no, sorry my bad wording. I mean “guix lint -c archival” checked allthe 260+ packages in gnu/packages/admin.scm then it checkedgnu/packages/android.scm, found some unarchived and scheduled them.

Toggle quote (11 lines)>> It is interesting to note lines as:>>>> Disarchive entry refers to non-existent SWH directory 'aeae11cb3c33ab33374e222dc3bdf17039808a5b'>>>> which shows that sources.json is maybe not fully working as expected.>> Investigations required.>> Weird. I guess we should pick a few specific cases, make sure they’re> listed in sources.json and they’re not 404, and then ask the SWH team> whether they can check their logs to understand what happened.
Yeah… investigations required. :-)

Cheers,simon
Z
Z
zimoun wrote on 15 Oct 00:14 +0200
[PATCH v2] swh: Allows token from Software Heritage authentication service.
(address . 51216@debbugs.gnu.org)(name . zimoun)(address . zimon.toutoune@gmail.com)
20211014221454.3272836-1-zimon.toutoune@gmail.com
The token is provided using the environment variable GUIX_SWH_TOKEN.
* guix/swh.scm (%swh-token): New variable.(call): Use it.--- guix/swh.scm | 11 +++++++++++ 1 file changed, 11 insertions(+)
Toggle diff (40 lines)diff --git a/guix/swh.scm b/guix/swh.scmindex 5c41685a24..c7c1c873a2 100644--- a/guix/swh.scm+++ b/guix/swh.scm@@ -2,6 +2,7 @@ ;;; Copyright © 2018, 2019, 2020, 2021 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2020 Jakub Kądziołka <kuba@kadziolka.net> ;;; Copyright © 2021 Xinglu Chen <public@yoctocell.xyz>+;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com> ;;; ;;; This file is part of GNU Guix. ;;;@@ -136,6 +137,12 @@ (define %verify-swh-certificate? ;; Whether to verify the X.509 HTTPS certificate for %SWH-BASE-URL. (make-parameter #t)) +;; Token from an account to the Software Heritage Authentication service+;; <https://archive.softwareheritage.org/api/>+(define %swh-token+ (make-parameter (and=> (getenv "GUIX_SWH_TOKEN")+ string->symbol)))+ (define (swh-url path . rest) ;; URLs returned by the API may be relative or absolute. This has changed ;; without notice before. Handle both cases by detecting whether the path@@ -246,6 +253,10 @@ (define* (call url decode #:optional (method http-get*) (and ((%allow-request?) url method) (let*-values (((response port) (method url #:streaming? #t+ #:headers+ (if (%swh-token)+ `((authorization . (Bearer ,(%swh-token))))+ '()) #:verify-certificate? (%verify-swh-certificate?)))) ;; See <https://archive.softwareheritage.org/api/#rate-limiting>.
base-commit: cf25ba2d773435af7af2cd315185acb16dcc6b58-- 2.32.0
L
L
Ludovic Courtès wrote on 15 Oct 22:54 +0200
Re: bug#51216: [PATCH 0/1] swh: Allows token from Software Heritage authentication service.
(name . zimoun)(address . zimon.toutoune@gmail.com)(address . 51216-done@debbugs.gnu.org)
87lf2udn5c.fsf_-_@gnu.org
zimoun <zimon.toutoune@gmail.com> skribis:
Toggle quote (5 lines)> The token is provided using the environment variable GUIX_SWH_TOKEN.>> * guix/swh.scm (%swh-token): New variable.> (call): Use it.
Applied, thanks! :-)
Ludo'.
Closed
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 51216@debbugs.gnu.org