Bioconductor URI, fallback and time-machine

  • Open
  • quality assurance status badge
Details
6 participants
  • Ludovic Courtès
  • Ludovic Courtès
  • Maxime Devos
  • Ricardo Wurmus
  • Timothy Sample
  • zimoun
Owner
unassigned
Submitted by
zimoun
Severity
normal
Z
Z
zimoun wrote on 3 Mar 2020 16:59
CAJ3okZ3dFunYgafRH6=9LsLKLf6OrZBpXqUMxZAjEhaiL93ARA@mail.gmail.com
Dear,

Currently, the URI scheme (see 'bioconductor-uri' in
guix/build-system/r.scm) is:

https://bioconductor.org/packages/release/data/type-url-part /src/contrib/ upstream-name - version .tar.gz

which leads to 2 issues:

1. when Bioconductor updates their release, some package versions are
updated too, and so, the upstream return 404.
2. for this reason 1., the "guix time-machine" is broken for all the
Bioconductor packages, at least if Berlin or SWH does not have a
substitute; which is not expected for 'annotation' packages.

However, the Bioconductor archive still serves the old release, i.e.,

https://bioconductor.org/packages/3.x/data/type-url-part /src/contrib/ upstream-name - version .tar.gz


The ways to fix the both issues are:

a) Add the Bioconductor release (known at packaging time) to all the
packages; provide as argument to 'bioconductor-uri'.
b) Add more URLs to fallback.

As discussed on IRC, Tobias seems more inclined with the option a) and
I am more in favour of option b.

Attached, a quick patch showing the option b).


Please also consider #36805 which was never merged or closed.


All the best,
simon
From 87e73e02202fe5e342d68f1fb17efdd4425737cd Mon Sep 17 00:00:00 2001
From: zimoun <zimon.toutoune@gmail.com>
Date: Tue, 3 Mar 2020 16:53:39 +0100
Subject: [PATCH] build-system: r: Use Bioconductor old releases to fallback.

* guix/build-system/r.scm (bioconductor-uri): Extend the fallback list.
---
guix/build-system/r.scm | 21 ++++++++++++---------
1 file changed, 12 insertions(+), 9 deletions(-)

Toggle diff (34 lines)
diff --git a/guix/build-system/r.scm b/guix/build-system/r.scm
index 2d328764b0..8638e1b888 100644
--- a/guix/build-system/r.scm
+++ b/guix/build-system/r.scm
@@ -54,15 +54,18 @@ release corresponding to NAME and VERSION."
('annotation "/data/annotation")
('experiment "/data/experiment")
(_ "/bioc"))))
- (list (string-append "https://bioconductor.org/packages/release"
- type-url-part
- "/src/contrib/"
- name "_" version ".tar.gz")
- ;; TODO: use %bioconductor-version from (guix import cran)
- (string-append "https://bioconductor.org/packages/3.10"
- type-url-part
- "/src/contrib/Archive/"
- name "_" version ".tar.gz"))))
+ (append (list (string-append "https://bioconductor.org/packages/release"
+ type-url-part
+ "/src/contrib/"
+ name "_" version ".tar.gz"))
+ (map (lambda (release)
+ (string-append "https://bioconductor.org/packages/"
+ release
+ type-url-part
+ "/src/contrib/"
+ name "_" version ".tar.gz"))
+ (list (@@ (guix import cran) %bioconductor-version)
+ "3.9" "3.8" "3.7")))))
(define %r-build-system-modules
;; Build-side modules imported by default.
--
2.25.0
R
R
Ricardo Wurmus wrote on 23 Mar 2020 22:20
(name . zimoun)(address . zimon.toutoune@gmail.com)
87ftdylqdn.fsf@elephly.net
zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (20 lines)
> 1. when Bioconductor updates their release, some package versions are
> updated too, and so, the upstream return 404.
> 2. for this reason 1., the "guix time-machine" is broken for all the
> Bioconductor packages, at least if Berlin or SWH does not have a
> substitute; which is not expected for 'annotation' packages.
>
> However, the Bioconductor archive still serves the old release, i.e.,
>
> https://bioconductor.org/packages/3.x/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
>
>
> The ways to fix the both issues are:
>
> a) Add the Bioconductor release (known at packaging time) to all the
> packages; provide as argument to 'bioconductor-uri'.
> b) Add more URLs to fallback.
>
> As discussed on IRC, Tobias seems more inclined with the option a) and
> I am more in favour of option b.

I think option a) is more explicit, which is probably what we generally
want to future-proof the time-machine. Fallbacks are okay in the case
of the CRAN URL where it’s not necessarily clear when a package tarball
moves from the release location to the archive.

In the case of Bioconductor URLs it seems that we can afford to be a bit
more accurate.

--
Ricardo
Z
Z
zimoun wrote on 22 May 2020 01:29
(name . Ricardo Wurmus)(address . rekado@elephly.net)
CAJ3okZ1Ttjh+iG3qU1a_PcK_m5-64+KLAGKYT7b5Cum7fGgkKA@mail.gmail.com
Dear Ricardo,

On Mon, 23 Mar 2020 at 22:21, Ricardo Wurmus <rekado@elephly.net> wrote:

Toggle quote (15 lines)
> > a) Add the Bioconductor release (known at packaging time) to all the
> > packages; provide as argument to 'bioconductor-uri'.
> > b) Add more URLs to fallback.
> >
> > As discussed on IRC, Tobias seems more inclined with the option a) and
> > I am more in favour of option b.
>
> I think option a) is more explicit, which is probably what we generally
> want to future-proof the time-machine. Fallbacks are okay in the case
> of the CRAN URL where it’s not necessarily clear when a package tarball
> moves from the release location to the archive.
>
> In the case of Bioconductor URLs it seems that we can afford to be a bit
> more accurate.

We are going for option a) which means rename all the URLs, right?

Because it is a lot, I suggest to first address the bug#36805, i.e.,
provide as an argument the BioConductor version to 'bioconductor-uri'
and applies this policy to all the new packages or any update of them.

Moreover, I have suggested to reorganise bioconductor.scm,
bioinformatics.scm, cran.scm, etc. and I have not dedicated enough
time to this boring task. But because I am working remotely
(semi-lockdown), I plan to work on it next week and so this change of
URLs could be part of the big reorganisation.

What do you think?



All the best,
simon
Z
Z
zimoun wrote on 24 Jun 2020 13:07
Re: bug#39885: Bioconductor URI, fallback and time-machine
CAJ3okZ1Xwd-2WArzNus3xE_KOayDdXPp+ku1SxYBon4Zm0qQQg@mail.gmail.com
Dear,

The time-machine is broken for some BioConductor packages.. For an
example, consider the package "r-genomegraphs" which has been removed
from the BioConductor in 3.11 release.

(Well, now the issue is mitigated because ci.guix.gnu.org serves a lot
of upstream substitutes but ci.guix.gnu.org could be down. Other
said, we should use the upstream resources where they are available.)


Concretely, there are 2 issues:

a) What to do for the removed packages? For 3.11, the list is there
[1]. Do we keep them in gnu/packages/bioconductor.scm but then
'bioconductor-uri' needs some tweaks? Or do we transfer them to the
channel guix-past (for example)?

b) The fallback URI in guix/build-system/r.scm(bioconductor-uri)
added by commit c586f427b4831b9b492e5b900b2226e898b8fcfa is not
correct, if I do not misread:

Toggle snippet (4 lines)
"https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/GenomeGraphs_1.46.0.tar.gz"
404 "Not Found"

The correct seems to be (without Archive):



All the best,
simon

L
L
Ludovic Courtès wrote on 28 Jun 2020 22:14
(name . zimoun)(address . zimon.toutoune@gmail.com)
87lfk7gd7d.fsf@gnu.org
Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (11 lines)
> b) The fallback URI in guix/build-system/r.scm(bioconductor-uri)
> added by commit c586f427b4831b9b492e5b900b2226e898b8fcfa is not
> correct, if I do not misread:
>
> "https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/GenomeGraphs_1.46.0.tar.gz"
> 404 "Not Found"
>
> The correct seems to be (without Archive):
>
> https://bioconductor.org/packages/3.10/bioc/src/contrib/GenomeGraphs_1.46.0.tar.gz

Could you provide a patch for this?

Thanks,
Ludo’.
Z
Z
zimoun wrote on 29 Jun 2020 19:36
(name . Ludovic Courtès)(address . ludo@gnu.org)
CAJ3okZ36KrALgMq69zdkDsHcfMv09Lk=DPBmzW-4SePZPRAvnA@mail.gmail.com
Hi Ludo,

On Sun, 28 Jun 2020 at 22:14, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (2 lines)
> Could you provide a patch for this?

About the url, for sure, see attached.

But it does not address the root of the problem. Well, I will try to
find a slot and propose something.


All the best,
simon
From c1c963a3b86e306a20c14626127e54d21843c22c Mon Sep 17 00:00:00 2001
From: zimoun <zimon.toutoune@gmail.com>
Date: Mon, 29 Jun 2020 19:18:20 +0200
Subject: [PATCH] build-system/r: bioconductor-uri: Fix archive URL.

* guix/build-system/r.scm (bioconductor-uri): Fix archive URL.
---
guix/build-system/r.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (17 lines)
diff --git a/guix/build-system/r.scm b/guix/build-system/r.scm
index c8ec9abd0d..5ef982d66a 100644
--- a/guix/build-system/r.scm
+++ b/guix/build-system/r.scm
@@ -61,7 +61,7 @@ release corresponding to NAME and VERSION."
;; TODO: use %bioconductor-version from (guix import cran)
(string-append "https://bioconductor.org/packages/3.11"
type-url-part
- "/src/contrib/Archive/"
+ "/src/contrib/"
name "_" version ".tar.gz"))))
(define %r-build-system-modules

base-commit: 6ebf300959a58fd1eda875205c75d21137862285
--
2.26.2
L
L
Ludovic Courtès wrote on 29 Jun 2020 22:42
(name . zimoun)(address . zimon.toutoune@gmail.com)
87r1txa9ix.fsf@gnu.org
zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (7 lines)
> From c1c963a3b86e306a20c14626127e54d21843c22c Mon Sep 17 00:00:00 2001
> From: zimoun <zimon.toutoune@gmail.com>
> Date: Mon, 29 Jun 2020 19:18:20 +0200
> Subject: [PATCH] build-system/r: bioconductor-uri: Fix archive URL.
>
> * guix/build-system/r.scm (bioconductor-uri): Fix archive URL.

Applied, thanks!

I let the rest of you discuss the other issues. :-)

Ludo’.
Z
Z
zimoun wrote on 19 Nov 2020 15:22
(address . 39885@debbugs.gnu.org)
87r1ope8a4.fsf@gmail.com
Hi,

Some explanations of the issue are provided here:


Since we are currently updating to 3.12, maybe it is the occasion to fix
the issue. See option a) below.


On Tue, 03 Mar 2020 at 16:59, zimoun <zimon.toutoune@gmail.com> wrote:

Toggle quote (14 lines)
> Currently, the URI scheme (see 'bioconductor-uri' in
> guix/build-system/r.scm) is:
>
> https://bioconductor.org/packages/release/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
>
> which leads to 2 issues:
>
> 1. when Bioconductor updates their release, some package versions are
> updated too, and so, the upstream return 404.
>
> 2. for this reason 1., the "guix time-machine" is broken for all the
> Bioconductor packages, at least if Berlin or SWH does not have a
> substitute; which is not expected for 'annotation' packages.

An example of this issue is for example:

Toggle snippet (24 lines)
$ guix time-machine --commit=aee183e -- import cran -a bioconductor CATALYST -r

Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...

Starting download of /tmp/guix-file.Nxajqh
From https://bioconductor.org/packages/release/bioc/src/contrib/CATALYST_1.12.2.tar.gz...
download failed "https://bioconductor.org/packages/release/bioc/src/contrib/CATALYST_1.12.2.tar.gz" 404 "Not Found"
failed to download "/tmp/guix-file.Nxajqh" from "https://bioconductor.org/packages/release/bioc/src/contrib/CATALYST_1.12.2.tar.gz"
error: failed to retrieve package information from "https://cran.r-project.org/web/packages/CATALYST/DESCRIPTION": 404 ("Not Found")
Backtrace:
4 (primitive-load "/home/simon/.cache/guix/inferiors/vznc…")
In guix/ui.scm:
2117:12 3 (run-guix-command _ . _)
In guix/scripts/import.scm:
120:11 2 (guix-import . _)
In srfi/srfi-1.scm:
586:17 1 (map1 (#f))
In guix/import/utils.scm:
258:2 0 (package->definition _)

guix/import/utils.scm:258:2: In procedure package->definition:
Throw to key `match-error' with args `("match" "no matching pattern" #f)'.

Aside the ugly backtrace which is tracked by #44115, the main issue is
because Bioconductor updated to 3.12 and Guix is still at 3.11.

Concretely, the issue is that ’release’ in the URL:


now refers to 3.12 (because Bioconductor update) and Guix still think it
is 3.11 (because Guix has not yet updated; work-in-progress). And
CATALYST in 3.12 is at version 1.14.0 against 1.12.2 for 3.11.
Therefore, the conflict and the error.

It means that while:

(define %bioconductor-version "3.11")

is not updated to 3.12, all the Bioconductor packages are broken; in the
meaning not buildable from source.


Toggle quote (9 lines)
> a) Add the Bioconductor release (known at packaging time) to all the
> packages; provide as argument to 'bioconductor-uri'.
> b) Add more URLs to fallback, e.g.:
>
> https://bioconductor.org/packages/release/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
> https://bioconductor.org/packages/3.11/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
>
> Attached, a quick patch showing the option b).

Then each time we update Bioconductor, we add an URL to the list.


Toggle quote (3 lines)
> As discussed on IRC, Tobias seems more inclined with the option a) and
> I am more in favour of option b.

Tobias and Ricardo are in favor for a) (see this thread). Which means a
lot of work IMHO, i.e., add 3.11 as arguments and then 3.12 to all the
Bioconductor packages and fix the importer, IIUC; while b) means do
nothing except merge the proposed patch (possibly re-worked).

Just to note that only the task to group in bioconductor.scm all the
Bioconductor packages scattered here and there is still not done, I
think option a) is not doable by hand – I do not volunteer! :-) Else,
any suggestion to script the task instead?

Since I am more in favor of b), I am less motivated to fix the a). ;-)
But I am motivated to fix the issue at hand. :-)


Other option c) is to switch all the Bioconductor to git-fetch instead
of url-fetch. I have not checked yet how could be the transition.


Toggle quote (3 lines)
> Please also consider #36805 which was never merged or closed.
> http://issues.guix.gnu.org/issue/36805

This patch could help for option a).


WDYT?

All the best,
simon
Z
Z
zimoun wrote on 22 Nov 2021 20:48
(address . 39885@debbugs.gnu.org)
87wnl0q8ei.fsf@gmail.com
Hi,

On Tue, 03 Mar 2020 at 16:59, zimoun <zimon.toutoune@gmail.com> wrote:

Toggle quote (17 lines)
> Currently, the URI scheme (see 'bioconductor-uri' in
> guix/build-system/r.scm) is:
>
> https://bioconductor.org/packages/release/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
>
> which leads to 2 issues:
>
> 1. when Bioconductor updates their release, some package versions are
> updated too, and so, the upstream return 404.
> 2. for this reason 1., the "guix time-machine" is broken for all the
> Bioconductor packages, at least if Berlin or SWH does not have a
> substitute; which is not expected for 'annotation' packages.
>
> However, the Bioconductor archive still serves the old release, i.e.,
>
> https://bioconductor.org/packages/3.x/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz

It is still the case and for concrete breakage, see [1]. I will not
detail but each time Guix lags behind Bioconductor new release, it is
broken. For sure, Guix upgrades more or less quickly. Each time
Bioconductor remove a package, it is broken. Well, because a lot
of care about R packages, the forward breakages happen barely. :-) But
backward breakages are not negligible, IMHO.


Well, this URL choice is not The Right Thing and somehow broken by design.



Toggle quote (11 lines)
> The ways to fix the both issues are:
>
> a) Add the Bioconductor release (known at packaging time) to all the
> packages; provide as argument to 'bioconductor-uri'.
> b) Add more URLs to fallback.
>
> As discussed on IRC, Tobias seems more inclined with the option a) and
> I am more in favour of option b.
>
> Attached, a quick patch showing the option b).

We are now 1.5 years after. And we did nothing; well we did other
things instead. ;-). Now, I have an strong opinion that option a) is
not doable: I speak using my janitor moves of Bioconductor packages.

Instead, something along the proposed patch below half-fixes the issue
now. We just have to append the releases and let the fallback mechanism
takes care. It reduces the maintenance burden, IMHO.

For sure, it is not perfect but it appears to me a pragmatical fix
waiting something better.


This better is unknown (at least from me :-)). On one hand Disarchive
would improve the situation for tarballs… but some work remains (check
that SWH ingestion and rebuild is bullet-proof). On the other hand,
Bioconductor uses Git, for instance:



And Bioconductor uses ’origin/RELEASE_3.14’ as Git tag. Based on this,
it would avoid the eternal inplace-change fixes.

For instance, the package tximeta [2], recently updated by Ricardo.
Well, from their Bioconductor Git repo,


it is not clear that the current version is at 1.12.3. And it is not
clear either if they tagged origin/RELEASE_3_14 at 1.12.0 and did
something ugly to then get 1.12.3. Anyway, switch from url-fetch to
git-fetch is an option. However, it is as option a) and I am not
convinced it is doable with the resource at hand.



What could a plan to have a bullet-proof “guix time-machine” for
Bioconductor?


Cheers,
simon


Toggle quote (42 lines)
> From 87e73e02202fe5e342d68f1fb17efdd4425737cd Mon Sep 17 00:00:00 2001
> From: zimoun <zimon.toutoune@gmail.com>
> Date: Tue, 3 Mar 2020 16:53:39 +0100
> Subject: [PATCH] build-system: r: Use Bioconductor old releases to fallback.
>
> * guix/build-system/r.scm (bioconductor-uri): Extend the fallback list.
> ---
> guix/build-system/r.scm | 21 ++++++++++++---------
> 1 file changed, 12 insertions(+), 9 deletions(-)
>
> diff --git a/guix/build-system/r.scm b/guix/build-system/r.scm
> index 2d328764b0..8638e1b888 100644
> --- a/guix/build-system/r.scm
> +++ b/guix/build-system/r.scm
> @@ -54,15 +54,18 @@ release corresponding to NAME and VERSION."
> ('annotation "/data/annotation")
> ('experiment "/data/experiment")
> (_ "/bioc"))))
> - (list (string-append "https://bioconductor.org/packages/release"
> - type-url-part
> - "/src/contrib/"
> - name "_" version ".tar.gz")
> - ;; TODO: use %bioconductor-version from (guix import cran)
> - (string-append "https://bioconductor.org/packages/3.10"
> - type-url-part
> - "/src/contrib/Archive/"
> - name "_" version ".tar.gz"))))
> + (append (list (string-append "https://bioconductor.org/packages/release"
> + type-url-part
> + "/src/contrib/"
> + name "_" version ".tar.gz"))
> + (map (lambda (release)
> + (string-append "https://bioconductor.org/packages/"
> + release
> + type-url-part
> + "/src/contrib/"
> + name "_" version ".tar.gz"))
> + (list (@@ (guix import cran) %bioconductor-version)
> + "3.9" "3.8" "3.7")))))
>
> (define %r-build-system-modules
> ;; Build-side modules imported by default.
Z
Z
zimoun wrote on 18 Jul 2022 18:03
(address . 39885@debbugs.gnu.org)
87lesqmmrr.fsf@gmail.com
Hi,

Since 2020, I provided several examples of breakage with bug#39885 [1].
Here another one:

Toggle snippet (40 lines)
$ guix time-machine --commit=77e2de365497bf4c8b81cbd78624f78293490485 \
-- build r-biocneighbors -S
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivation will be built:
/gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv
building /gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv...

Starting download of /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz
From https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz...
download failed "https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz" 404 "Not Found"

Starting download of /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz
From https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/BiocNeighbors_1.4.1.tar.gz...
download failed "https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/BiocNeighbors_1.4.1.tar.gz" 404 "Not Found"

Starting download of /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz
From https://ci.guix.gnu.org/file/BiocNeighbors_1.4.1.tar.gz/sha256/05vi1cij37s8wgj92k3l6a3f3dwldj8jvijdp4695zczka6kypdf...
download failed "https://ci.guix.gnu.org/file/BiocNeighbors_1.4.1.tar.gz/sha256/05vi1cij37s8wgj92k3l6a3f3dwldj8jvijdp4695zczka6kypdf" 404 "Not Found"

Starting download of /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz
From https://tarballs.nixos.org/sha256/05vi1cij37s8wgj92k3l6a3f3dwldj8jvijdp4695zczka6kypdf...
download failed "https://tarballs.nixos.org/sha256/05vi1cij37s8wgj92k3l6a3f3dwldj8jvijdp4695zczka6kypdf" 404 "Not Found"

Starting download of /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz
From https://archive.softwareheritage.org/api/1/content/sha256:ae5d3f8d9a9ffd920cb94dc62d916c94b7e18632744c91e4e3489f21230b7117/raw/...
download failed "https://archive.softwareheritage.org/api/1/content/sha256:ae5d3f8d9a9ffd920cb94dc62d916c94b7e18632744c91e4e3489f21230b7117/raw/" 404 "Not Found"

Starting download of /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz
From https://web.archive.org/web/20220718175152/https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz...
download failed "https://web.archive.org/web/20220718175152/https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz" 404 "NOT FOUND"
Trying to use Disarchive to assemble /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz...
could not find its Disarchive specification
failed to download "/gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz" from ("https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz" "https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/BiocNeighbors_1.4.1.tar.gz")
builder for `/gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv' failed to produce output path `/gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz'
build of /gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv failed
View build log at '/var/log/guix/drvs/q9/ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv.gz'.
guix build: error: build of `/gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv' failed

Well, several comments:

1. Berlin or Bordeaux do not have it as substitutes,
2. Diasarchive does not have it,
3. Many others neither.

but the question in the first place is: why is Bioconductor failing?
Because they do ugly things!

Our history reads:

f431d5e299 Sun Dec 15 15:38:51 2019 +0100 guix: Upgrade to Bioconductor 3.10
12e2aa96dc Sun Dec 15 15:38:55 2019 +0100 gnu: r-biocneighbors: Update to 1.4.1.
aece78fe2f Sun Mar 1 23:38:12 2020 +0100 gnu: r-biocneighbors: Update to 1.4.2.
8e518d4802 Sat Jun 13 01:19:38 2020 +0200 guix: Update to Bioconductor 3.11.

which means that Bioconductor removes v1.4.1 from their URI scheme
(even, I do not know if the tarball is still available on their infra)
and despite the fact Bioconductor v3.10 had released v1.4.1, then it is
not stable.

At the cost of more bandwidth, we could switch from url-fetch to
git-fetch. Or we also could examine why Disarchive is failing here.



Cheers,
simon
R
R
Ricardo Wurmus wrote on 18 Jul 2022 18:21
(name . zimoun)(address . zimon.toutoune@gmail.com)
87bktmtmol.fsf@elephly.net
zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (3 lines)
> At the cost of more bandwidth, we could switch from url-fetch to
> git-fetch.

Let’s do it! I’m tired of Bioconductor archive shenanigans messing with
package availability.

--
Ricardo
R
R
Ricardo Wurmus wrote on 10 Aug 2022 20:25
(name . zimoun)(address . zimon.toutoune@gmail.com)
878rnwuemq.fsf@elephly.net
Ricardo Wurmus <rekado@elephly.net> writes:

Toggle quote (8 lines)
> zimoun <zimon.toutoune@gmail.com> writes:
>
>> At the cost of more bandwidth, we could switch from url-fetch to
>> git-fetch.
>
> Let’s do it! I’m tired of Bioconductor archive shenanigans messing with
> package availability.

I have finally taken the time to review this and implement a first draft
of a change to the bioconductor importer and updater.

There are some limitations:

- we cannot use the updater to go from “url-fetch” to “git-fetch”.
That’s because “package-update” in (guix upstream) decides whether to
use package-update/url-fetch or package-update/git-fetch based on the
*current* package value’s origin fetch procedure. For the switch we
can hack around this (adding an exception for bioconductor packages),
but there is no pretty way to do this in a generic fashion that could
be committed.

Perhaps we could operate on the url included in the <upstream-source>
instead of looking at the *current* package value. We’re only
accessing “package” once in the url-fetch case, so maybe we can work
around this problem.

- the repositories at https://git.bioconductor.org/package/NAMEdo not
tag package versions. The only method of organization is branches
that are named after *Bioconductor releases* (not package releases),
e.g. RELEASE_3_15. We can only determine the package version by
reading its DESCRIPTION file or by looking up the version index for
all Bioconductor packages (we do that already). This means that there
could be different commits for the same package version in the same
release branch — so we have to include the commit hash and a revision
counter in the version string.

- the updater doesn’t work on version expressions like (git-version
"1.12" revision commit). It expects to be able to replace literal
strings. Because of that my changes let the importer generate a
string literal such as "1.12-0.cafebab" without a let-bound commit
string.

- “experiment” or “data” packages are not kept in Git. They only exist
as volatile tarballs that will be overwritten. Thankfully, they don’t
change all that often, so they have a good chance of making it into
our archives.

- the above exception means that we need to litter the importer and
updater code with extra checks.

With all these notes out of the way I’ll prepare a series of patches
next.

--
Ricardo
M
M
Maxime Devos wrote on 10 Aug 2022 21:44
4041b7dd-df1d-318c-0ca9-efe296203ea9@telenet.be
On 10-08-2022 20:25, Ricardo Wurmus wrote:
Toggle quote (5 lines)
> - the updater doesn’t work on version expressions like (git-version
> "1.12" revision commit). It expects to be able to replace literal
> strings. Because of that my changes let the importer generate a
> string literal such as "1.12-0.cafebab" without a let-bound commit
> string.
I've a patch that implements replacing (revision "N") by (revision
"N+1"), apparently it's not applied yet but let me search for it ...
Attachment: OpenPGP_0x49E3EE22191725EE.asc
Attachment: OpenPGP_signature
M
M
Maxime Devos wrote on 10 Aug 2022 21:48
0d6b6bae-4ab3-aad5-f03d-f6e369620267@telenet.be
On 10-08-2022 21:44, Maxime Devos wrote:
Toggle quote (9 lines)
>
> On 10-08-2022 20:25, Ricardo Wurmus wrote:
>> - the updater doesn’t work on version expressions like (git-version
>>    "1.12" revision commit).  It expects to be able to replace literal
>>    strings.  Because of that my changes let the importer generate a
>>    string literal such as "1.12-0.cafebab" without a let-bound commit
>>    string.
> I've a patch that implements replacing (revision "N") by (revision
> "N+1"), apparently it's not applied yet but let me search for it ...
Found it:
That patch series was written with Minetest / ContentDB and a new
'latest-git' updater in mind, but the ContentDB and latest-git bits
should be separable without much trouble.
Greetings,
Maxime.
Attachment: OpenPGP_0x49E3EE22191725EE.asc
Attachment: OpenPGP_signature
Z
Z
zimoun wrote on 9 Sep 2022 19:23
(name . Ricardo Wurmus)(address . rekado@elephly.net)
865yhwpim7.fsf@gmail.com
Hi Ricardo,

I am late. This message landed when I was traveling for holidays. :-)

On Wed, 10 Aug 2022 at 20:25, Ricardo Wurmus <rekado@elephly.net> wrote:

Toggle quote (8 lines)
> - we cannot use the updater to go from “url-fetch” to “git-fetch”.
> That’s because “package-update” in (guix upstream) decides whether to
> use package-update/url-fetch or package-update/git-fetch based on the
> *current* package value’s origin fetch procedure. For the switch we
> can hack around this (adding an exception for bioconductor packages),
> but there is no pretty way to do this in a generic fashion that could
> be committed.

It appears to me acceptable to have an exception. Or even to do it just
once as a big replacement of Bioconductor packages.

Toggle quote (10 lines)
> - the repositories at https://git.bioconductor.org/package/NAMEdo not
> tag package versions. The only method of organization is branches
> that are named after *Bioconductor releases* (not package releases),
> e.g. RELEASE_3_15. We can only determine the package version by
> reading its DESCRIPTION file or by looking up the version index for
> all Bioconductor packages (we do that already). This means that there
> could be different commits for the same package version in the same
> release branch — so we have to include the commit hash and a revision
> counter in the version string.

This is the most annoying part. Indeed, when I check out some
Bioconductor Git repositories, I am always confused by their Git
structure.

From my understanding, the tarball you fetch from bioconductor.org has
the same content than the commit tagged “Bioconductor release”
(RELEASE_X_Y). The content of the upstream release can mismatch the
content of the Bioconductor tarball release.

I do not know how it would be complicated or inaccurate to consider the
package version from the Bioconductor index and assign this version to
the commit tagged RELEASE_X_Y. This commit would appear in the Guix
package definition though. Or maybe we transparently could RELEASE_X_Y
to determine this commit.


Toggle quote (6 lines)
> - the updater doesn’t work on version expressions like (git-version
> "1.12" revision commit). It expects to be able to replace literal
> strings. Because of that my changes let the importer generate a
> string literal such as "1.12-0.cafebab" without a let-bound commit
> string.

Maxime pointed patch#53144 [1] but I have not looked at it yet.




Toggle quote (5 lines)
> - “experiment” or “data” packages are not kept in Git. They only exist
> as volatile tarballs that will be overwritten. Thankfully, they don’t
> change all that often, so they have a good chance of making it into
> our archives.

That’s an interesting question for Disarchive and Software Heritage.


Cheers,
simon
L
L
Ludovic Courtès wrote on 22 Dec 2023 14:40
Bioconductor tarballs are not archived
(name . zimoun)(address . zimon.toutoune@gmail.com)
874jgacq4u.fsf_-_@gnu.org
Hello!

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (6 lines)
> Since 2020, I provided several examples of breakage with bug#39885 [1].
> Here another one:
>
> $ guix time-machine --commit=77e2de365497bf4c8b81cbd78624f78293490485 \
> -- build r-biocneighbors -S

[...]

Toggle quote (17 lines)
> Starting download of /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz
>>From https://web.archive.org/web/20220718175152/https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz...
> download failed "https://web.archive.org/web/20220718175152/https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz" 404 "NOT FOUND"
> Trying to use Disarchive to assemble /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz...
> could not find its Disarchive specification
> failed to download "/gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz" from ("https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz" "https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/BiocNeighbors_1.4.1.tar.gz")
> builder for `/gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv' failed to produce output path `/gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz'
> build of /gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv failed
> View build log at '/var/log/guix/drvs/q9/ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv.gz'.
> guix build: error: build of `/gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv' failed
>
> Well, several comments:
>
> 1. Berlin or Bordeaux do not have it as substitutes,
> 2. Diasarchive does not have it,
> 3. Many others neither.

I was wondering whether we’re now doing better for Bioconductor
tarballs. The answer, based on small sample, seems to be “not quite”:

Toggle snippet (27 lines)
$ guix lint -c archival $(guix package -A ^r-bioc | cut -f1)
gnu/packages/bioconductor.scm:19708:12: r-biocbaseutils@1.4.0: Disarchive entry refers to non-existent SWH directory '726af85395d163b5a21e52e4df1bf18aa0072f6b'
gnu/packages/bioconductor.scm:19752:12: r-bioccheck@1.38.0: Disarchive entry refers to non-existent SWH directory '12cfedcbc27005a3fb7e01c5c4b727e0116f596f'
gnu/packages/bioconductor.scm:16892:5: r-biocfilecache@2.10.1: Disarchive entry refers to non-existent SWH directory '6a2d6d909a7cedd56e96f5a98770deeaaaa8d220'
gnu/packages/bioconductor.scm:4540:12: r-biocgenerics@0.48.1: Disarchive entry refers to non-existent SWH directory '6f19ea14f46dbc75909b77bc08e9023daae6fb9e'
gnu/packages/bioconductor.scm:19785:5: r-biocgraph@1.64.0: Disarchive entry refers to non-existent SWH directory '977ff052b4e6c948af7af0fc14ae61f71427cb1a'
gnu/packages/bioconductor.scm:21524:6: r-biocio@1.12.0: Disarchive entry refers to non-existent SWH directory '29d8fef9a5b386384f20513c612f1e34f6118532'
gnu/packages/bioconductor.scm:13090:5: r-biocneighbors@1.20.0: Disarchive entry refers to non-existent SWH directory '6d3728b2dee78cceecdeba0318f3e57b6013d96f'
gnu/packages/bioconductor.scm:19957:5: r-bioconcotk@1.22.0: Disarchive entry refers to non-existent SWH directory '251081d4bc3f061ef8e16338eb042ad4c71ed02d'
gnu/packages/bioconductor.scm:20003:5: r-biocor@1.26.0: Disarchive entry refers to non-existent SWH directory '0cc9d3dcde06fb353cdd77f3b538845d16a77720'
gnu/packages/bioconductor.scm:6613:12: r-biocparallel@1.36.0: Disarchive entry refers to non-existent SWH directory '41e09414898f61655bcc99fdd44d69b0531c0b2d'
gnu/packages/bioconductor.scm:20030:5: r-biocpkgtools@1.20.0: Disarchive entry refers to non-existent SWH directory '55de8618648ed16797a8effd5b508c652a5d7cbe'
gnu/packages/bioconductor.scm:20144:5: r-biocset@1.16.0: Disarchive entry refers to non-existent SWH directory '1cfa6cac0cb453f2882a35c8f5ae6ddfa713ad2d'
gnu/packages/bioconductor.scm:13276:5: r-biocsingular@1.18.0: Disarchive entry refers to non-existent SWH directory '992d3f9d48633fa5d46b9a7640a825054e9538aa'
gnu/packages/bioconductor.scm:19806:12: r-biocstyle@2.30.0: Disarchive entry refers to non-existent SWH directory 'bb17c3bd9ac7c373b24782fcfecdde5fa2f0a965'
gnu/packages/bioconductor.scm:22965:5: r-biocthis@1.12.0: Disarchive entry refers to non-existent SWH directory '3d08f77aae1e81ce9ca9bb9ae2adf4d4c7421d11'
gnu/packages/bioconductor.scm:4521:5: r-biocversion@3.18.1: source not archived on Software Heritage and missing from the Disarchive database
gnu/packages/bioconductor.scm:19830:12: r-biocviews@1.70.0: Disarchive entry refers to non-existent SWH directory '47e0877ab988469fc09a37505dd769f9626cac2e'
gnu/packages/bioconductor.scm:20182:5: r-biocworkflowtools@1.28.0: Disarchive entry refers to non-existent SWH directory '393f3472cc27f632caea3488aef93a7675b403ef'
$ guix describe
Generation 285 Dec 17 2023 23:31:56 (current)
guix 6ab2426
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 6ab242609daec00e8bd54f7bff54557c92695724

In all cases but one, we’re doing the right thing Disarchive-wise, but
our SWH did not archive them.


Toggle snippet (15 lines)
{
"type": "url",
"urls": [
"https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz",
"https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz",
"https://bordeaux.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas",
"https://ci.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas",
"https://tarballs.nixos.org/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas"
],
"integrity": "sha256-WlUXSI41dP7xSTx81ibFsrIsACW5jmzaX5I/lxJ4vCg=",
"outputHashAlgo": "sha256",
"outputHashMode": "flat"
},

Note that we have at least one copy on our infra:

Toggle snippet (4 lines)
$ wget -qO- "https://bordeaux.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas"|guix hash - -f base64
WlUXSI41dP7xSTx81ibFsrIsACW5jmzaX5I/lxJ4vCg=

is 404 (but I can see why: for /file, ‘guix publish’ relies on things
being available in the store and we no longer keep them on ci.guix; we
do have a substitute at
though; we should fix this).

What about hypothesis (2)? This is what we have:

Toggle snippet (4 lines)
$ wget -qO- https://disarchive.guix.gnu.org/sha256//5a5517488e3574fef1493c7cd626c5b2b22c0025b98e6cda5f923f971278bc28 |grep swh
(swhid "swh:1:dir:6d3728b2dee78cceecdeba0318f3e57b6013d96f"))

I checked with folks on #swh-devel and it turns out that “the legacy
nixguix lister that is still used in production did not detect the
fallback URL as a tarball URL” (the bordeaux.guix.gnu.org URL), but this
is fixed in the new lister, which should be in production “soon”.

As for past tarballs, #swh-devel comrades say we could send them a list
of URLs and they’d create “Save Code Now” requests on our behalf (we
cannot do it ourselves since the site doesn’t accept plain tarballs.)

Any volunteer to write a script that’d generate a list of Bioconductor
content-addressed URLs (the bordeaux.guix.gnu.org/file ones) for say the
past couple of years?

Thanks!

Ludo’.
L
L
Ludovic Courtès wrote on 22 Dec 2023 21:57
Re: bug#39885: Bioconductor URI, fallback and time-machine
(name . zimoun)(address . zimon.toutoune@gmail.com)
87ttoa7y6p.fsf@gnu.org
zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (3 lines)
> $ guix time-machine --commit=77e2de365497bf4c8b81cbd78624f78293490485 \
> -- build r-biocneighbors -S

[...]

Toggle quote (5 lines)
> Trying to use Disarchive to assemble /gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz...
> could not find its Disarchive specification
> failed to download "/gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz" from ("https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.4.1.tar.gz" "https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/BiocNeighbors_1.4.1.tar.gz")
> builder for `/gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv' failed to produce output path `/gnu/store/zgf7x09kgiqbvj0dmhplxi1xzpljxd7k-BiocNeighbors_1.4.1.tar.gz'

In hindsight this is not surprising: this is a Dec. 2019 commit and I
disarchive.guix.gnu.org, and related machinery in Sep/Oct 2021.

Of course Timothy set up https://disarchive.ngyro.com earlier, but not
so much—Timothy started work on Disarchive ca. July 2020:

(Not that it helps but at least it’s a relief to know that this
particular problem predates our more serious efforts.)

Ludo’.
S
S
Simon Tournier wrote on 2 Jan 10:20 +0100
(name . Ludovic Courtès)(address . ludo@gnu.org)
877cks2isk.fsf@gmail.com
Hi Ludo,

On Fri, 22 Dec 2023 at 21:57, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (11 lines)
> In hindsight this is not surprising: this is a Dec. 2019 commit and I
> set up <https://ci.guix.gnu.org/jobset/disarchive>,
> disarchive.guix.gnu.org, and related machinery in Sep/Oct 2021.
>
> Of course Timothy set up <https://disarchive.ngyro.com> earlier, but not
> so much—Timothy started work on Disarchive ca. July 2020:
> <https://issues.guix.gnu.org/42162#15>.
>
> (Not that it helps but at least it’s a relief to know that this
> particular problem predates our more serious efforts.)

Yeah! On the other hand, I wish that Guix will be able to build all –
or at least most of – the packages that time-machine is able to reach –
say Guix v1.0. :-) Let be ambitious. ;-)

Cheers,
simon
S
S
Simon Tournier wrote on 8 Jan 10:09 +0100
Re: bug#39885: Bioconductor tarballs are not archived
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
87le9019aj.fsf@gmail.com
Hi,

On Fri, 22 Dec 2023 at 14:40, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

Toggle quote (2 lines)
>> guix build: error: build of `/gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv' failed

First thing first, please note that we are speaking about tag 1.4.1 and
not 1.20.0. And this 1.4.1 is gone from “our” infra since… ?? That’s
one of the things I do not like with Guix: I never know what to expect
from the infra. Anyway, I have my list of TODOs for improving the
annoyances (I and maybe others have :-)); stay tuned. ;-)

Considering the state of “our” infra and how Bioconductor manages the
tarballs, many tarballs are lost forever, sadly. Although the content
is still around, I guess.


Toggle quote (3 lines)
> I was wondering whether we’re now doing better for Bioconductor
> tarballs. The answer, based on small sample, seems to be “not quite”:

Thanks for diving into this.


Toggle quote (18 lines)
>
> --8<---------------cut here---------------start------------->8---
> {
> "type": "url",
> "urls": [
> "https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz",
> "https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz",
> "https://bordeaux.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas",
> "https://ci.guix.gnu.org/file/BiocNeighbors_1.20.0.tar.gz/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas",
> "https://tarballs.nixos.org/sha256/0a5wg099fgwjbzd6r3mr4l02rcmjqlkdcz1w97qzwx1mir41fmas"
> ],
> "integrity": "sha256-WlUXSI41dP7xSTx81ibFsrIsACW5jmzaX5I/lxJ4vCg=",
> "outputHashAlgo": "sha256",
> "outputHashMode": "flat"
> },
> --8<---------------cut here---------------end--------------->8---

Please note that Bioconductor 3.18 released BiocNeighbors v1.20.0 but
then updated to v1.20.1 still under Bioconductor 3.18 and Ricardo did
this update with 5673484cbc2ed74c61ae81d623646fa7829fbc32. On a side
note, between the Bioconductor update and the update on our side, there
is a mismatch where the source of r-biocneighbors is unreachable.

Other said, post-update on our side,

Toggle snippet (20 lines)
$ zcat sources.json | jq | grep BiocNeighbors | grep bioconductor | sed 's/"//g' | sed 's/,//g'
https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.1.tar.gz
https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.1.tar.gz

$ for url in $(zcat sources.json | jq | grep BiocNeighbors | grep bioconductor | sed 's/"//g' | sed 's/,//g'); \
do guix download $url ;done

Starting download of /tmp/guix-file.STc9fQ
From https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.1.tar.gz...
…_1.20.1.tar.gz 1015KiB 60.3MiB/s 00:00 ???????????????????? 100.0%
/gnu/store/nxab1pskh9zcjspczph6jcs5fk79pb7k-BiocNeighbors_1.20.1.tar.gz
0w7hd6w0lmj1jaaq9zd5gwnnpkzcr0byqm5q584wjg4xgvsb981j

Starting download of /tmp/guix-file.aZFRLv
From https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.1.tar.gz...
…_1.20.1.tar.gz 1015KiB 63.5MiB/s 00:00 ???????????????????? 100.0%
/gnu/store/nxab1pskh9zcjspczph6jcs5fk79pb7k-BiocNeighbors_1.20.1.tar.gz
0w7hd6w0lmj1jaaq9zd5gwnnpkzcr0byqm5q584wjg4xgvsb981j

but, now the past reads,

Toggle snippet (30 lines)
$ for url in https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz \
https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz ; \
do guix download $url ;done

> >
Starting download of /tmp/guix-file.MUB3ow
From https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz...
download failed "https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz" 404 "Not Found"

Starting download of /tmp/guix-file.MUB3ow
From https://web.archive.org/web/20240102105016/https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz...
download failed "https://web.archive.org/web/20240102105016/https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz" 404 "NOT FOUND"
Trying to use Disarchive to assemble /tmp/guix-file.MUB3ow...
could not find its Disarchive specification
failed to download "/tmp/guix-file.MUB3ow" from "https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz"
guix download: error: https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz: download failed

Starting download of /tmp/guix-file.ZO9N08
From https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz...
download failed "https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz" 404 "Not Found"

Starting download of /tmp/guix-file.ZO9N08
From https://web.archive.org/web/20240102105018/https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz...
download failed "https://web.archive.org/web/20240102105018/https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz" 404 "NOT FOUND"
Trying to use Disarchive to assemble /tmp/guix-file.ZO9N08...
could not find its Disarchive specification
failed to download "/tmp/guix-file.ZO9N08" from "https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz"
guix download: error: https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz: download failed

As explained in [1], Bioconductor removes v1.20.0 from their URI scheme
and despite the fact Bioconductor v3.18 had released v1.20.0, arf! And
I do not even know if the tarball of v1.20.0 is kept on Bioconductor
infra. Hum?

Hence the discussion we had: switch from url-fetch to git-fetch.
However, after some investigations, it does not seem straightforward:
The main issue being the almost automatic current updater. See for
details [2].


1: bug#39885: Bioconductor URI, fallback and time-machine
zimoun <zimon.toutoune@gmail.com>
Mon, 18 Jul 2022 18:03:04 +0200
id:87lesqmmrr.fsf@gmail.com

2: bug#39885: Bioconductor URI, fallback and time-machine
Ricardo Wurmus <rekado@elephly.net>
Wed, 10 Aug 2022 20:25:00 +0200
id:878rnwuemq.fsf@elephly.net


Toggle quote (4 lines)
> Any volunteer to write a script that’d generate a list of Bioconductor
> content-addressed URLs (the bordeaux.guix.gnu.org/file ones) for say the
> past couple of years?

I did stuff past week about that. I will report this week what I did.

Cheers,
simon
L
L
Ludovic Courtès wrote on 8 Jan 16:02 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
878r4z7tr6.fsf@inria.fr
Hi!

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (3 lines)
>> I was wondering whether we’re now doing better for Bioconductor
>> tarballs. The answer, based on small sample, seems to be “not quite”:

[...]

Toggle quote (6 lines)
> but, now the past reads,
>
> $ for url in https://bioconductor.org/packages/release/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz \
> https://bioconductor.org/packages/3.18/bioc/src/contrib/BiocNeighbors_1.20.0.tar.gz ; \
> do guix download $url ;done

Thanks for investigating & explaining!

I my previous message, I wrote:

Toggle quote (4 lines)
> As for past tarballs, #swh-devel comrades say we could send them a list
> of URLs and they’d create “Save Code Now” requests on our behalf (we
> cannot do it ourselves since the site doesn’t accept plain tarballs.)

Were you able to retrieve some of these? What are the chances of
success?

Toggle quote (5 lines)
> Hence the discussion we had: switch from url-fetch to git-fetch.
> However, after some investigations, it does not seem straightforward:
> The main issue being the almost automatic current updater. See for
> details [2].

[...]

Toggle quote (2 lines)
Indeed, thanks for the link. I agree that long-term moving to
‘git-fetch’ sounds preferable, but there are quite a few obstacles to
overcome.

Ludo’.
L
L
Ludovic Courtès wrote on 8 Jan 16:07 +0100
Re: bug#39885: Bioconductor URI, fallback and time-machine
(name . Ricardo Wurmus)(address . rekado@elephly.net)
87y1cz6ez3.fsf@gnu.org
Hi,

(Replying to a 1.5-year-old message…)

Ricardo Wurmus <rekado@elephly.net> skribis:

Toggle quote (18 lines)
> I have finally taken the time to review this and implement a first draft
> of a change to the bioconductor importer and updater.
>
> There are some limitations:
>
> - we cannot use the updater to go from “url-fetch” to “git-fetch”.
> That’s because “package-update” in (guix upstream) decides whether to
> use package-update/url-fetch or package-update/git-fetch based on the
> *current* package value’s origin fetch procedure. For the switch we
> can hack around this (adding an exception for bioconductor packages),
> but there is no pretty way to do this in a generic fashion that could
> be committed.
>
> Perhaps we could operate on the url included in the <upstream-source>
> instead of looking at the *current* package value. We’re only
> accessing “package” once in the url-fetch case, so maybe we can work
> around this problem.

Alternatively, how about writing a custom one-shot tool to change the
‘source’ field of all the Bioconductor packages to ‘git-fetch’?

It may be easier than adjusting (guix upstream) to cater to this
probably unusual case.

Toggle quote (10 lines)
> - the repositories at https://git.bioconductor.org/package/NAMEdo not
> tag package versions. The only method of organization is branches
> that are named after *Bioconductor releases* (not package releases),
> e.g. RELEASE_3_15. We can only determine the package version by
> reading its DESCRIPTION file or by looking up the version index for
> all Bioconductor packages (we do that already). This means that there
> could be different commits for the same package version in the same
> release branch — so we have to include the commit hash and a revision
> counter in the version string.

OK, sounds acceptable.

Toggle quote (6 lines)
> - the updater doesn’t work on version expressions like (git-version
> "1.12" revision commit). It expects to be able to replace literal
> strings. Because of that my changes let the importer generate a
> string literal such as "1.12-0.cafebab" without a let-bound commit
> string.

Maybe we can build upon Maxime’s patch at

Toggle quote (11 lines)
> - “experiment” or “data” packages are not kept in Git. They only exist
> as volatile tarballs that will be overwritten. Thankfully, they don’t
> change all that often, so they have a good chance of making it into
> our archives.
>
> - the above exception means that we need to litter the importer and
> updater code with extra checks.
>
> With all these notes out of the way I’ll prepare a series of patches
> next.

I don’t think it happened but it’d still be nice. :-)

Thanks,
Ludo’.
R
R
R
Ricardo Wurmus wrote on 10 Jan 13:41 +0100
Re: bug#39885: Bioconductor tarballs are not archived
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
87y1cx5phh.fsf@elephly.net
Ludovic Courtès <ludovic.courtes@inria.fr> writes:

Toggle quote (9 lines)
> I my previous message, I wrote:
>
>> As for past tarballs, #swh-devel comrades say we could send them a list
>> of URLs and they’d create “Save Code Now” requests on our behalf (we
>> cannot do it ourselves since the site doesn’t accept plain tarballs.)
>
> Were you able to retrieve some of these? What are the chances of
> success?

Do we have a list of desired tarballs? I still have an archived
/gnu/store from before we moved the shared cluster installation at the
MDC to different storage. It might contain old tarballs that we no
longer have elsewhere.

--
Ricardo
S
S
Simon Tournier wrote on 10 Jan 16:23 +0100
(name . Ricardo Wurmus)(address . rekado@elephly.net)
CAJ3okZ0vUO=gQ13Oasw1-E0zafWu5-va2JJ1f=e_bhS4y_nbBg@mail.gmail.com
Hi,

On Wed, 10 Jan 2024 at 13:42, Ricardo Wurmus <rekado@elephly.net> wrote:

Toggle quote (2 lines)
> Do we have a list of desired tarballs?

I made one. :-) I will share the manifest once a bit more polish. ;-)

Roughly, until 3.11 -- for earlier, I am still fighting with
time-machine -- and only the versions right before a Bioconductor
upgrade (and sadly not the version after such upgrade that, for many
have disappeared from Bioconductor). If you have them, cool! Well,
for now, I am speaking about all the Bioconductor packages except
annotations for these Guix revisions:

Toggle snippet (6 lines)
git log --format="%P %s" --after=2019-04-30 \
| grep -i -E 'bioconductor' | grep -i -E '(update|upgrade)' \
| cut -f1 -d' ' \
| head -8

And for instance, I have this kind of sources.json.

Toggle snippet (28 lines)
{
"sources": [
{
"type": "url",
"urls": [
"https://bordeaux.guix.gnu.org/file/progeny_1.22.0.tar.gz/sha256/047x6by3xa15gvi3kny5pkqxaq8d2kzcfi55ic5j7a351715l6l7",
"https://ci.guix.gnu.org/file/progeny_1.22.0.tar.gz/sha256/047x6by3xa15gvi3kny5pkqxaq8d2kzcfi55ic5j7a351715l6l7",
"https://bioconductor.org/packages/3.17/bioc/src/contrib/progeny_1.22.0.tar.gz",
"https://tarballs.nixos.org/sha256/047x6by3xa15gvi3kny5pkqxaq8d2kzcfi55ic5j7a351715l6l7"
],
"integrity": "sha256-hxpawgllqCMLi6VEx/4UDWHV8bzF2znifiWoPvwy/RA=",
"outputHashAlgo": "sha256",
"outputHashMode": "flat"
},
{
"type": "url",
"urls": [
"https://bordeaux.guix.gnu.org/file/AWFisher_1.14.0.tar.gz/sha256/1c6rr1z1rhvn8w1kb3nnjlfacfr22vwm1rsa1xqm2hmghs01bq4x",
"https://ci.guix.gnu.org/file/AWFisher_1.14.0.tar.gz/sha256/1c6rr1z1rhvn8w1kb3nnjlfacfr22vwm1rsa1xqm2hmghs01bq4x",
"https://bioconductor.org/packages/3.17/bioc/src/contrib/AWFisher_1.14.0.tar.gz",
"https://tarballs.nixos.org/sha256/1c6rr1z1rhvn8w1kb3nnjlfacfr22vwm1rsa1xqm2hmghs01bq4x"
],
"integrity": "sha256-neAVgIavQlFxD0rnUPkWIjumHJXWjjUDR3bDHH7I2bA=",
"outputHashAlgo": "sha256",
"outputHashMode": "flat"
},

Where the URL for ci.guix is incorrect, I guess.

Cheers,
simon
S
S
Simon Tournier wrote on 11 Jan 17:11 +0100
Re: bug#39885: Bioconductor URI, fallback and time-machine
87h6jj26l2.fsf@gmail.com
Hi,

On Mon, 08 Jan 2024 at 16:34, Ricardo Wurmus <rekado@elephly.net> wrote:

Toggle quote (4 lines)
> The WIP commit is here:
>
> https://git.savannah.gnu.org/cgit/guix.git/commit/?h=wip-r&id=e81a75a7b28c633a658ceeb0a728255674f56c58

IIRC, the main feedback [1] of this approach is:

- the repositories at https://git.bioconductor.org/package/NAMEdo not
tag package versions. The only method of organization is branches
that are named after *Bioconductor releases* (not package releases),
e.g. RELEASE_3_15. We can only determine the package version by
reading its DESCRIPTION file or by looking up the version index for
all Bioconductor packages (we do that already). This means that there
could be different commits for the same package version in the same
release branch — so we have to include the commit hash and a revision
counter in the version string.

Have you tried the wip commit at scale?

Cheers,
simon



1:bug#39885: Bioconductor URI, fallback and time-machine
Ricardo Wurmus <rekado@elephly.net>
Wed, 10 Aug 2022 20:25:00 +0200
id:878rnwuemq.fsf@elephly.net
T
T
Timothy Sample wrote on 19 Jan 16:46 +0100
Re: Bioconductor tarballs are not archived
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)
8734utba2a.fsf@ngyro.com
Hello,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

Toggle quote (8 lines)
> As for past tarballs, #swh-devel comrades say we could send them a list
> of URLs and they’d create “Save Code Now” requests on our behalf (we
> cannot do it ourselves since the site doesn’t accept plain tarballs.)
>
> Any volunteer to write a script that’d generate a list of Bioconductor
> content-addressed URLs (the bordeaux.guix.gnu.org/file ones) for say the
> past couple of years?

Sorry I’m a little late to this party, but I wrote a similar script a
while ago. It creates a “sources.json” file of all the sources that the
PoG database analyzed and found missing in SWH. It only covers what PoG
monitors (which is *almost* everything, but not quite).

$ cd preservation-of-guix

[Wait a long time because my server is sloooow.]

$ guile -L . etc/sources.scm pog.db > missing-sources.json

With some modifications, I used it to generate the attached list of
Bioconductor sources (based off of recent, unpublished PoG data). I’ve
also attached the modifications in case anyone is curious or wants to
make a similar list. I will publish the PoG database soon (today?), so
maybe wait for that before generating any lists.


-- Tim
Attachment: bioconductor-sources.json.gz
Toggle diff (53 lines)
diff --git a/etc/sources.scm b/etc/sources.scm
index 71d157d..515cf00 100644
--- a/etc/sources.scm
+++ b/etc/sources.scm
@@ -1,5 +1,5 @@
;;; Preservation of Guix
-;;; Copyright © 2022 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2022, 2024 Timothy Sample <samplet@ngyro.com>
;;;
;;; This file is part of Preservation of Guix.
;;;
@@ -61,6 +61,7 @@ FROM fods f
WHERE f.algorithm = 'sha256'
AND (fr.reference LIKE '\"%'
OR fr.reference LIKE '(\"%')
+ AND fr.reference LIKE '%bioconductor.org%'
AND NOT fr.is_error
AND f.is_in_swh IS NOT NULL
AND NOT f.is_in_swh")
@@ -85,22 +86,25 @@ Subresource Integrity metadata value."
(define b64 (base64-encode bv))
(string-append "sha256-" b64))
-(define (web-reference-urls reference)
+(define (web-reference-filename reference)
(define uris
(match (call-with-input-string reference read)
((urls ...) (map string->uri urls))
(url (list (string->uri url)))))
- (append-map (lambda (uri)
- (map uri->string
- (maybe-expand-mirrors uri %mirrors)))
- uris))
+ (or (any (lambda (uri)
+ (and (string-suffix? "bioconductor.org" (uri-host uri))
+ (basename (uri-path uri))))
+ uris)
+ (error "Not a 'bioconductor.org' refernce" reference)))
(define (record->url-source rec)
(match-let ((#(digest reference) rec))
- (let ((urls (web-reference-urls reference))
- (integrity (nix-base32-sha256->subresource-integrity digest)))
+ (let* ((filename (web-reference-filename reference))
+ (url (string-append "https://bordeaux.guix.gnu.org/file/"
+ filename "/sha256/" digest))
+ (integrity (nix-base32-sha256->subresource-integrity digest)))
`(("type" . "url")
- ("urls" . ,(list->vector urls))
+ ("urls" . ,(vector url))
("integrity" . ,integrity)))))
(define (lookup-missing-sources db)
L
L
Ludovic Courtès wrote on 23 Jan 10:10 +0100
(name . Timothy Sample)(address . samplet@ngyro.com)
87cytswh2z.fsf@inria.fr
Hi Timothy,

Timothy Sample <samplet@ngyro.com> skribis:

Toggle quote (17 lines)
> Hello,
>
> Ludovic Courtès <ludovic.courtes@inria.fr> writes:
>
>> As for past tarballs, #swh-devel comrades say we could send them a list
>> of URLs and they’d create “Save Code Now” requests on our behalf (we
>> cannot do it ourselves since the site doesn’t accept plain tarballs.)
>>
>> Any volunteer to write a script that’d generate a list of Bioconductor
>> content-addressed URLs (the bordeaux.guix.gnu.org/file ones) for say the
>> past couple of years?
>
> Sorry I’m a little late to this party, but I wrote a similar script a
> while ago. It creates a “sources.json” file of all the sources that the
> PoG database analyzed and found missing in SWH. It only covers what PoG
> monitors (which is *almost* everything, but not quite).

Excellent!

Toggle quote (6 lines)
> With some modifications, I used it to generate the attached list of
> Bioconductor sources (based off of recent, unpublished PoG data). I’ve
> also attached the modifications in case anyone is curious or wants to
> make a similar list. I will publish the PoG database soon (today?), so
> maybe wait for that before generating any lists.

After discussing it on #swh-devel, I filed this issue:


Tim, you were planning to offer a larger list of missing sources
extracted from the PoG database, right?

Thank you!

Ludo’.
S
S
Simon Tournier wrote on 14 Feb 16:23 +0100
Re: bug#39885: Bioconductor tarballs are not archived
87wmr7kr23.fsf@gmail.com
Hi,

On ven., 19 janv. 2024 at 09:46, Timothy Sample <samplet@ngyro.com> wrote:

Toggle quote (8 lines)
> $ cd preservation-of-guix
> $ wget https://ngyro.com/pog-reports/latest/pog.db
>
> [Wait a long time because my server is sloooow.]
>
> $ guile -L . etc/sources.scm pog.db > missing-sources.json

Cool!

Can we consider that this report is now done? Because:

1. SWH supports ExtID and nar hash lookup.

2. Missing origins are currently ingested by SWH.
(via specific sources.json)


Cheers,
simon
T
T
Timothy Sample wrote on 16 Feb 17:14 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87y1bkwfly.fsf@ngyro.com
Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (9 lines)
> Cool!
>
> Can we consider that this report is now done? Because:
>
> 1. SWH supports ExtID and nar hash lookup.
>
> 2. Missing origins are currently ingested by SWH.
> (via specific sources.json)

I think that would be jumping the gun a little bit.

In some sense, the report is only *done* when “stored” hits 100% (or
close to it, with the remainder being stuff we are pretty sure no longer
exists). This won’t happen just because of your second point there.
When the historical “sources.json” is loaded, things will be much, much,
better, sure. Sources will still be missing, though. To me, this is an
invitation to more subtle analysis, like weighing sources by their
“importance” in the package graph. Then there’s still shortcomings with
Disarchive that have to be resolved (which is work best guided by
numbers in the report).

Also, it will always be a good idea to verify that things are working.
Ideally this could be simpler (leveraging ExtID lookup) and continuous.


-- Tim
S
S
Simon Tournier wrote on 19 Feb 17:50 +0100
(name . Timothy Sample)(address . samplet@ngyro.com)
87v86k4cuk.fsf@gmail.com
Hi,

On ven., 16 févr. 2024 at 10:14, Timothy Sample <samplet@ngyro.com> wrote:

Toggle quote (13 lines)
>> Can we consider that this report is now done? Because:
>>
>> 1. SWH supports ExtID and nar hash lookup.
>>
>> 2. Missing origins are currently ingested by SWH.
>> (via specific sources.json)
>
> I think that would be jumping the gun a little bit.
>
> In some sense, the report is only *done* when “stored” hits 100% (or
> close to it, with the remainder being stuff we are pretty sure no longer
> exists). This won’t happen just because of your second point there.

Just to be sure: we are speaking about Bioconductor only, right?


Toggle quote (3 lines)
> When the historical “sources.json” is loaded, things will be much, much,
> better, sure. Sources will still be missing, though.

Yeah, sources will still be missing but I expect that Bioconductor will
be not. The only issue is about “annotation” and maybe “experiment”.
However, here we are hitting the boundary between code and data:
annotation and experiment might be very large and potentially skipped by
SWH and they contain few if no code but plain data.

We can still discuss what to do here; in this already long thread. :-)
Or we can open another thread for this specific case about Bioconductor
annotation and experiment.

Toggle quote (6 lines)
> To me, this is an
> invitation to more subtle analysis, like weighing sources by their
> “importance” in the package graph. Then there’s still shortcomings with
> Disarchive that have to be resolved (which is work best guided by
> numbers in the report).

Yeah. But that seems a large scope than Bioconductor case, no?


Toggle quote (3 lines)
> Also, it will always be a good idea to verify that things are working.
> Ideally this could be simpler (leveraging ExtID lookup) and continuous.

Indeed, checking that all Bioconductor sources can be extracted from
SWH+Disarchive seems the path forward closing this report. :-)

Cheers,
simon
T
T
Timothy Sample wrote on 21 Feb 19:16 +0100
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87sf1lwumv.fsf@ngyro.com
Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

Toggle quote (2 lines)
> Just to be sure: we are speaking about Bioconductor only, right?

I took “this report” to be the PoG report not the bug report. My
mistake! Sorry for the confusion.


-- Tim
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 39885@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 39885
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch