Bioconductor URI, fallback and time-machine

Open

Details

4 participants

Ludovic Courtès
Maxime Devos
Ricardo Wurmus
zimoun

Owner: unassigned

Submitted by: zimoun

Severity: normal

zimoun wrote on 3 Mar 2020 16:59

Recipients:

Message-ID:CAJ3okZ3dFunYgafRH6=9LsLKLf6OrZBpXqUMxZAjEhaiL93ARA@mail.gmail.com

Dear,

Currently, the URI scheme (see 'bioconductor-uri' in

guix/build-system/r.scm) is:

https://bioconductor.org/packages/release/data/type-url-part /src/contrib/ upstream-name - version .tar.gz

which leads to 2 issues:

1. when Bioconductor updates their release, some package versions are

updated too, and so, the upstream return 404.

2. for this reason 1., the "guix time-machine" is broken for all the

Bioconductor packages, at least if Berlin or SWH does not have a

substitute; which is not expected for 'annotation' packages.

However, the Bioconductor archive still serves the old release, i.e.,

https://bioconductor.org/packages/3.x/data/type-url-part /src/contrib/ upstream-name - version .tar.gz

The ways to fix the both issues are:

a) Add the Bioconductor release (known at packaging time) to all the

packages; provide as argument to 'bioconductor-uri'.

b) Add more URLs to fallback.

As discussed on IRC, Tobias seems more inclined with the option a) and

I am more in favour of option b.

Attached, a quick patch showing the option b).

Please also consider #36805 which was never merged or closed.

http://issues.guix.gnu.org/issue/36805

All the best,

simon

From 87e73e02202fe5e342d68f1fb17efdd4425737cd Mon Sep 17 00:00:00 2001

* guix/build-system/r.scm (bioconductor-uri): Extend the fallback list.

---

guix/build-system/r.scm | 21 ++++++++++++---------

1 file changed, 12 insertions(+), 9 deletions(-)

Toggle diff (34 lines)diff --git a/guix/build-system/r.scm b/guix/build-system/r.scm
index 2d328764b0..8638e1b888 100644
--- a/guix/build-system/r.scm
+++ b/guix/build-system/r.scm
@@ -54,15 +54,18 @@ release corresponding to NAME and VERSION."
                          ('annotation "/data/annotation")
                          ('experiment "/data/experiment")
                          (_ "/bioc"))))
-    (list (string-append "https://bioconductor.org/packages/release"
-                         type-url-part
-                         "/src/contrib/"
-                         name "_" version ".tar.gz")
-          ;; TODO: use %bioconductor-version from (guix import cran)
-          (string-append "https://bioconductor.org/packages/3.10"
-                         type-url-part
-                         "/src/contrib/Archive/"
-                         name "_" version ".tar.gz"))))
+    (append (list (string-append "https://bioconductor.org/packages/release"
+                                 type-url-part
+                                 "/src/contrib/"
+                                 name "_" version ".tar.gz"))
+            (map (lambda (release)
+                   (string-append "https://bioconductor.org/packages/"
+                                  release
+                                  type-url-part
+                                  "/src/contrib/"
+                                  name "_" version ".tar.gz"))
+                 (list (@@ (guix import cran) %bioconductor-version)
+                       "3.9" "3.8" "3.7")))))
 
 (define %r-build-system-modules
   ;; Build-side modules imported by default.
-- 
2.25.0

Ricardo Wurmus wrote on 23 Mar 2020 22:20

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87ftdylqdn.fsf@elephly.net

zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (20 lines)>  1. when Bioconductor updates their release, some package versions are
> updated too, and so, the upstream return 404.
>  2. for this reason 1., the "guix time-machine" is broken for all the
> Bioconductor packages, at least if Berlin or SWH does not have a
> substitute; which is not expected for 'annotation' packages.
>
> However, the Bioconductor archive still serves the old release, i.e.,
>
> https://bioconductor.org/packages/3.x/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
>
>
> The ways to fix the both issues are:
>
>  a) Add the Bioconductor release (known at packaging time) to all the
> packages; provide as argument to 'bioconductor-uri'.
>  b) Add more URLs to fallback.
>
> As discussed on IRC, Tobias seems more inclined with the option a) and
> I am more in favour of option b.

I think option a) is more explicit, which is probably what we generally

want to future-proof the time-machine. Fallbacks are okay in the case

of the CRAN URL where it’s not necessarily clear when a package tarball

moves from the release location to the archive.

In the case of Bioconductor URLs it seems that we can afford to be a bit

more accurate.

Ricardo

zimoun wrote on 22 May 2020 01:29

Recipients:(name . Ricardo Wurmus)(address . rekado@elephly.net)

Message-ID:CAJ3okZ1Ttjh+iG3qU1a_PcK_m5-64+KLAGKYT7b5Cum7fGgkKA@mail.gmail.com

Dear Ricardo,

On Mon, 23 Mar 2020 at 22:21, Ricardo Wurmus <rekado@elephly.net> wrote:

Toggle quote (15 lines)> >  a) Add the Bioconductor release (known at packaging time) to all the
> > packages; provide as argument to 'bioconductor-uri'.
> >  b) Add more URLs to fallback.
> >
> > As discussed on IRC, Tobias seems more inclined with the option a) and
> > I am more in favour of option b.
>
> I think option a) is more explicit, which is probably what we generally
> want to future-proof the time-machine.  Fallbacks are okay in the case
> of the CRAN URL where it’s not necessarily clear when a package tarball
> moves from the release location to the archive.
>
> In the case of Bioconductor URLs it seems that we can afford to be a bit
> more accurate.

We are going for option a) which means rename all the URLs, right?

Because it is a lot, I suggest to first address the bug#36805, i.e.,

provide as an argument the BioConductor version to 'bioconductor-uri'

and applies this policy to all the new packages or any update of them.

Moreover, I have suggested to reorganise bioconductor.scm,

bioinformatics.scm, cran.scm, etc. and I have not dedicated enough

time to this boring task. But because I am working remotely

(semi-lockdown), I plan to work on it next week and so this change of

URLs could be part of the big reorganisation.

What do you think?

[1] http://issues.guix.gnu.org/issue/36805

All the best,

simon

zimoun wrote on 24 Jun 2020 13:07

Re: bug#39885: Bioconductor URI, fallback and time-machine

Recipients:

Message-ID:CAJ3okZ1Xwd-2WArzNus3xE_KOayDdXPp+ku1SxYBon4Zm0qQQg@mail.gmail.com

Dear,

The time-machine is broken for some BioConductor packages.. For an

example, consider the package "r-genomegraphs" which has been removed

from the BioConductor in 3.11 release.

(Well, now the issue is mitigated because ci.guix.gnu.org serves a lot

of upstream substitutes but ci.guix.gnu.org could be down. Other

said, we should use the upstream resources where they are available.)

Concretely, there are 2 issues:

a) What to do for the removed packages? For 3.11, the list is there

[1]. Do we keep them in gnu/packages/bioconductor.scm but then

'bioconductor-uri' needs some tweaks? Or do we transfer them to the

channel guix-past (for example)?

b) The fallback URI in guix/build-system/r.scm(bioconductor-uri)

added by commit c586f427b4831b9b492e5b900b2226e898b8fcfa is not

correct, if I do not misread:

Toggle snippet (4 lines)

"https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/GenomeGraphs_1.46.0.tar.gz"

404 "Not Found"

The correct seems to be (without Archive):

https://bioconductor.org/packages/3.10/bioc/src/contrib/GenomeGraphs_1.46.0.tar.gz

All the best,

simon

1: https://bioconductor.org/news/bioc_3_11_release/#deprecated-and-defunct-packages

Ludovic Courtès wrote on 28 Jun 2020 22:14

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87lfk7gd7d.fsf@gnu.org

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (11 lines)>  b) The fallback URI in guix/build-system/r.scm(bioconductor-uri)
> added by commit c586f427b4831b9b492e5b900b2226e898b8fcfa is not
> correct, if I do not misread:
>
> "https://bioconductor.org/packages/3.10/bioc/src/contrib/Archive/GenomeGraphs_1.46.0.tar.gz"
> 404 "Not Found"
>
> The correct seems to be (without Archive):
>
> https://bioconductor.org/packages/3.10/bioc/src/contrib/GenomeGraphs_1.46.0.tar.gz

Could you provide a patch for this?

Thanks,

Ludo’.

zimoun wrote on 29 Jun 2020 19:36

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:CAJ3okZ36KrALgMq69zdkDsHcfMv09Lk=DPBmzW-4SePZPRAvnA@mail.gmail.com

Hi Ludo,

On Sun, 28 Jun 2020 at 22:14, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (2 lines)

> Could you provide a patch for this?

About the url, for sure, see attached.

But it does not address the root of the problem. Well, I will try to

find a slot and propose something.

All the best,

simon

From c1c963a3b86e306a20c14626127e54d21843c22c Mon Sep 17 00:00:00 2001

* guix/build-system/r.scm (bioconductor-uri): Fix archive URL.

---

guix/build-system/r.scm | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (17 lines)diff --git a/guix/build-system/r.scm b/guix/build-system/r.scm
index c8ec9abd0d..5ef982d66a 100644
--- a/guix/build-system/r.scm
+++ b/guix/build-system/r.scm
@@ -61,7 +61,7 @@ release corresponding to NAME and VERSION."
           ;; TODO: use %bioconductor-version from (guix import cran)
           (string-append "https://bioconductor.org/packages/3.11"
                          type-url-part
-                         "/src/contrib/Archive/"
+                         "/src/contrib/"
                          name "_" version ".tar.gz"))))
 
 (define %r-build-system-modules

base-commit: 6ebf300959a58fd1eda875205c75d21137862285
-- 
2.26.2

Ludovic Courtès wrote on 29 Jun 2020 22:42

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87r1txa9ix.fsf@gnu.org

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (7 lines)

> From c1c963a3b86e306a20c14626127e54d21843c22c Mon Sep 17 00:00:00 2001

> From: zimoun <zimon.toutoune@gmail.com>

> Date: Mon, 29 Jun 2020 19:18:20 +0200

> Subject: [PATCH] build-system/r: bioconductor-uri: Fix archive URL.

> * guix/build-system/r.scm (bioconductor-uri): Fix archive URL.

Applied, thanks!

I let the rest of you discuss the other issues. :-)

Ludo’.

zimoun wrote on 19 Nov 2020 15:22

Recipients:(address . 39885@debbugs.gnu.org)

Message-ID:87r1ope8a4.fsf@gmail.com

Hi,

Some explanations of the issue are provided here:

http://issues.guix.gnu.org/issue/39885

Since we are currently updating to 3.12, maybe it is the occasion to fix

the issue. See option a) below.

On Tue, 03 Mar 2020 at 16:59, zimoun <zimon.toutoune@gmail.com> wrote:

Toggle quote (14 lines)> Currently, the URI scheme (see 'bioconductor-uri' in
> guix/build-system/r.scm) is:
>
>  https://bioconductor.org/packages/release/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
>
> which leads to 2 issues:
>
>  1. when Bioconductor updates their release, some package versions are
> updated too, and so, the upstream return 404.
>
>  2. for this reason 1., the "guix time-machine" is broken for all the
> Bioconductor packages, at least if Berlin or SWH does not have a
> substitute; which is not expected for 'annotation' packages.

An example of this issue is for example:

Toggle snippet (24 lines)

$ guix time-machine --commit=aee183e -- import cran -a bioconductor CATALYST -r

Updating channel 'guix' from Git repository at 'https://git.savannah.gnu.org/git/guix.git'...

Starting download of /tmp/guix-file.Nxajqh

From https://bioconductor.org/packages/release/bioc/src/contrib/CATALYST_1.12.2.tar.gz...

download failed "https://bioconductor.org/packages/release/bioc/src/contrib/CATALYST_1.12.2.tar.gz" 404 "Not Found"

failed to download "/tmp/guix-file.Nxajqh" from "https://bioconductor.org/packages/release/bioc/src/contrib/CATALYST_1.12.2.tar.gz"

error: failed to retrieve package information from "https://cran.r-project.org/web/packages/CATALYST/DESCRIPTION": 404 ("Not Found")

Backtrace:

4 (primitive-load "/home/simon/.cache/guix/inferiors/vznc…")

In guix/ui.scm:

2117:12 3 (run-guix-command _ . _)

In guix/scripts/import.scm:

120:11 2 (guix-import . _)

In srfi/srfi-1.scm:

586:17 1 (map1 (#f))

In guix/import/utils.scm:

258:2 0 (package->definition _)

guix/import/utils.scm:258:2: In procedure package->definition:

Throw to key `match-error' with args `("match" "no matching pattern" #f)'.

Aside the ugly backtrace which is tracked by #44115, the main issue is

because Bioconductor updated to 3.12 and Guix is still at 3.11.

Concretely, the issue is that ’release’ in the URL:

https://bioconductor.org/packages/release/bioc/src/contrib/CATALYST_1.12.2.tar.gz

now refers to 3.12 (because Bioconductor update) and Guix still think it

is 3.11 (because Guix has not yet updated; work-in-progress). And

CATALYST in 3.12 is at version 1.14.0 against 1.12.2 for 3.11.

Therefore, the conflict and the error.

It means that while:

(define %bioconductor-version "3.11")

is not updated to 3.12, all the Bioconductor packages are broken; in the

meaning not buildable from source.

Toggle quote (9 lines)

> a) Add the Bioconductor release (known at packaging time) to all the

> packages; provide as argument to 'bioconductor-uri'.

> b) Add more URLs to fallback, e.g.:

> https://bioconductor.org/packages/release/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz

> https://bioconductor.org/packages/3.11/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz

> Attached, a quick patch showing the option b).

Then each time we update Bioconductor, we add an URL to the list.

Toggle quote (3 lines)

> As discussed on IRC, Tobias seems more inclined with the option a) and

> I am more in favour of option b.

Tobias and Ricardo are in favor for a) (see this thread). Which means a

lot of work IMHO, i.e., add 3.11 as arguments and then 3.12 to all the

Bioconductor packages and fix the importer, IIUC; while b) means do

nothing except merge the proposed patch (possibly re-worked).

Just to note that only the task to group in bioconductor.scm all the

Bioconductor packages scattered here and there is still not done, I

think option a) is not doable by hand – I do not volunteer! :-) Else,

any suggestion to script the task instead?

Since I am more in favor of b), I am less motivated to fix the a). ;-)

But I am motivated to fix the issue at hand. :-)

Other option c) is to switch all the Bioconductor to git-fetch instead

of url-fetch. I have not checked yet how could be the transition.

Toggle quote (3 lines)

> Please also consider #36805 which was never merged or closed.

> http://issues.guix.gnu.org/issue/36805

This patch could help for option a).

WDYT?

All the best,

simon

zimoun wrote on 22 Nov 2021 20:48

Recipients:(address . 39885@debbugs.gnu.org)

Message-ID:87wnl0q8ei.fsf@gmail.com

Hi,

On Tue, 03 Mar 2020 at 16:59, zimoun <zimon.toutoune@gmail.com> wrote:

Toggle quote (17 lines)> Currently, the URI scheme (see 'bioconductor-uri' in
> guix/build-system/r.scm) is:
>
>  https://bioconductor.org/packages/release/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz
>
> which leads to 2 issues:
>
>  1. when Bioconductor updates their release, some package versions are
> updated too, and so, the upstream return 404.
>  2. for this reason 1., the "guix time-machine" is broken for all the
> Bioconductor packages, at least if Berlin or SWH does not have a
> substitute; which is not expected for 'annotation' packages.
>
> However, the Bioconductor archive still serves the old release, i.e.,
>
> https://bioconductor.org/packages/3.x/data/<type-url-part>/src/contrib/<upstream-name>-<version>.tar.gz

It is still the case and for concrete breakage, see [1]. I will not

detail but each time Guix lags behind Bioconductor new release, it is

broken. For sure, Guix upgrades more or less quickly. Each time

Bioconductor remove a package, it is broken. Well, because a lot

of care about R packages, the forward breakages happen barely. :-) But

backward breakages are not negligible, IMHO.

Well, this URL choice is not The Right Thing and somehow broken by design.

1: https://issues.guix.gnu.org/39885#7

Toggle quote (11 lines)> The ways to fix the both issues are:
>
>  a) Add the Bioconductor release (known at packaging time) to all the
> packages; provide as argument to 'bioconductor-uri'.
>  b) Add more URLs to fallback.
>
> As discussed on IRC, Tobias seems more inclined with the option a) and
> I am more in favour of option b.
>
> Attached, a quick patch showing the option b).

We are now 1.5 years after. And we did nothing; well we did other

things instead. ;-). Now, I have an strong opinion that option a) is

not doable: I speak using my janitor moves of Bioconductor packages.

Instead, something along the proposed patch below half-fixes the issue

now. We just have to append the releases and let the fallback mechanism

takes care. It reduces the maintenance burden, IMHO.

For sure, it is not perfect but it appears to me a pragmatical fix

waiting something better.

This better is unknown (at least from me :-)). On one hand Disarchive

would improve the situation for tarballs… but some work remains (check

that SWH ingestion and rebuild is bullet-proof). On the other hand,

Bioconductor uses Git, for instance:

git clone https://git.bioconductor.org/packages/CATALYST

https://bioconductor.org/packages/release/bioc/html/CATALYST.html

And Bioconductor uses ’origin/RELEASE_3.14’ as Git tag. Based on this,

it would avoid the eternal inplace-change fixes.

For instance, the package tximeta [2], recently updated by Ricardo.

Well, from their Bioconductor Git repo,

git clone https://git.bioconductor.org/packages/tximeta

it is not clear that the current version is at 1.12.3. And it is not

clear either if they tagged origin/RELEASE_3_14 at 1.12.0 and did

something ugly to then get 1.12.3. Anyway, switch from url-fetch to

git-fetch is an option. However, it is as option a) and I am not

convinced it is doable with the resource at hand.

2: https://bioconductor.org/packages/3.14/bioc/html/tximeta.html

What could a plan to have a bullet-proof “guix time-machine” for

Bioconductor?

Cheers,

simon

Toggle quote (42 lines)> From 87e73e02202fe5e342d68f1fb17efdd4425737cd Mon Sep 17 00:00:00 2001
> From: zimoun <zimon.toutoune@gmail.com>
> Date: Tue, 3 Mar 2020 16:53:39 +0100
> Subject: [PATCH] build-system: r: Use Bioconductor old releases to fallback.
>
> * guix/build-system/r.scm (bioconductor-uri): Extend the fallback list.
> ---
>  guix/build-system/r.scm | 21 ++++++++++++---------
>  1 file changed, 12 insertions(+), 9 deletions(-)
>
> diff --git a/guix/build-system/r.scm b/guix/build-system/r.scm
> index 2d328764b0..8638e1b888 100644
> --- a/guix/build-system/r.scm
> +++ b/guix/build-system/r.scm
> @@ -54,15 +54,18 @@ release corresponding to NAME and VERSION."
>                           ('annotation "/data/annotation")
>                           ('experiment "/data/experiment")
>                           (_ "/bioc"))))
> -    (list (string-append "https://bioconductor.org/packages/release"
> -                         type-url-part
> -                         "/src/contrib/"
> -                         name "_" version ".tar.gz")
> -          ;; TODO: use %bioconductor-version from (guix import cran)
> -          (string-append "https://bioconductor.org/packages/3.10"
> -                         type-url-part
> -                         "/src/contrib/Archive/"
> -                         name "_" version ".tar.gz"))))
> +    (append (list (string-append "https://bioconductor.org/packages/release"
> +                                 type-url-part
> +                                 "/src/contrib/"
> +                                 name "_" version ".tar.gz"))
> +            (map (lambda (release)
> +                   (string-append "https://bioconductor.org/packages/"
> +                                  release
> +                                  type-url-part
> +                                  "/src/contrib/"
> +                                  name "_" version ".tar.gz"))
> +                 (list (@@ (guix import cran) %bioconductor-version)
> +                       "3.9" "3.8" "3.7")))))
>
>  (define %r-build-system-modules
>    ;; Build-side modules imported by default.

zimoun wrote on 18 Jul 2022 18:03

Recipients:(address . 39885@debbugs.gnu.org)

Message-ID:87lesqmmrr.fsf@gmail.com

Hi,

Since 2020, I provided several examples of breakage with bug#39885 [1].

Here another one:

Toggle snippet (40 lines)

$ guix time-machine --commit=77e2de365497bf4c8b81cbd78624f78293490485 \

-- build r-biocneighbors -S

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%

substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%

The following derivation will be built:

/gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv

building /gnu/store/q9ggmh5a9bzmnr49p10x1w9sv6pzjarv-BiocNeighbors_1.4.1.tar.gz.drv...