Missing source code

Hello,

As of current ‘master’ (ca. ec0e05ff306c950142c9ead7c712c749617069e7),

the following source code is missing (these are fixed-output

derivations):

/gnu/store/3814l5f74b30yy0j84a4wrv8nl16fx26-eye-1.4.1.tar.gz.drv

/gnu/store/fjk0q2ajk3y18yh5ngif8ir14xckxqa5-weightwatcher-1.12.tar.gz.drv

/gnu/store/syndcxqrzqph12k22y5x6i454jajjw00-skymaker-3.10.5.tar.gz.drv

/gnu/store/ginlb786zg4kqnzvxxfh8qx7wqajhl10-stuff-1.26.0.tar.gz.drv

/gnu/store/hkgpygplymyyaifk6pmvs2cvmkmfhpkq-clusterProfiler_4.0.4.tar.gz.drv

/gnu/store/ghg86w6i14jkrq8nvcazr7pvxz8i82s2-ggtree_3.0.3.tar.gz.drv

/gnu/store/prbkg6vldsh1688fsigr33g4s8qla7rl-propeller-gcc-b4f45a4725e0b6d0af59e594c4e3e35ca4105867-checkout.drv

These two fail to build with a hash mismatch error:

/gnu/store/v1h0djr2daz1ymmsb7043r79llcdamxa-canada1500.zip.drv

/gnu/store/gj7g1hxjja58shki02hcdpbahrxwbcjr-texlive-marginnote-51265-checkout.drv

If you know how to retrieve them, please share!

Ludo’.

Ludovic Courtès schreef op ma 13-09-2021 om 15:16 [+0200]:

The Wayback Machine has some copies of software at https://www.atromatic.net:

Greetings,

Maxime.

Ludo',

I'm afraind I only have:

Ludovic Courtès ???

Let me know where & how to send it.

Kind regards,

T G-R

Tobias Geerinckx-Rice via Bug reports for GNU Guix ???

FTR, I don't think this one's actually down.

Kind regards,

T G-R

Hi,

Maxime Devos <maximedevos@telenet.be> skribis:

Thanks, they’re now all in store on berlin.

It’s kinda ironic that it’s the Wayback Machine that’s saving us here…

I wonder if there’s a way to construct generic web.archive.org URLs that

we could use as a fallback in (guix build download)?

Thanks,

Ludo’.

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

Oops, indeed, this one’s available.

Thanks,

Ludo’.

severity 50568 important

quit

Hi,

On Tue, 14 Sept 2021 at 10:40, Ludovic Courtès <ludo@gnu.org> wrote:

Do we do the usual trick to save them on SWH?

That would help, I guess. :-)

Cheers,

simon

On Tue, Sep 14, 2021 at 10:38:55AM +0200, Ludovic Courtès wrote:

AFAIK Just replace the date in

On Mon, Sep 13, 2021 at 03:49:05PM +0200, Maxime Devos wrote:

by the current time.

Regards,

Florian

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

I think their contents are already saved on berlin, just not the

tarballs themselves.

For example:

and:

Hmm actually maybe they vanished before they got a chance to be

archived?

Ludo’.

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

Oh nice, that seems to work:

How ’bout the attached patch? Works for me.

Thanks,

Ludo’.

Hi,

On Thu, 16 Sept 2021 at 21:28, Ludovic Courtès <ludo@gnu.org> wrote:

I mean, download the tarball from Berlin, then turn locally the source

to a Git source, archive it on SWH.

As it had been done for guile-dbi for instance.

Probably. One thing that I have on my TODO is to write a sources.json

with all the source Berlin has. Or as a start add the URL of Berlin

as "mirror".

Cheers,

simon

On Tue, Sep 14, 2021 at 10:38:55AM +0200, Ludovic Courtès wrote:

On Thu, Sep 16, 2021 at 09:45:26PM +0200, Ludovic Courtès wrote:

Thank you for implementing. I guix pull with the patch applied, add the IP

from

guix environment --ad-hoc bind:utils -- dig web.archive.org

to /etc/hosts, set my DNS server to a non-existing 10.0.0.50, disable

IPv6, then “sudo herd stop nscd”, re-connect Network Manager (ping

gnu.org fails now) and LC_ALL=en_US.utf8 guix install hello. It

fails. Did I misunderstand the patch’s purpose or test in a wrong

way? This is the output:

florian@florianacer ~$ LC_ALL=en_US.utf8 guix install hello

The following package will be installed:

hello 2.10

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 0.0%guix substitute: warning: ci.guix.gnu.org: host not found: Temporary failure in name resolution

substitute:

substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 0.0%guix substitute: warning: bordeaux.guix.gnu.org: host not found: Temporary failure in name resolution

substitute:

The following derivations will be built:

/gnu/store/4hsqwl19n4m2scipkzy0rsc04n30vhbk-profile.drv

/gnu/store/xr30pd6zw8lp0xz3kq2d7gm3c8pmh674-hello-2.10.drv

/gnu/store/wy24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 0.0%

substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 0.0%

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 0.0%

substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 0.0%

applying 5 grafts for /gnu/store/rsizfxkag8gkbccr5d4g9xdacgwh8wya-at-spi2-core-2.34.0.drv ...

building /gnu/store/wy24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv...

\builder for `/gnu/store/wy24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv' failed to produce output path `/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz'

build of /gnu/store/wy24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv failed

View build log at '/var/log/guix/drvs/wy/24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv.bz2'.

applying 4 grafts for /gnu/store/8jj80f4p5bp9s09zawxmli13chrdx2ri-at-spi2-atk-2.34.1.drv ...

cannot build derivation `/gnu/store/xr30pd6zw8lp0xz3kq2d7gm3c8pmh674-hello-2.10.drv': 1 dependencies couldn't be built

cannot build derivation `/gnu/store/4hsqwl19n4m2scipkzy0rsc04n30vhbk-profile.drv': 1 dependencies couldn't be built

guix install: error: build of `/gnu/store/4hsqwl19n4m2scipkzy0rsc04n30vhbk-profile.drv' failed

florian@florianacer ~$ bzcat /var/log/guix/drvs/wy/24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv.bz2

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz

From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...

In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz

From ftp://ftp.cs.tu-berlin.de/pub/gnu/hello/hello-2.10.tar.gz...

In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz

From ftp://ftp.funet.fi/pub/mirrors/ftp.gnu.org/gnu/hello/hello-2.10.tar.gz...

In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz

From http://ftp.gnu.org/pub/gnu/hello/hello-2.10.tar.gz...

In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz

From https://ci.guix.gnu.org/file/hello-2.10.tar.gz/sha256/0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i...

In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz

From https://tarballs.nixos.org/sha256/0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i...

In procedure getaddrinfo: Temporary failure in name resolution

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz

From https://archive.softwareheritage.org/api/1/content/sha256:31e066137a962676e89f69d1b65382de95a7ef7d914b8cb956f41ea72e0f516b/raw/...

In procedure getaddrinfo: Temporary failure in name resolution

Trying to use Disarchive to assemble /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz...

In procedure getaddrinfo: Temporary failure in name resolution

could not find its Disarchive specification

failed to download "/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz" from "mirror://gnu/hello/hello-2.10.tar.gz"

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

The thing is that this code is used by the internal ‘guix

perform-download’ command, invoked by ‘guix-daemon’ when performing a

source download. Thus, since you were still running the old daemon, you

were using code lacking the Internet Archive fallback.

To test it, you can first start the new daemon, along these lines:

sudo herd stop guix-daemon

sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild

Another option is to call ‘url-fetch’ manually at the REPL.

Thanks for testing!

Ludo’.

On Mon, Sep 20, 2021 at 09:25:27AM +0200, Ludovic Courtès wrote:

Thank you for enlightening me; I had not expected guix-daemon.

Success! After

./bootstrap

./configure --localstatedir=/var

make

GUIX_ALLOW_ME_TO_USE_PRIVATE_COMMIT=y make update-guix-package

and git commit -a, guix pull, reconfigure and reboot, your

web.archive.org patch downloads from web.archive.org with the steps

from before:

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

/var/log/guix/drvs/wy/24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv.bz2

says after various failures in name resolution it got downloaded from

web.archive.org. I enjoy GNU Hello now. Thank you for making

reproducibility easier.

No, going back one generation and guix gc -D’ing the

/gnu/store/*hello-2.10* after adding web.archive.org to /etc/hosts

again, I for some reason get a

florian@florianacer ~$ bzcat /var/log/guix/drvs/wy/24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv.bz2

error: failed to run download program '/home/florian/src/guix/scripts/guix': Keine Berechtigung

(Keine Berechtigung is German for Permission denied) but it does not

matter, since the reconfigure and reboot works when and only when

adding web.archive.org to /etc/hosts.

Actually I will later take another look at this, because the manual

says ./pre-inst-env guix-daemon should work.

This I didn’t try.

Regards,

Florian

On Tue, Sep 21, 2021 at 12:28:50PM +0200, pelzflorian (Florian Pelz) wrote:

There is no regression. Without the web.archive.org patch,

`guix install hello' via

sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild

always shows the same error, regardless of whether I misdirected DNS

lookups or not. The patch is not at fault. Though I don’t know why.

sudo -E strace -f ./pre-inst-env guix-daemon --build-users-group=guixbuild

shows

[pid 1854] execve("/home/florian/src/guix/scripts/guix", ["guix", "perform-download", "/gnu/store/wy24jyd0mm2jlgpzsxxli"..., "/gnu/store/hbdalsf5lpf01x4dcknwx"...], 0x1d8f9a0 /* 80 vars */) = -1 EACCES (Keine Berechtigung)

https://pubs.opengroup.org/onlinepubs/9699919799/ says:

I run:

florian@florianacer ~$ ls -ahl /home/florian/src/guix/scripts/guix

-r-xr-xr-x 1 florian users 2,0K 21. Sep 07:30 /home/florian/src/guix/scripts/guix

florian@florianacer ~$ ls -ahl /home/florian/src/guix | grep scripts

drwxr-xr-x 2 florian users 4,0K 21. Sep 07:30 scripts/

florian@florianacer ~$ lsattr /home/florian/src/guix/scripts/guix

--------------e----- /home/florian/src/guix/scripts/guix

florian@florianacer ~ [env]$ cat test-execv.c

#include <unistd.h>

int main ()

{

const char *const argv[] =

{

"guix", "perform-download",

"/gnu/store/wy24jyd0mm2jlgpzsxxlip03df065g35-hello-2.10.tar.gz.drv",

"/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz.lock",

NULL

};

execv ("/home/florian/src/guix/scripts/guix", (char *const *) argv);

return 0;

}

florian@florianacer ~ [env]$ gcc test-execv.c -o test-execv

florian@florianacer ~ [env]$ LC_ALL=en_US.utf8 ~/test-execv

Starting download of /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz.lock

From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...

following redirection to `https://ftp.wayne.edu/gnu/hello/hello-2.10.tar.gz'...

In procedure open-file: Read-only file system: "/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz.lock"

[and more expected failures …]

Whatever. It is not important.

Regards,

Florian

Hi!

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

That’s a heavy test, but great that it works as intended. I’ll push it

soonish.

That is weird indeed.

Thanks for testing!

Ludo’.

Hi!

Ludovic Courtès <ludovic.courtes@inria.fr> skribis:

We now have ‘etc/source-manifest.scm’, meaning one can run¹:

guix weather -m etc/source-manifest.scm

This manifest is now built at ci.guix.gnu.org:

https://ci.guix.gnu.org/jobset/source

… which gives us a good overview of currently missing/mismatching

source:

https://ci.guix.gnu.org/eval/27221?status=failed

So far 27 failures out of 16,315, roughly 0.2%.

If you have copies of the missing/modified files that show up there,

please share.

Ludo’.

¹ https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00174.html

Ludovic Courtès <ludo@gnu.org> skribis:

Hi,

I have the following ones in the store:

- cl-gserver-1.8.0-checkout

- gnu-standards-2020-11-25-checkout

- golden-utils-0.0.0-3.fe1898f-checkout

- xorg-server_1.20.10-3.diff.gz

The attached archive contains the nars made with:

The signing key of the machine is:

Hi Guillaume,

Guillaume Le Vaillant <glv@posteo.net> skribis:

Thanks! I’ve been able to address these in several ways:

commit f7f036e7c369515ad678cc08dd5bc9e254804d38 (do take a look at

the comment in that commit…)

CVS doesn’t go through the firewall at berlin, so I manually added it

to the store (not great).

commit ef93b238f64c1a27270cae78772b60d63088a026

manually added to the store; problem is that this file has disappeared

upstream, AFAICS, so it would be best to change this package

definition, possibly adding copy of the diff.gz file to the repo?

Ludo’.

Ludovic Courtès <ludo@gnu.org> skribis:

We’re left with the following missing bioconductor.org tarballs:

AnnotationFuncs_1.40.0.tar.gz

AneuFinderData_1.18.0.tar.gz

BiocCaseStudies_1.52.0.tar.gz

chromstaRData_1.16.0.tar.gz

genomationData_1.22.0.tar.gz

and a hash mismatch for:

texlive-marginnote-51265-checkout

Zimoun, Ricardo: do you know how to address the bioconductor.org issue?

Ricardo: do you still have a copy of texlive-marginnote-51265-checkout

with the hash as it currently appears on master? We’ll probably end up

just updating the hash, but it’d be nice to check the differences.

Once we’re done with those, we’ll have 100% source coverage on ci.guix.

At that point, we can start building Disarchive metadata.

TIA!

Ludo’.

Ludovic Courtès schreef op do 07-10-2021 om 15:53 [+0200]:

Running "guix build --source r-annotationfuncs r-aneufinderdata r-bioccasestudies r-chromstardata r-genomationdata"

succeeds on my computer. They are downloaded from bayfront.guix.gnu.org.

Greetings,

Maxime

Maxime Devos <maximedevos@telenet.be> skribis:

Indeed, fixed!

Currently bayfront.guix.gnu.org doesn’t run ‘guix publish’, so it

doesn’t support the /file URL scheme, which means I had to do this

manually. We’ll fix it!

Ludo’.

Hi Ludo,

On Thu, 7 Oct 2021 at 15:53, Ludovic Courtès <ludo@gnu.org> wrote:

I pick this one. Cuirass says Succeeded with the log:

Well, I do not know what is this 141.80.167.131 IP address. Something

related to Bayfront, right?

From the evaluation you mention above, all these had succeeded. Could

you confirm it is fine?

BTW, for the interested reader, I would like to point this bug#39885

[1] about Bioconductor, URI, fallback and time-machine which appears

to me related. In addition to the two options proposed, another one

not listed but discussed elsewhere is to switch Bioconductor packages

from url-fetch to git-fecth.

1: http://issues.guix.gnu.org/issue/39885

All the best,

simoon

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

It is: I fetched copies from bayfront, IIRC.

Very relevant! How can we avoid those brittle bioconductor.org URLs?

Thanks,

Ludo’.

Hi,

On Tue, 12 Oct 2021 at 11:50, Ludovic Courtès <ludo@gnu.org> wrote:

By addressing the mentioned bug. ;-) Do not hold your breath, it is low

in my TODO list. The next Bioconductor release v3.14 is planned for

Wednesday October 27. Therefore, let make it happens for v3.15. :-)

Cheers,

simon