Content-addressed mirror is not used upon invalid hash

Jan Nieuwenhuizen wrote on 1 Oct 2017 12:16

v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail

Recipients:(address . bug-guix@gnu.org)

Message-ID:877ewf18d4.fsf@gnu.org

Hi!
As reported by laertus on irc[0]: guix pull on 0.13 without substitutes fails
      guix pull
    Starting download of /tmp/guix-file.3r6cH0    From https://git.savannah.gnu.org/cgit/guix.git/snapshot/master.tar.gz...     ….tar.gz                                   5.7MiB/s 00:02 | 13.6MiB transferred    unpacking '/gnu/store/sginfwnrcfqn1far31gmzlaffd8xlxyy-guix-latest.tar.gz'...
    Starting download of /gnu/store/c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz    From https://github.com/libgit2/libgit2/archive/v0.25.1.tar.gz...    following redirection to `https://codeload.github.com/libgit2/libgit2/tar.gz/v0.25.1'...     v0.25.1                                     6.1MiB/s 00:01 | 4.1MiB transferred    output path `/gnu/store/c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz' should have sha256 hash `1cdwcw38frc1wf28x5ppddazv9hywc718j92f3xa3ybzzycyds3s', instead has `0ywcxw1mwd56c8qc14hbx31bf198gxck3nja3laxyglv7l57qp26'    cannot build derivation `/gnu/store/z1ky970mnamnbairnpyxxb72qnc485zq-libgit2-0.25.1.drv': 1 dependencies couldn't be built    cannot build derivation `/gnu/store/rl7ms8rmbywvydy4qf656g1sdfxafb7r-guile-git-0.0-2.06f9fc3.drv': 1 dependencies couldn't be built    guix pull: error: build failed: build of `/gnu/store/rl7ms8rmbywvydy4qf656g1sdfxafb7r-guile-git-0.0-2.06f9fc3.drv' failed
because the libgit2-0.25.1 content hash does not check out.
I verified this on version-0.13.  The same goes for 0.26.0 on master
    $ guix build -S libgit2 --no-substitutes    The following derivations will be built:       /gnu/store/5szrmzmfgxk6pylk5fh9bk8apj4x8axf-libgit2-0.26.0.tar.xz.drv       /gnu/store/mgh4yjxkxfyqmc7c61vwq4vs8v837602-libgit2-0.26.0.tar.gz.drv    @ build-started /gnu/store/mgh4yjxkxfyqmc7c61vwq4vs8v837602-libgit2-0.26.0.tar.gz.drv - x86_64-linux /var/log/guix/drvs/mg//h4yjxkxfyqmc7c61vwq4vs8v837602-libgit2-0.26.0.tar.gz.drv.bz2
    Starting download of /gnu/store/53lj4z9cavl7n27r89zjnvyd8fk854kj-libgit2-0.26.0.tar.gz    From https://github.com/libgit2/libgit2/archive/v0.26.0.tar.gz...    following redirection to `https://codeload.github.com/libgit2/libgit2/tar.gz/v0.26.0'...     v0.26.0  4.5MiB                    3.1MiB/s 00:01 [####################] 100.0%    sha256 hash mismatch for output path `/gnu/store/53lj4z9cavl7n27r89zjnvyd8fk854kj-libgit2-0.26.0.tar.gz'      expected: 1fdk9yhwvl1w1z71ykzcvgh4nsf8scxcbclz5anh98zpplmhmisa      actual:   1b3figbhp5l83vd37vq6j2narrq4yl9pfw6mw0px0dzb1hz3jqka    @ build-failed /gnu/store/mgh4yjxkxfyqmc7c61vwq4vs8v837602-libgit2-0.26.0.tar.gz.drv - 1 sha256 hash mismatch for output path `/gnu/store/53lj4z9cavl7n27r89zjnvyd8fk854kj-libgit2-0.26.0.tar.gz'      expected: 1fdk9yhwvl1w1z71ykzcvgh4nsf8scxcbclz5anh98zpplmhmisa      actual:   1b3figbhp5l83vd37vq6j2narrq4yl9pfw6mw0px0dzb1hz3jqka    cannot build derivation `/gnu/store/5szrmzmfgxk6pylk5fh9bk8apj4x8axf-libgit2-0.26.0.tar.xz.drv': 1 dependencies couldn't be built    guix build: error: build failed: build of `/gnu/store/5szrmzmfgxk6pylk5fh9bk8apj4x8axf-libgit2-0.26.0.tar.xz.drv' failed
I found no apparent difference in the content
    -r--r--r-- 1 janneke janneke  4252130 Oct  1 09:08 c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz    -rw-r--r-- 1 janneke janneke  4252139 Oct  1 09:09 NEW-c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz    -rw-r--r-- 1 janneke janneke 16363520 Oct  1 09:14 c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar    -rw-r--r-- 1 janneke janneke 16363520 Oct  1 09:14 NEW-c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar
but there's this difference between the tar balls...
    12:13:57 janneke@dundal:~/src/guix-0.13     $ cmp -l c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar NEW-c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar    13122049   0 157    13122050   0 162    13122051   0 151    13122052   0 147    13122053   0 151    13122054   0 156    13122055   0  57    13122490  57   0    13122491 157   0    13122492 162   0    13122493 151   0    13122494 147   0    13122495 151   0    13122496 156   0    13270529   0 157    13270530   0 162    13270531   0 151    13270532   0 147    13270533   0 151    13270534   0 156    13270535   0  57    13270972  57   0    13270973 157   0    13270974 162   0    13270975 151   0    13270976 147   0    13270977 151   0    13270978 156   0    13294081   0 157    13294082   0 162    13294083   0 151    13294084   0 147    13294085   0 151    13294086   0 156    13294087   0  57    13294519  57   0    13294520 157   0    13294521 162   0    13294522 151   0    13294523 147   0    13294524 151   0    13294525 156   0
janneke
[0] https://gnunet.org/bot/log/guix/2017-10-01#T1517584
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com

Jan Nieuwenhuizen wrote on 1 Oct 2017 21:20

Recipients:(address . 28659@debbugs.gnu.org)

Message-ID:87wp4e8yk5.fsf@gnu.org

Jan Nieuwenhuizen writes:
The changing of the libgit-0.26.0 checksum was already reported about 3weeks ago (github seems to only show relative dates)
https://github.com/libgit2/libgit2/issues/4343
and the bug is still open. It seems to be a github thing. As Iunderstand it, currently our options are to update the hash and pray itwon't happen again or host libgit2 tarballs ourselves.
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com

Leo Famulari wrote on 1 Oct 2017 22:42

Recipients:(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)(address . 28659@debbugs.gnu.org)

Message-ID:20171001204237.GA11804@jasmine.lan

On Sun, Oct 01, 2017 at 09:20:42PM +0200, Jan Nieuwenhuizen wrote:

Toggle quote (11 lines)> Jan Nieuwenhuizen writes:> > The changing of the libgit-0.26.0 checksum was already reported about 3> weeks ago (github seems to only show relative dates)> >     https://github.com/libgit2/libgit2/issues/4343> > and the bug is still open.  It seems to be a github thing.  As I> understand it, currently our options are to update the hash and pray it> won't happen again or host libgit2 tarballs ourselves.

I contacted GitHub about this issue a few weeks ago and they said that:
1) They do not guarantee bit-reproducibility of the snapshots theygenerate automatically for each release tag, and they wish that peoplewould not rely on them as we do. However, since people *are* relying onthem, they are discussing this issue internally.2) This is the relevant code change:https://git.kernel.org/pub/scm/git/git.git/commit/?id=22f0dcd9634a818a0c83f23ea1a48f2d620c0546
In the meantime, we can add this to the list of reasons thatreproducibility is difficult in the long term.
I don't have any solutions in mind besides keeping substitutes availablefor as long as possible and, for users, using substitutes. We might alsopetition upstream projects to offer a "real" release tarball.

ng0 wrote on 1 Oct 2017 23:05

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:20171001210527.ym24ubylu7mh5huv@abyayala

Leo Famulari transcribed 2.3K bytes:

Toggle quote (28 lines)> On Sun, Oct 01, 2017 at 09:20:42PM +0200, Jan Nieuwenhuizen wrote:> > Jan Nieuwenhuizen writes:> > > > The changing of the libgit-0.26.0 checksum was already reported about 3> > weeks ago (github seems to only show relative dates)> > > >     https://github.com/libgit2/libgit2/issues/4343> > > > and the bug is still open.  It seems to be a github thing.  As I> > understand it, currently our options are to update the hash and pray it> > won't happen again or host libgit2 tarballs ourselves.> > I contacted GitHub about this issue a few weeks ago and they said that:> > 1) They do not guarantee bit-reproducibility of the snapshots they> generate automatically for each release tag, and they wish that people> would not rely on them as we do. However, since people *are* relying on> them, they are discussing this issue internally.> 2) This is the relevant code change:> https://git.kernel.org/pub/scm/git/git.git/commit/?id=22f0dcd9634a818a0c83f23ea1a48f2d620c0546> > In the meantime, we can add this to the list of reasons that> reproducibility is difficult in the long term.> > I don't have any solutions in mind besides keeping substitutes available> for as long as possible and, for users, using substitutes. We might also> petition upstream projects to offer a "real" release tarball.

Given that we depend on this for our core functionality,can't we just keep this on our ftp directory at gnu.orgas a fall-back source in a list?
-- ng0GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588GnuPG: https://krosos.org/dist/keys/https://www.infotropique.orghttps://krosos.org

Ludovic Courtès wrote on 2 Oct 2017 16:57

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:87vajxoavx.fsf@gnu.org

Hi!
Leo Famulari <leo@famulari.name> skribis:

Toggle quote (7 lines)

> I contacted GitHub about this issue a few weeks ago and they said that:>> 1) They do not guarantee bit-reproducibility of the snapshots they> generate automatically for each release tag, and they wish that people> would not rely on them as we do. However, since people *are* relying on> them, they are discussing this issue internally.

Oh?! Then we’re in trouble.
Perhaps we should start using ‘git-fetch’ more, with Software Heritageas a fallback content-addressed mirror? Though again the difficulty isthat SWH uses Git’s method to hash directory contents, so we’d end uphaving to provide both a Nix hash and a Git hash in ‘origin’. :-/

Toggle quote (3 lines)

> In the meantime, we can add this to the list of reasons that> reproducibility is difficult in the long term.

Heh.
Ludo’.

Ludovic Courtès wrote on 2 Oct 2017 17:09

Recipients:(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)(address . 28659@debbugs.gnu.org)

Message-ID:87o9ppoabw.fsf@gnu.org

Hello,
Jan Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (2 lines)

> As reported by laertus on irc[0]: guix pull on 0.13 without substitutes fails

I just checked and we do have substitutes, but I understand it doesn’thelp here.

Toggle quote (13 lines)>       guix pull>>     Starting download of /tmp/guix-file.3r6cH0>     From https://git.savannah.gnu.org/cgit/guix.git/snapshot/master.tar.gz...>      ….tar.gz                                   5.7MiB/s 00:02 | 13.6MiB transferred>     unpacking '/gnu/store/sginfwnrcfqn1far31gmzlaffd8xlxyy-guix-latest.tar.gz'...>>     Starting download of /gnu/store/c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz>     From https://github.com/libgit2/libgit2/archive/v0.25.1.tar.gz...>     following redirection to `https://codeload.github.com/libgit2/libgit2/tar.gz/v0.25.1'...>      v0.25.1                                     6.1MiB/s 00:01 | 4.1MiB transferred>     output path `/gnu/store/c3npgqn9ag2ypi9bda1g779wwwlcqqrf-libgit2-0.25.1.tar.gz' should have sha256 hash `1cdwcw38frc1wf28x5ppddazv9hywc718j92f3xa3ybzzycyds3s', instead has `0ywcxw1mwd56c8qc14hbx31bf198gxck3nja3laxyglv7l57qp26'

What’s sad here is that we do have the right tarball at:
https://mirror.hydra.gnu.org/file/libgit2-0.25.1.tar.gz/sha256/1cdwcw38frc1wf28x5ppddazv9hywc718j92f3xa3ybzzycyds3s
The problem is that the hash check is performed by guix-daemon itself,not by “guix perform-download”. So when guix-daemon diagnoses a hashmismatch, it’s too late and we cannot try again and use thecontent-addressed mirror.
A crude but helpful fix would be to have perform-download compute thehash by itself and act accordingly. It’s crude because that means thatwe’d be computing the hash twice: once in ‘guix perform-download’ and asecond time in guix-daemon. For archives below ~20 MiB it’s probably OKthough.
Thoughts?
In the future, with the daemon written in Guile, it’s one area where wecould achieve better integration and coordination among the variouspieces.
Ludo’.

Ludovic Courtès wrote on 2 Oct 2017 17:16

control message for bug #28659

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87fub1oa0j.fsf@gnu.org

retitle 28659 Content-addressed mirror is not used upon invalid hash

Ludovic Courtès wrote on 2 Oct 2017 17:16

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87efqloa0f.fsf@gnu.org

severity 28659 important

Jan Nieuwenhuizen wrote on 2 Oct 2017 19:05

Re: bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 28659@debbugs.gnu.org)

Message-ID:87infx8oqr.fsf@gnu.org

Ludovic Courtès writes:

Toggle quote (4 lines)

> What’s sad here is that we do have the right tarball at:>> https://mirror.hydra.gnu.org/file/libgit2-0.25.1.tar.gz/sha256/1cdwcw38frc1wf28x5ppddazv9hywc718j92f3xa3ybzzycyds3s

Sad indeed!

Toggle quote (5 lines)

> The problem is that the hash check is performed by guix-daemon itself,> not by “guix perform-download”. So when guix-daemon diagnoses a hash> mismatch, it’s too late and we cannot try again and use the> content-addressed mirror.

Why don't we try our content-addressed mirror first?

Toggle quote (8 lines)

> A crude but helpful fix would be to have perform-download compute the> hash by itself and act accordingly. It’s crude because that means that> we’d be computing the hash twice: once in ‘guix perform-download’ and a> second time in guix-daemon. For archives below ~20 MiB it’s probably OK> though.>> Thoughts?

We may want more guix hackers' viewpoints here, I don't feel veryqualified...As this would be a temporary workaround only until we have

Toggle quote (4 lines)

> In the future, with the daemon written in Guile, it’s one area where we> could achieve better integration and coordination among the various> pieces.

...it might be fine?
Do we want/need to bring out a new release for this, e.g. 0.13.1, oreven 0.14? I'm not sure how bad it is that --no-substitutes does notwork. I think working on guix pull to not compile everything locallymay have priority?
janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com

Leo Famulari wrote on 2 Oct 2017 20:19

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:20171002181929.GA10773@jasmine.lan

On Mon, Oct 02, 2017 at 04:57:38PM +0200, Ludovic Courtès wrote:

Toggle quote (13 lines)> Hi!> > Leo Famulari <leo@famulari.name> skribis:> > > I contacted GitHub about this issue a few weeks ago and they said that:> >> > 1) They do not guarantee bit-reproducibility of the snapshots they> > generate automatically for each release tag, and they wish that people> > would not rely on them as we do. However, since people *are* relying on> > them, they are discussing this issue internally.> > Oh?!  Then we’re in trouble.

I wonder, are there really that many affected packages? My sense is thatmost GitHub-hosted projects offer their own release tarballs in additionto the problematic auto-generated snapshots, and we tend to prefer theupstream-provided tarballs in this case.
We'd need to survey our package sources to know what sort of reaction ismost appropriate.
In general, we should try to make Guix as resilient as possible tounstable upstream sources, since the problem is not limited to GitHub.

Toggle quote (5 lines)

> Perhaps we should start using ‘git-fetch’ more, with Software Heritage> as a fallback content-addressed mirror? Though again the difficulty is> that SWH uses Git’s method to hash directory contents, so we’d end up> having to provide both a Nix hash and a Git hash in ‘origin’. :-/

And the Git hashes will change from SHA1 to SHA256 sooner or later, andSHA1 hashes will become less reliable as CPUs get faster (collisionattacks), compounding the problem...

Leo Famulari wrote on 2 Oct 2017 20:22

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:20171002182208.GB10773@jasmine.lan

On Mon, Oct 02, 2017 at 05:09:39PM +0200, Ludovic Courtès wrote:

Toggle quote (4 lines)

> What’s sad here is that we do have the right tarball at:> > https://mirror.hydra.gnu.org/file/libgit2-0.25.1.tar.gz/sha256/1cdwcw38frc1wf28x5ppddazv9hywc718j92f3xa3ybzzycyds3s

It seems to me that there are several reasons someone may choose not touse substitutes. Some of those reasons (reproducibility and securityconcerns) are obviated for fixed-output derivations like upstreamsources, and I think it would be fine to still use substitutes for thesederivations.
But the motivations of privacy, self-sufficiency, etc are not addressedby that idea.

Ludovic Courtès wrote on 2 Oct 2017 22:00

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:878tgt721q.fsf@gnu.org

Leo Famulari <leo@famulari.name> skribis:

Toggle quote (5 lines)

> On Mon, Oct 02, 2017 at 05:09:39PM +0200, Ludovic Courtès wrote:>> What’s sad here is that we do have the right tarball at:>> >> https://mirror.hydra.gnu.org/file/libgit2-0.25.1.tar.gz/sha256/1cdwcw38frc1wf28x5ppddazv9hywc718j92f3xa3ybzzycyds3s

Just to be clear: this URL is not that of a substitute, but that of acontent-addressed file (corresponding to the output of a fixed-outputderivation.)

Toggle quote (9 lines)

> It seems to me that there are several reasons someone may choose not to> use substitutes. Some of those reasons (reproducibility and security> concerns) are obviated for fixed-output derivations like upstream> sources, and I think it would be fine to still use substitutes for these> derivations.>> But the motivations of privacy, self-sufficiency, etc are not addressed> by that idea.

Right. Jan suggested checking the content-addressed mirrors *before*the real upstream address. That would address the problem of upstreamsources modified in-place, but at the cost of privacy/self-sufficiencyas you note. (Though it’s not really making “privacy” any worse in thiscase: it’s gnu.org vs. github.com.)
Perhaps we should make content-addressed mirrors configurable in a waythat’s orthogonal to derivations, something similar in spirit to--substitute-urls? The difficulty is that content-addressed mirrors arenot just URLs; see (guix download).
Thoughts?
Ludo’.

Jan Nieuwenhuizen wrote on 2 Oct 2017 22:22

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:87a8198fli.fsf@gnu.org

Ludovic Courtès writes:

Toggle quote (6 lines)

> Right. Jan suggested checking the content-addressed mirrors *before*> the real upstream address. That would address the problem of upstream> sources modified in-place, but at the cost of privacy/self-sufficiency> as you note. (Though it’s not really making “privacy” any worse in this> case: it’s gnu.org vs. github.com.)

Yes, that may not preferrable in general without override.

Toggle quote (5 lines)

> Perhaps we should make content-addressed mirrors configurable in a way> that’s orthogonal to derivations, something similar in spirit to> --substitute-urls? The difficulty is that content-addressed mirrors are> not just URLs; see (guix download).

Hmm.  I'm not sure what problem we are solving.  Should we only do thisfor github(-like) tarballs?  Do we see this problem with other sources,should we prevent it?  Possibly github will never do something like thisagain.  Or we could banish github/gitlab(?) auto-generated tarballs andgo for git checkouts+commits?
janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com

Leo Famulari wrote on 2 Oct 2017 22:29

Recipients:(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)

Message-ID:20171002202907.GA23960@jasmine.lan

On Mon, Oct 02, 2017 at 10:22:33PM +0200, Jan Nieuwenhuizen wrote:

Toggle quote (6 lines)

> Hmm. I'm not sure what problem we are solving. Should we only do this> for github(-like) tarballs? Do we see this problem with other sources,> should we prevent it? Possibly github will never do something like this> again. Or we could banish github/gitlab(?) auto-generated tarballs and> go for git checkouts+commits?

Files referenced by URL (location-addressing vs content-addressing) havebeen changed in place by a variety of hosters and upstream projectssince I've started paying attention to these issues. I don't think weneed to do anything special regarding GitHub.

Maxim Cournoyer wrote on 3 Oct 2017 00:47

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:87infx2mmt.fsf@gmail.com

Leo Famulari <leo@famulari.name> writes:

Toggle quote (16 lines)> On Mon, Oct 02, 2017 at 04:57:38PM +0200, Ludovic Courtès wrote:>> Hi!>> >> Leo Famulari <leo@famulari.name> skribis:>> >> > I contacted GitHub about this issue a few weeks ago and they said that:>> >>> > 1) They do not guarantee bit-reproducibility of the snapshots they>> > generate automatically for each release tag, and they wish that people>> > would not rely on them as we do. However, since people *are* relying on>> > them, they are discussing this issue internally.>> >> Oh?!  Then we’re in trouble.>> I wonder, are there really that many affected packages?

There's a list here:https://github.com/Homebrew/homebrew-core/issues/18044,compiled by oneof the homebrew project's maintainers.
Maxim

Ludovic Courtès wrote on 3 Oct 2017 14:30

Recipients:(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)

Message-ID:871smk9zx9.fsf@gnu.org

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (2 lines)

> Ludovic Courtès writes:

[...]

Toggle quote (11 lines)>> Perhaps we should make content-addressed mirrors configurable in a way>> that’s orthogonal to derivations, something similar in spirit to>> --substitute-urls?  The difficulty is that content-addressed mirrors are>> not just URLs; see (guix download).>> Hmm.  I'm not sure what problem we are solving.  Should we only do this> for github(-like) tarballs?  Do we see this problem with other sources,> should we prevent it?  Possibly github will never do something like this> again.  Or we could banish github/gitlab(?) auto-generated tarballs and> go for git checkouts+commits?

Content-addressed mirrors help with disappearing and modified tarballsin general; it’s not just GitHub.
Occasionally we see that problem with tarballs coming from elsewhere:404 is quite frequent, and in-place modification happens from time totime (even on ftp.gnu.org…).
Ludo’.

Ludovic Courtès wrote on 3 Oct 2017 14:31

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)

Message-ID:87wp4c8lbc.fsf@gnu.org

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (22 lines)> Leo Famulari <leo@famulari.name> writes:>>> On Mon, Oct 02, 2017 at 04:57:38PM +0200, Ludovic Courtès wrote:>>> Hi!>>> >>> Leo Famulari <leo@famulari.name> skribis:>>> >>> > I contacted GitHub about this issue a few weeks ago and they said that:>>> >>>> > 1) They do not guarantee bit-reproducibility of the snapshots they>>> > generate automatically for each release tag, and they wish that people>>> > would not rely on them as we do. However, since people *are* relying on>>> > them, they are discussing this issue internally.>>> >>> Oh?!  Then we’re in trouble.>>>> I wonder, are there really that many affected packages?>> There's a list here:> https://github.com/Homebrew/homebrew-core/issues/18044, compiled by one> of the homebrew project's maintainers.

Interesting. Thanks for the link!
Ludo’.

Leo Famulari wrote on 3 Oct 2017 16:24

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)

Message-ID:20171003142449.GB23431@jasmine.lan

On Mon, Oct 02, 2017 at 06:47:06PM -0400, Maxim Cournoyer wrote:

Toggle quote (7 lines)

> Leo Famulari <leo@famulari.name> writes:> > I wonder, are there really that many affected packages?> > There's a list here:> https://github.com/Homebrew/homebrew-core/issues/18044, compiled by one> of the homebrew project's maintainers.

I meant, how many Guix packages use the auto-generated GitHub snapshots?
I believe the tell-tale sign is that the download link will have thelink text 'Source code', as for this release:
https://github.com/libgit2/libgit2/releases/tag/v0.26.0

Maxim Cournoyer wrote on 4 Oct 2017 06:22

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:874lrfee45.fsf@gmail.com

Leo Famulari <leo@famulari.name> writes:

Toggle quote (15 lines)> On Mon, Oct 02, 2017 at 06:47:06PM -0400, Maxim Cournoyer wrote:>> Leo Famulari <leo@famulari.name> writes:>> > I wonder, are there really that many affected packages?>> >> There's a list here:>> https://github.com/Homebrew/homebrew-core/issues/18044, compiled by one>> of the homebrew project's maintainers.>> I meant, how many Guix packages use the auto-generated GitHub snapshots?>> I believe the tell-tale sign is that the download link will have the> link text 'Source code', as for this release:>> https://github.com/libgit2/libgit2/releases/tag/v0.26.0

The following script:

;;; A script to find packages possibly affected by GitHub;;; infrastructure update that caused minor changes in the;;; automatically generated tarballs.
(use-modules (ice-9 match)	     (gnu packages)	     (guix download)	     (guix packages))
(define (problematic-uri? uri)
  (define (contains-github-archive? uri)    (string-match "github.com/.*/archive/" uri))
  ;; URI can be a string or a list of string.  (match uri    ((uri1 uri2 ...)			;match list of strings     (filter contains-github-archive? uri))    (uri1				;match string     (contains-github-archive? uri1))))
(define (problematic-github-package? package)  (let ((source (package-source package)))    (and (origin? source)	 (eq? (origin-method source) url-fetch)	 (problematic-uri? (origin-uri source)))))
(define (problematic-github-packages)  "List of all the potentially problematic GitHub packages."  (fold-packages (lambda (p r)		   (if (problematic-github-package? p)		       (cons p r)		       r))		 '()))(define (main)  "Find and print the names of the potentially problematic GitHub packages."  (let ((packages (problematic-github-packages)))    (format #t "Number of potentially problematic GitHub packages:~a~%"	    (length packages))    (for-each (lambda (p)		(format #t "~a~%" (package-name p)))	      packages)))
;;; Run the program.(main)

outputs that there could be up to 1011 affected packages.
The scripts checks for a url-fetch uri of the form"github.com/.*/archive/", which seems to be the one used for thedynamically generated archives.
Here are the first 10 lines of the output:

Toggle snippet (13 lines)Number of potentially problematic GitHub packages:1011fdupescbatticonsedsedcpulimitautojumpsudothermaldprogressdstat[...]

I've checked the first few with for example:

Toggle snippet (3 lines)

guix build --source --no-substitutes sedsed

and they were OK though.
Maxim

Leo Famulari wrote on 4 Oct 2017 18:54

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)

Message-ID:20171004165413.GA4596@jasmine.lan

On Wed, Oct 04, 2017 at 12:22:34AM -0400, Maxim Cournoyer wrote:

Toggle quote (10 lines)> Here are the first 10 lines of the output:> --8<---------------cut here---------------start------------->8---> Number of potentially problematic GitHub packages:1011> fdupes> cbatticon> sedsed> cpulimit> autojump> sudo

I think the script is buggy; sudo's source is not downloaded from GitHubas far as I can tell.

Maxim Cournoyer wrote on 5 Oct 2017 01:53

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:87r2uih3lx.fsf@gmail.com

Leo Famulari <leo@famulari.name> writes:

Toggle quote (14 lines)> On Wed, Oct 04, 2017 at 12:22:34AM -0400, Maxim Cournoyer wrote:>> Here are the first 10 lines of the output:>> --8<---------------cut here---------------start------------->8--->> Number of potentially problematic GitHub packages:1011>> fdupes>> cbatticon>> sedsed>> cpulimit>> autojump>> sudo>> I think the script is buggy; sudo's source is not downloaded from GitHub> as far as I can tell.

Good catch! I was assuming empty lists were falsy, but that's not thecase! I've ensured purely boolean predicates now and it gets the listdown to 650.
Here's the corrected script:

;;; A script to find packages possibly affected by GitHub;;; infrastructure update that caused minor changes in the;;; automatically generated tarballs.
(use-modules (ice-9 match)	     (gnu packages)	     (guix download)	     (guix packages))
(define (problematic-uri? uri)
  (define (contains-github-archive? uri)    (regexp-match? (string-match "github.com/.*/archive/" uri)))
  ;; URI can be a string or a list of string.  (match uri    ((uri1 uri2 ...)			;match list of strings     (not (null? (filter contains-github-archive? uri))))    (uri1				;match string     (contains-github-archive? uri1))))
(define (problematic-github-package? package)  (let ((source (package-source package)))    (and (origin? source)	 (eq? (origin-method source) url-fetch)	 (problematic-uri? (origin-uri source)))))
(define (problematic-github-packages)  "List of all the potentially problematic GitHub packages."  (fold-packages (lambda (p r)		   (if (problematic-github-package? p)		       (cons p r)		       r))		 '()))(define (main)  "Find and print the names of the potentially problematic GitHub packages."  (let ((packages (problematic-github-packages)))    (format #t "Number of potentially problematic GitHub packages: ~a~%"	    (length packages))    (for-each (lambda (p)		(format #t "~a~%" (package-name p)))	      packages)))
;;; Run the program.(main)

And sample output:

Toggle snippet (21 lines)Number of potentially problematic GitHub packages: 650fdupescbatticoncpulimitthefuckthermaldneofetchautojumpprogressnnn[...]wxwidgetsxclipxcapesxhkdmaimsloptinyxml2xlsx2csv

Maxim

Maxim Cournoyer wrote on 5 Oct 2017 06:52

Recipients:(name . Leo Famulari)(address . leo@famulari.name)(address . 28659@debbugs.gnu.org)

Message-ID:87h8vegpqu.fsf@gmail.com

I've modified the script to sort the packages it prints:

Toggle snippet (8 lines)

- (for-each (lambda (p)- (format #t "~a~%" (package-name p)))- packages)))+ (for-each (lambda (name)+ (format #t "~a~%" name))+ (sort (map package-name packages) string<?))))

and compared it to the list here: https://github.com/Homebrew/homebrew-core/issues/18044
If we can trust the Homebrew list to be extensive, it seems we gotlucky; there's only one affected package that we share which isyaml-cpp. Here's how it fails on our side:

Toggle snippet (20 lines)guix build -S --no-substitutes yaml-cppThe following derivation will be built:   /gnu/store/mlap8jmadirnbii6sppb6vj9x56s8azw-yaml-cpp-0.5.3.tar.gz.drv@ build-started /gnu/store/mlap8jmadirnbii6sppb6vj9x56s8azw-yaml-cpp-0.5.3.tar.gz.drv - x86_64-linux /var/log/guix/drvs/ml//ap8jmadirnbii6sppb6vj9x56s8azw-yaml-cpp-0.5.3.tar.gz.drv.bz2
Starting download of /gnu/store/qwflwafrzjbr2b7dy4nv18nxykghhmnk-yaml-cpp-0.5.3.tar.gzFrom https://github.com/jbeder/yaml-cpp/archive/yaml-cpp-0.5.3.tar.gz...following redirection to `https://codeload.github.com/jbeder/yaml-cpp/tar.gz/yaml-cpp-0.5.3'... ...p-0.5.3                                  1.7MiB/s 00:01 | 1.9MiB transferredsha256 hash mismatch for output path `/gnu/store/qwflwafrzjbr2b7dy4nv18nxykghhmnk-yaml-cpp-0.5.3.tar.gz'  expected: 1vk6pjh0f5k6jwk2sszb9z5169whmiha9ainbdpa1arxlkq7v3b6  actual:   1ck7jk0wjfigrf4cgcjqsir4yp1s6vamhhxhpsgfvs46pgm5pk6y@ build-failed /gnu/store/mlap8jmadirnbii6sppb6vj9x56s8azw-yaml-cpp-0.5.3.tar.gz.drv - 1 sha256 hash mismatch for output path `/gnu/store/qwflwafrzjbr2b7dy4nv18nxykghhmnk-yaml-cpp-0.5.3.tar.gz'  expected: 1vk6pjh0f5k6jwk2sszb9z5169whmiha9ainbdpa1arxlkq7v3b6  actual:   1ck7jk0wjfigrf4cgcjqsir4yp1s6vamhhxhpsgfvs46pgm5pk6yguix build: error: build failed: build of`/gnu/store/mlap8jmadirnbii6sppb6vj9x56s8azw-yaml-cpp-0.5.3.tar.gz.drv'failed

Maxim

Jan Nieuwenhuizen wrote on 5 Oct 2017 08:08

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)

Message-ID:87infu6sah.fsf@gnu.org

Maxim Cournoyer writes:

Toggle quote (4 lines)

> If we can trust the Homebrew list to be extensive, it seems we got> lucky; there's only one affected package that we share which is> yaml-cpp. Here's how it fails on our side:

I needed to also use (ice-9 regex) and then I found these to fail
    antlr3    csound    erlang    font-google-material-design-icons    fritzing    libgit2    lxqt-common    ogre    plexus-interpolation    red-eclipse    yaml-cpp
out of 646 packages it's not many but it includes our core dependencylibgit2 which breaks guix pull --no-substitutes; that's hardly beinglucky?
janneke
-- Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.orgFreelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com

Leo Famulari wrote on 20 Oct 2017 23:17

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:20171020211700.GA32355@jasmine.lan

On Mon, Oct 02, 2017 at 10:00:33PM +0200, Ludovic Courtès wrote:

Toggle quote (6 lines)

Yeah, I don't personally think there is a privacy issue with fetchingsources from our mirrors at gnu.org, or other domains we control.

Toggle quote (7 lines)

I do think we should make it so that users don't suffer from unreliableupstream sources when we know the sources are available on our servers(or the Nix mirror), even with --no-substitutes.

Ludovic Courtès wrote on 28 Nov 2017 14:30

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:87d1421qek.fsf@gnu.org

Leo Famulari <leo@famulari.name> skribis:

Toggle quote (21 lines)> On Mon, Oct 02, 2017 at 10:00:33PM +0200, Ludovic Courtès wrote:>> Right.  Jan suggested checking the content-addressed mirrors *before*>> the real upstream address.  That would address the problem of upstream>> sources modified in-place, but at the cost of privacy/self-sufficiency>> as you note.  (Though it’s not really making “privacy” any worse in this>> case: it’s gnu.org vs. github.com.)>> Yeah, I don't personally think there is a privacy issue with fetching> sources from our mirrors at gnu.org, or other domains we control.>>> Perhaps we should make content-addressed mirrors configurable in a way>> that’s orthogonal to derivations, something similar in spirit to>> --substitute-urls?  The difficulty is that content-addressed mirrors are>> not just URLs; see (guix download).>>>> Thoughts?>> I do think we should make it so that users don't suffer from unreliable> upstream sources when we know the sources are available on our servers> (or the Nix mirror), even with --no-substitutes.

The more I think about it, the more I’m inclined to simply movecontent-addressed mirrors to the front of the list. This means thatusers, in practice, would be fetching all the source frommirror.hydra.gnu.org.
The main issue is making it configurable. Currently thecontent-addressed mirror configuration for regular files in (guixdownload) looks like this:

Toggle snippet (19 lines)(define %content-addressed-mirrors  ;; List of content-addressed mirrors.  Each mirror is represented as a  ;; procedure that takes a file name, an algorithm (symbol) and a hash  ;; (bytevector), and returns a URL or #f.  ;; Note: Avoid 'https' to mitigate <http://bugs.gnu.org/22774>.  ;; TODO: Add more.  '(list (lambda (file algo hash)           ;; Files served by 'guix publish' are accessible under a single           ;; hash algorithm.           (string-append "http://mirror.hydra.gnu.org/file/"                          file "/" (symbol->string algo) "/"                          (bytevector->nix-base32-string hash)))         (lambda (file algo hash)           ;; 'tarballs.nixos.org' supports several algorithms.           (string-append "http://tarballs.nixos.org/"                          (symbol->string algo) "/"                          (bytevector->nix-base32-string hash)))))

That for VCS checkouts in (guix build download-nar) looks like this:

Toggle snippet (13 lines)(define (urls-for-item item)  "Return the fallback nar URL for ITEM--e.g.,\"/gnu/store/cabbag3…-foo-1.2-checkout\"."  ;; Here we hard-code nar URLs without checking narinfos.  That's probably OK  ;; though.  ;; TODO: Use HTTPS?  The downside is the extra dependency.  (let ((bases '("http://mirror.hydra.gnu.org/guix"                 "http://berlin.guixsd.org"))        (item  (basename item)))    (append (map (cut string-append <> "/nar/gzip/" item) bases)            (map (cut string-append <> "/nar/" item) bases))))

The latter could be expressed by a command-line flag. In fact it’s thesame as --substitute-urls.
(Time passes…)
Thinking more about it, why not simply always enable substitutes forfixed-output derivations, like this:

Toggle diff (16 lines)diff --git a/nix/libstore/build.cc b/nix/libstore/build.ccindex d68e8b2bc..03a8f5080 100644--- a/nix/libstore/build.cc+++ b/nix/libstore/build.cc@@ -1034,8 +1034,10 @@ void DerivationGoal::haveDerivation()      /* We are first going to try to create the invalid output paths        through substitutes.  If that doesn't work, we'll build-       them. */-    if (settings.useSubstitutes && substitutesAllowed(drv))+       them.  Always enable substitutes for fixed-output derivations to+       protect against disappearing files and in-place modifications on+       upstream sites.  */+    if ((fixedOutput || settings.useSubstitutes) && substitutesAllowed(drv))         foreach (PathSet::iterator, i, invalidOutputs)             addWaitee(worker.makeSubstitutionGoal(*i, buildMode == bmRepair));

This solves all our problems and makes download-nar.scm useless.
As an added bonus, it provides a improves the UI since we now alwayssee:

Toggle snippet (4 lines)

0.1 MB will be downloaded: /gnu/store/plx9848n6waj6zghn3d54ybx8ihcn23k-guile-git-0.0-4.951a32c-checkout

… instead of:

Toggle snippet (4 lines)

The following derivation will be built: /gnu/store/y86rlb6pdm35im7q02y6479ca84zwylz-guile-git-000.0-4.951a32c-checkout.drv

The downside is that it still requires one to authorize the server’skey, although it’s in theory unnecessary since it’s content addressed.I’m not sure how to solve that because ‘guix substitute’ doesn’t knowthat it’s substituting a fixed-output derivation. I suppose we’d needto modify the “protocol” between guix-daemon and ‘guix substitute’.
Thoughts?
Ludo’.

Ludovic Courtès wrote on 14 Dec 2017 17:53

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:874lot9rou.fsf@gnu.org

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (20 lines)> Thinking more about it, why not simply always enable substitutes for> fixed-output derivations, like this:>> diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc> index d68e8b2bc..03a8f5080 100644> --- a/nix/libstore/build.cc> +++ b/nix/libstore/build.cc> @@ -1034,8 +1034,10 @@ void DerivationGoal::haveDerivation()>  >      /* We are first going to try to create the invalid output paths>         through substitutes.  If that doesn't work, we'll build> -       them. */> -    if (settings.useSubstitutes && substitutesAllowed(drv))> +       them.  Always enable substitutes for fixed-output derivations to> +       protect against disappearing files and in-place modifications on> +       upstream sites.  */> +    if ((fixedOutput || settings.useSubstitutes) && substitutesAllowed(drv))>          foreach (PathSet::iterator, i, invalidOutputs)>              addWaitee(worker.makeSubstitutionGoal(*i, buildMode == bmRepair));

[...]

Toggle quote (6 lines)

> The downside is that it still requires one to authorize the server’s> key, although it’s in theory unnecessary since it’s content addressed.> I’m not sure how to solve that because ‘guix substitute’ doesn’t know> that it’s substituting a fixed-output derivation. I suppose we’d need> to modify the “protocol” between guix-daemon and ‘guix substitute’.

I looked at how to address this by having ‘guix substitute’automatically determine whether it’s being asked for a content-addresseditem or not.  The guts of it is this procedure:
  (define* (content-addressed-item? item hash                                    #:key (hash-algo 'sha256))    "Return true if ITEM, a store file name, is definitely a content-addressed  item (result of a fixed-output derivation) with the given HASH of type  HASH-ALGO, false otherwise.
  Note: This procedure is useful when the deriver of ITEM is unknown.  In other  cases, the recommended approach is to check 'fixed-output-derivation?' on the  deriver."    ;; XXX: This returns #f for "text" items produced by 'add-text-to-store'.    ;; There's not much we can do because the file name for these is a function    ;; of their content.    (let ((name (store-path-package-name item)))      (or (string=? item (fixed-output-path name hash #:recursive? #f                                            #:hash-algo hash-algo))          (string=? item (fixed-output-path name hash #:recursive? #t                                            #:hash-algo hash-algo)))))
It works as expected for the result of “recursive fixed-outputderivations”—i.e., fixed-output derivations that produce a directory,such as VCS checkouts.
However it doesn’t work for fixed-output derivations that produce a flatfile, such as origins with the ‘url-fetch’ method.  The reason isbecause in the case of non-recursive derivations, the store file name iscomputed as a function of the file hash, not as a function of the narhash, whereas narinfos only contains the nar hash (the thing that ‘guixhash -r’ computes.)
So I think we have to communicate more info from the daemon to ‘guixsubstitute’.
Ludo’.

Ludovic Courtès wrote on 15 Dec 2017 10:30

Always enable substitutes for fixed-output derivations

Recipients:(name . Leo Famulari)(address . leo@famulari.name)

Message-ID:87a7ykmj7k.fsf_-_@gnu.org

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (3 lines)

> So I think we have to communicate more info from the daemon to ‘guix> substitute’.

The attached patch addresses that by simply calling out to the daemon todetermine whether we’re dealing with a content-addressed item.
To summarize, the new behavior is that substitutes are always enabledfor fixed-output derivations. That way, people willing to buildeverything from source can still use ‘--no-substitutes’ and yet be ableto retrieve source code without being penalized compared to someoneenabling substitutes wholesale.
Of course, when substitutes are missing, we fall back to regulardownloads or VCS checkouts. It is also still possible to choose wheresubstitutes are downloaded from, using ‘--substitute-urls’, or even topass an empty list of URLs.
Feedback welcome!
Ludo’.

From aab42bcb212698bc1f61beb9f321ffbd751f36f5 Mon Sep 17 00:00:00 2001From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>Date: Fri, 15 Dec 2017 09:57:04 +0100Subject: [PATCH 1/2] substitute: Always allow substitutes for fixed-output derivation results.
Fixes https://bugs.gnu.org/28659.
* guix/scripts/substitute.scm (content-addressed-item?): New procedure.(valid-narinfo?): Use it.* nix/libstore/build.cc (DerivationGoal::haveDerivation): Always make asubstitution goal when 'fixedOutput' is true.* tests/substitute.scm ("query unsigned narinfo for content-addresseditem"): New test.--- guix/scripts/substitute.scm | 31 ++++++++++++++++++++++++++++++- nix/libstore/build.cc       |  6 ++++-- tests/substitute.scm        | 24 +++++++++++++++++++++++- 3 files changed, 57 insertions(+), 4 deletions(-)

Toggle diff (121 lines)diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scmindex 2fd2bf810..670a9b4dd 100755--- a/guix/scripts/substitute.scm+++ b/guix/scripts/substitute.scm@@ -25,6 +25,9 @@   #:use-module (guix config)   #:use-module (guix records)   #:use-module ((guix serialization) #:select (restore-file))+  #:use-module ((guix derivations)+                #:select (read-derivation-from-file+                          fixed-output-derivation?))   #:use-module (guix hash)   #:use-module (guix base32)   #:use-module (guix base64)@@ -406,10 +409,36 @@ No authentication and authorization checks are performed here!"        (let ((above-signature (string-take contents index)))          (sha256 (string->utf8 above-signature))))))) +(define* (content-addressed-item? item)+  "Return true if ITEM is content-addressed---i.e., if ITEM is the result of a+fixed-output derivation."+  (guard (c ((nix-connection-error? c)+             ;; We failed to connect, maybe because we have the wrong+             ;; GUIX_DAEMON_SOCKET?  Let's conservatively assume that+             ;; nothing's content-addressed.+             #f))+    (with-store store+      (match (valid-derivers store item)+        (()+         ;; If there are no valid derivers it's most likely because ITEM is a+         ;; source (added with 'add-to-store' or similar).  Nevertheless,+         ;; since we can't be certain, return #f.+         #f)+        ((drv . _)+         (fixed-output-derivation?+          (read-derivation-from-file drv)))))))+ (define* (valid-narinfo? narinfo #:optional (acl (current-acl))                          #:key verbose?)-  "Return #t if NARINFO's signature is not valid."+  "Return #t if NARINFO is \"valid\"---signed by an authorized key, or+designating a content-addressed item."   (or %allow-unauthenticated-substitutes?++      ;; If NARINFO designates a content-addressed item, there's no point+      ;; authenticating it.  Don't explicitly check 'narinfo-hash' for+      ;; integrity: this will be done by the daemon once we've downloaded it.+      (content-addressed-item? (narinfo-path narinfo))+       (let ((hash      (narinfo-sha256 narinfo))             (signature (narinfo-signature narinfo))             (uri       (uri->string (narinfo-uri narinfo))))diff --git a/nix/libstore/build.cc b/nix/libstore/build.ccindex d68e8b2bc..03a8f5080 100644--- a/nix/libstore/build.cc+++ b/nix/libstore/build.cc@@ -1034,8 +1034,10 @@ void DerivationGoal::haveDerivation()      /* We are first going to try to create the invalid output paths        through substitutes.  If that doesn't work, we'll build-       them. */-    if (settings.useSubstitutes && substitutesAllowed(drv))+       them.  Always enable substitutes for fixed-output derivations to+       protect against disappearing files and in-place modifications on+       upstream sites.  */+    if ((fixedOutput || settings.useSubstitutes) && substitutesAllowed(drv))         foreach (PathSet::iterator, i, invalidOutputs)             addWaitee(worker.makeSubstitutionGoal(*i, buildMode == bmRepair)); diff --git a/tests/substitute.scm b/tests/substitute.scmindex 0ad624795..03579b9f1 100644--- a/tests/substitute.scm+++ b/tests/substitute.scm@@ -21,15 +21,17 @@   #:use-module (guix scripts substitute)   #:use-module (guix base64)   #:use-module (guix hash)+  #:use-module (guix derivations)   #:use-module (guix serialization)   #:use-module (guix pk-crypto)   #:use-module (guix pki)   #:use-module (guix config)   #:use-module (guix base32)-  #:use-module ((guix store) #:select (%store-prefix))+  #:use-module ((guix store) #:select (%store-prefix with-store))   #:use-module ((guix ui) #:select (guix-warning-port))   #:use-module ((guix build utils)                 #:select (mkdir-p delete-file-recursively))+  #:use-module (guix tests)   #:use-module (guix tests http)   #:use-module (rnrs bytevectors)   #:use-module (rnrs io ports)@@ -241,6 +243,26 @@ a file for NARINFO."            (lambda ()              (guix-substitute "--query")))))))) +(test-assert "query unsigned narinfo for content-addressed item"+  (with-store store+    (let* ((hash (sha256 (random-bytevector 128)))+           (drv  (derivation store "content-addressed"+                             "builtin:download" '()+                             #:hash-algo 'sha256 #:hash hash)))+      (define output+        (with-output-to-string+          (lambda ()+            (with-derivation-narinfo drv (sha256 => hash)+              (with-input-from-string (string-append "have "+                                                     (derivation->output-path drv))+                (lambda ()+                  (set! (@@ (guix scripts substitute)+                            %allow-unauthenticated-substitutes?)+                    #f)+                  (guix-substitute "--query")))))))++      (string=? (string-trim-both output) (derivation->output-path drv)))))+ (test-quit "substitute, no signature"     "no valid substitute"   (with-narinfo %narinfo-- 2.15.1

From 9bcf90b99a79f9f3e126cde5fe1cf51b0dfa58aa Mon Sep 17 00:00:00 2001From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= <ludo@gnu.org>Date: Fri, 15 Dec 2017 10:03:39 +0100Subject: [PATCH 2/2] Revert "download: Download a nar when a VCS checkout fails."
This reverts commit 37ce440dcffa9ff4f5401bacbc9619bd8ea561c1, which isuseless now that substitutes are always enabled for content-addresseditems.--- Makefile.am                 |   1 - guix/build/download-nar.scm | 125 -------------------------------------------- guix/cvs-download.scm       |  38 ++++---------- guix/git-download.scm       |  37 +++---------- guix/hg-download.scm        |  36 ++++--------- 5 files changed, 26 insertions(+), 211 deletions(-) delete mode 100644 guix/build/download-nar.scm

Toggle diff (337 lines)diff --git a/Makefile.am b/Makefile.amindex 85b9ab36d..d2660b0a7 100644--- a/Makefile.am+++ b/Makefile.am@@ -110,7 +110,6 @@ MODULES =					\   guix/ui.scm					\   guix/build/ant-build-system.scm		\   guix/build/download.scm			\-  guix/build/download-nar.scm			\   guix/build/cargo-build-system.scm		\   guix/build/cmake-build-system.scm		\   guix/build/dub-build-system.scm		\diff --git a/guix/build/download-nar.scm b/guix/build/download-nar.scmdeleted file mode 100644index 13f01fb1e..000000000--- a/guix/build/download-nar.scm+++ /dev/null@@ -1,125 +0,0 @@-;;; GNU Guix --- Functional package management for GNU-;;; Copyright © 2017 Ludovic Courtès <ludo@gnu.org>-;;;-;;; This file is part of GNU Guix.-;;;-;;; GNU Guix is free software; you can redistribute it and/or modify it-;;; under the terms of the GNU General Public License as published by-;;; the Free Software Foundation; either version 3 of the License, or (at-;;; your option) any later version.-;;;-;;; GNU Guix is distributed in the hope that it will be useful, but-;;; WITHOUT ANY WARRANTY; without even the implied warranty of-;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the-;;; GNU General Public License for more details.-;;;-;;; You should have received a copy of the GNU General Public License-;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.--(define-module (guix build download-nar)-  #:use-module (guix build download)-  #:use-module (guix build utils)-  #:use-module (guix serialization)-  #:use-module (guix zlib)-  #:use-module (guix progress)-  #:use-module (web uri)-  #:use-module (srfi srfi-11)-  #:use-module (srfi srfi-26)-  #:use-module (ice-9 format)-  #:use-module (ice-9 match)-  #:export (download-nar))--;;; Commentary:-;;;-;;; Download a normalized archive or "nar", similar to what 'guix substitute'-;;; does.  The intent here is to use substitute servers as content-addressed-;;; mirrors of VCS checkouts.  This is mostly useful for users who have-;;; disabled substitutes.-;;;-;;; Code:--(define (urls-for-item item)-  "Return the fallback nar URL for ITEM--e.g.,-\"/gnu/store/cabbag3…-foo-1.2-checkout\"."-  ;; Here we hard-code nar URLs without checking narinfos.  That's probably OK-  ;; though.-  ;; TODO: Use HTTPS?  The downside is the extra dependency.-  (let ((bases '("http://mirror.hydra.gnu.org/guix"-                 "http://berlin.guixsd.org"))-        (item  (basename item)))-    (append (map (cut string-append <> "/nar/gzip/" item) bases)-            (map (cut string-append <> "/nar/" item) bases))))--(define (restore-gzipped-nar port item size)-  "Restore the gzipped nar read from PORT, of SIZE bytes (compressed), to-ITEM."-  ;; Since PORT is typically a non-file port (for instance because 'http-get'-  ;; returns a delimited port), create a child process so we're back to a file-  ;; port that can be passed to 'call-with-gzip-input-port'.-  (match (pipe)-    ((input . output)-     (match (primitive-fork)-       (0-        (dynamic-wind-          (const #t)-          (lambda ()-            (close-port output)-            (close-port port)-            (catch #t-              (lambda ()-                (call-with-gzip-input-port input-                  (cut restore-file <> item)))-              (lambda (key . args)-                (print-exception (current-error-port)-                                 (stack-ref (make-stack #t) 1)-                                 key args)-                (primitive-exit 1))))-          (lambda ()-            (primitive-exit 0))))-       (child-        (close-port input)-        (dump-port* port output-                    #:reporter (progress-reporter/file item size-                                                       #:abbreviation-                                                       store-path-abbreviation))-        (close-port output)-        (newline)-        (match (waitpid child)-          ((_ . status)-           (unless (zero? status)-             (error "nar decompression failed" status)))))))))--(define (download-nar item)-  "Download and extract the normalized archive for ITEM.  Return #t on-success, #f otherwise."-  ;; Let progress reports go through.-  (setvbuf (current-error-port) _IONBF)-  (setvbuf (current-output-port) _IONBF)--  (let loop ((urls (urls-for-item item)))-    (match urls-      ((url rest ...)-       (format #t "Trying content-addressed mirror at ~a...~%"-               (uri-host (string->uri url)))-       (let-values (((port size)-                     (catch #t-                       (lambda ()-                         (http-fetch (string->uri url)))-                       (lambda args-                         (values #f #f)))))-         (if (not port)-             (loop rest)-             (begin-               (if size-                   (format #t "Downloading from ~a (~,2h MiB)...~%" url-                           (/ size (expt 2 20.)))-                   (format #t "Downloading from ~a...~%" url))-               (if (string-contains url "/gzip")-                   (restore-gzipped-nar port item size)-                   (begin-                     ;; FIXME: Add progress report.-                     (restore-file port item)-                     (close-port port)))-               #t))))-      (()-       #f))))diff --git a/guix/cvs-download.scm b/guix/cvs-download.scmindex 8b46f8ef8..85744c5b5 100644--- a/guix/cvs-download.scm+++ b/guix/cvs-download.scm@@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU-;;; Copyright © 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2014 Sree Harsha Totakura <sreeharsha@totakura.in> ;;; Copyright © 2015 Mark H Weaver <mhw@netris.org> ;;;@@ -23,7 +23,6 @@   #:use-module (guix gexp)   #:use-module (guix store)   #:use-module (guix monads)-  #:use-module (guix modules)   #:use-module (guix packages)   #:use-module (ice-9 match)   #:export (cvs-reference@@ -60,35 +59,16 @@   "Return a fixed-output derivation that fetches REF, a <cvs-reference> object.  The output is expected to have recursive hash HASH of type HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."-  (define zlib-    (module-ref (resolve-interface '(gnu packages compression)) 'zlib))--  (define config.scm-    (scheme-file "config.scm"-                 #~(begin-                     (define-module (guix config)-                       #:export (%libz))--                     (define %libz-                       #+(file-append zlib "/lib/libz")))))--  (define modules-    (cons `((guix config) => ,config.scm)-          (delete '(guix config)-                  (source-module-closure '((guix build cvs)-                                           (guix build download-nar))))))   (define build-    (with-imported-modules modules+    (with-imported-modules '((guix build cvs)+                             (guix build utils))       #~(begin-          (use-modules (guix build cvs)-                       (guix build download-nar))--          (or (cvs-fetch '#$(cvs-reference-root-directory ref)-                         '#$(cvs-reference-module ref)-                         '#$(cvs-reference-revision ref)-                         #$output-                         #:cvs-command (string-append #+cvs "/bin/cvs"))-              (download-nar #$output)))))+          (use-modules (guix build cvs))+          (cvs-fetch '#$(cvs-reference-root-directory ref)+                     '#$(cvs-reference-module ref)+                     '#$(cvs-reference-revision ref)+                     #$output+                     #:cvs-command (string-append #+cvs "/bin/cvs")))))    (mlet %store-monad ((guile (package->derivation guile system)))     (gexp->derivation (or name "cvs-checkout") builddiff --git a/guix/git-download.scm b/guix/git-download.scmindex 731e549b3..7397cbe7f 100644--- a/guix/git-download.scm+++ b/guix/git-download.scm@@ -25,7 +25,6 @@   #:use-module (guix monads)   #:use-module (guix records)   #:use-module (guix packages)-  #:use-module (guix modules)   #:autoload   (guix build-system gnu) (standard-packages)   #:use-module (ice-9 match)   #:use-module (ice-9 popen)@@ -78,31 +77,12 @@ HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."         (standard-packages)         '())) -  (define zlib-    (module-ref (resolve-interface '(gnu packages compression)) 'zlib))--  (define config.scm-    (scheme-file "config.scm"-                 #~(begin-                     (define-module (guix config)-                       #:export (%libz))--                     (define %libz-                       #+(file-append zlib "/lib/libz")))))--  (define modules-    (cons `((guix config) => ,config.scm)-          (delete '(guix config)-                  (source-module-closure '((guix build git)-                                           (guix build utils)-                                           (guix build download-nar))))))-   (define build-    (with-imported-modules modules+    (with-imported-modules '((guix build git)+                             (guix build utils))       #~(begin           (use-modules (guix build git)                        (guix build utils)-                       (guix build download-nar)                        (ice-9 match))            ;; The 'git submodule' commands expects Coreutils, sed,@@ -112,13 +92,12 @@ HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."                                            (((names dirs) ...)                                             dirs))) -          (or (git-fetch (getenv "git url") (getenv "git commit")-                         #$output-                         #:recursive? (call-with-input-string-                                          (getenv "git recursive?")-                                        read)-                         #:git-command (string-append #+git "/bin/git"))-              (download-nar #$output)))))+          (git-fetch (getenv "git url") (getenv "git commit")+                     #$output+                     #:recursive? (call-with-input-string+                                      (getenv "git recursive?")+                                    read)+                     #:git-command (string-append #+git "/bin/git")))))    (mlet %store-monad ((guile (package->derivation guile system)))     (gexp->derivation (or name "git-checkout") builddiff --git a/guix/hg-download.scm b/guix/hg-download.scmindex 6b25b87b6..842098090 100644--- a/guix/hg-download.scm+++ b/guix/hg-download.scm@@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU-;;; Copyright © 2014, 2015, 2016, 2017 Ludovic Courtès <ludo@gnu.org>+;;; Copyright © 2014, 2015, 2016 Ludovic Courtès <ludo@gnu.org> ;;; Copyright © 2016 Ricardo Wurmus <rekado@elephly.net> ;;; ;;; This file is part of GNU Guix.@@ -22,7 +22,6 @@   #:use-module (guix store)   #:use-module (guix monads)   #:use-module (guix records)-  #:use-module (guix modules)   #:use-module (guix packages)   #:autoload   (guix build-system gnu) (standard-packages)   #:use-module (ice-9 match)@@ -60,35 +59,18 @@   "Return a fixed-output derivation that fetches REF, a <hg-reference> object.  The output is expected to have recursive hash HASH of type HASH-ALGO (a symbol).  Use NAME as the file name, or a generic name if #f."-  (define zlib-    (module-ref (resolve-interface '(gnu packages compression)) 'zlib))--  (define config.scm-    (scheme-file "config.scm"-                 #~(begin-                     (define-module (guix config)-                       #:export (%libz))--                     (define %libz-                       #+(file-append zlib "/lib/libz")))))--  (define modules-    (cons `((guix config) => ,config.scm)-          (delete '(guix config)-                  (source-module-closure '((guix build hg)-                                           (guix build download-nar))))))-   (define build-    (with-imported-modules modules+    (with-imported-modules '((guix build hg)+                             (guix build utils))       #~(begin           (use-modules (guix build hg)-                       (guix build download-nar))+                       (guix build utils)+                       (ice-9 match)) -          (or (hg-fetch '#$(hg-reference-url ref)-                        '#$(hg-reference-changeset ref)-                        #$output-                        #:hg-command (string-append #+hg "/bin/hg"))-              (download-nar #$output)))))+          (hg-fetch '#$(hg-reference-url ref)+                    '#$(hg-reference-changeset ref)+                    #$output+                    #:hg-command (string-append #+hg "/bin/hg")))))    (mlet %store-monad ((guile (package->derivation guile system)))     (gexp->derivation (or name "hg-checkout") build-- 2.15.1

Ludovic Courtès wrote on 14 Feb 22:34 +0100

Re: bug#39575: guix time-machine fails when a tarball was modified in-place

Recipients:(name . Jan Nieuwenhuizen)(address . janneke@gnu.org)

Message-ID:878sl47t0q.fsf@gnu.org

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

Toggle quote (2 lines)

> Ludovic Courtès writes:

[...]

Toggle quote (8 lines)

>> The problem here is really that we fall back to content-addressed>> mirrors instead of using them directly:>>>> https://issues.guix.gnu.org/issue/28659>> Wait, what happened here; you finally proposed a patch two years ago and> nothing happened/we all forgot to follow up?

I think we forgot, indeed.
One thing I don’t quite like about the patch is the fact that ‘guixsubstitutes’ connects to the daemon in ‘content-addressed-item?’.
Also, one could argue that we’d steer users towards downloading from ourserver, which could be a privacy concern (probably not a strong argumentsince one can easily change the substitute URLs.)
Thoughts?
Ludo’.

zimoun wrote on 15 Feb 16:43 +0100

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:CAJ3okZ0-zPcs+pC4tQEymD-On-aN_-hgKRkRzBJzusdbtdYdAg@mail.gmail.com

Hi,
On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (4 lines)

> Also, one could argue that we’d steer users towards downloading from our> server, which could be a privacy concern (probably not a strong argument> since one can easily change the substitute URLs.)

I am not following the privacy concern.What do you mean?
Cheers,simon

Ludovic Courtès wrote on 16 Feb 11:59 +0100

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87k14m3iiy.fsf@gnu.org

Hi!
zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (9 lines)

> On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:>>> Also, one could argue that we’d steer users towards downloading from our>> server, which could be a privacy concern (probably not a strong argument>> since one can easily change the substitute URLs.)>> I am not following the privacy concern.> What do you mean?

I mean that by default, someone who’s disabled substitutes (presumablyout of security or privacy concerns) would find themself downloadingsource code from ci.guix.gnu.org instead of various upstream sites.
Ludo’.

zimoun wrote on 17 Feb 11:18 +0100

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:CAJ3okZ08ibXTBqsZMwnuEVdhpyXgHVp6+rNGXB02gsHVqwu53A@mail.gmail.com

Hi Ludo,
On Sun, 16 Feb 2020 at 11:59, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (14 lines)

> zimoun <zimon.toutoune@gmail.com> skribis:> > On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:
> >> Also, one could argue that we’d steer users towards downloading from our> >> server, which could be a privacy concern (probably not a strong argument> >> since one can easily change the substitute URLs.)> >> > I am not following the privacy concern.> > What do you mean?>> I mean that by default, someone who’s disabled substitutes (presumably> out of security or privacy concerns) would find themself downloading> source code from ci.guix.gnu.org instead of various upstream sites.

I do not see the difference between mirroring and traveling back intime with missing upstream sources.And because it is content-addressed, it seems even more secure thandownloading from a upstream URL, IMHO.If one trusts Guix, then an attacker needs to corrupt in the same timethe Guix history and Berlin (and/or any other farm).If one does not trust Guix, why does they use the recipe coming fromGuix? To be precise, this person has to check all the recipes of allthe dependencies.
Well, I do not see a security concern because we are talking aboutserving the sources.It is another story when the substitutes serve the results of thebuild (binaries); because one does not have any strong guarantee thatthe substitute serves the expected binaries.
By privacy concern, do you mean that Guix could collect who downloadswhat; in a central fashion? Which is not the case when one downloadsfrom several distributed upstream sources. Right?Well, I am not convinced because the case of missing upstream sourceis rare. And it is easy to protect against such collecting dataprocess.In paranoid mode, traveling back in time is becoming difficult becauseof the reliability of the sources; I mean if the sources werereliable, SWH would not exist. ;-) The solution should be an IPFS /GNUnet / full distributed archive... which is not ready... yet! :-)

Well, maybe for the TODO list of the time-machine: add an option toallow substitutes *only* for the sources (substitutes meaningci.guix.gnu.org and/or SWH). If this option does not exist yet. ;-)

Cheers,simon

Ludovic Courtès wrote on 17 Feb 15:40 +0100

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87pned6zw2.fsf@gnu.org

Hi,
zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (15 lines)> On Sun, 16 Feb 2020 at 11:59, Ludovic Courtès <ludo@gnu.org> wrote:>> zimoun <zimon.toutoune@gmail.com> skribis:>> > On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:>>> >> Also, one could argue that we’d steer users towards downloading from our>> >> server, which could be a privacy concern (probably not a strong argument>> >> since one can easily change the substitute URLs.)>> >>> > I am not following the privacy concern.>> > What do you mean?>>>> I mean that by default, someone who’s disabled substitutes (presumably>> out of security or privacy concerns) would find themself downloading>> source code from ci.guix.gnu.org instead of various upstream sites.

[...]

Toggle quote (4 lines)

> By privacy concern, do you mean that Guix could collect who downloads> what; in a central fashion? Which is not the case when one downloads> from several distributed upstream sources. Right?

Exactly. But like I wrote above, I don’t think it’s a strong argument.
What remains is the issue with ‘content-addressed-item?’, then.
Ludo’.

zimoun wrote on 17 Feb 16:04 +0100

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:CAJ3okZ0BrbfqmrsmfJykwqPA8PQFtSUB3JBdNaN1=npZNa36Eg@mail.gmail.com

On Mon, 17 Feb 2020 at 15:40, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (2 lines)

> Exactly. But like I wrote above, I don’t think it’s a strong argument.

I agree and the big picture depends on the audience.Scientific communities would be fine with centralized archives such asSWH. And only centralized archives IMHO can provide a reliable "longterm" support which is the point for that communities. (Quote becausenot clearly defined what it is. :-))Other communities would prefer distributed archive such as IPFS orGNUnet but 1. it still needs some work and 2. the "long term" is notguarantee by nature, IMHO. But it is probably not an issue for thatcommunities.

Toggle quote (2 lines)

> What remains is the issue with ‘content-addressed-item?’, then.

I agree.The bridge with SWH is in good shape, IMHO.And the pending IPFS patch would deserve more love. :-) Maybe soon...

Cheers,simon

zimoun wrote on 9 Sep 16:31 +0200

Re: bug#28659: Content-addressed mirror is not used upon invalid hash

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:87eenbuim8.fsf_-_@gmail.com

Hi,
On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (3 lines)

> One thing I don’t quite like about the patch is the fact that ‘guix> substitutes’ connects to the daemon in ‘content-addressed-item?’.

What is the status of this patch [1] following the recent discussion abouttar “disarchive” and SWH?
Related: - http://issues.guix.gnu.org/issue/39575 - http://issues.guix.gnu.org/42162 - https://git.ngyro.com/disarchive/ All the best,simon
[1] http://issues.guix.gnu.org/issue/28659#26

Ludovic Courtès wrote on 10 Sep 10:14 +0200

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87363qjbez.fsf@gnu.org

Hello,
zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (13 lines)> On Fri, 14 Feb 2020 at 22:34, Ludovic Courtès <ludo@gnu.org> wrote:>>> One thing I don’t quite like about the patch is the fact that ‘guix>> substitutes’ connects to the daemon in ‘content-addressed-item?’.>> What is the status of this patch [1] following the recent discussion about> tar “disarchive” and SWH?>> Related:>  - http://issues.guix.gnu.org/issue/39575>  - http://issues.guix.gnu.org/42162>  - https://git.ngyro.com/disarchive/

Thanks for the reminder. I don’t think Timothy’s work changes anythingwrt. to this issue: it would still need to be addressed.
Ludo’.

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date