tarballs generated on github are generated on demand (leading to different hash sums)

Done

Details

5 participants

Jan Nieuwenhuizen
Ludovic Courtès
Maxim Cournoyer
ng0
Ricardo Wurmus

Owner: unassigned

Submitted by: ng0

Severity: important

Debbugs page

ng0 wrote 7 years ago

Recipients:(address . bug-guix@gnu.org)

Message-ID:20171008114009.3tyhcuioaau6tlya@abyayala

Past and recent discussion in our IRC channel and on the mailing list

show that we can not rely on tarballs on github keeping the same

hash forever.

According to github they are "generated on demand", leading to

regular hash mismatches.

Since some of our own dependencies are on github (at the very least

guile-git), we need to come up with a solution.

Right now we have around 449 packages with tarball sources from

github in our gnu/packages.

We could:

- Move them all to use git-download and just use

the commit that has been tagged in the versions that produce

the tarballs on github.

- Mirror the content somewhere reliable in snapshots for

some time. Problem here: we start to rely on this "somewhere"

to be trustworthy and introduce one more point to trust

(however due to pre-recorded hash sum this is just an annoyance,

not a grave issue).

- Your idea here.

ng0

GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588

GnuPG: https://dist.krosos.org/dist/keys/

https://www.infotropique.orghttps://krosos.org

ng0 wrote 7 years ago

Recipients:(address . 28745@debbugs.gnu.org)

Message-ID:20171008114402.wxwhra6xxnxvc3pt@abyayala

ng0 transcribed 2.1K bytes:

…

Toggle quote (2 lines)

> Since some of our own dependencies are on github (at the very least

> guile-git), we need to come up with a solution.

…

Correction: libgit2 is on github, a dependency of guile-git (which is on gitlab).

ng0

GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588

GnuPG: https://dist.krosos.org/dist/keys/

https://www.infotropique.orghttps://krosos.org

Jan Nieuwenhuizen wrote 7 years ago

Recipients:(name . ng0)(address . ng0@infotropique.org)(address . 28745@debbugs.gnu.org)

Message-ID:87r2ud49tv.fsf@gnu.org

ng0 writes:

Toggle quote (8 lines)

> ng0 transcribed 2.1K bytes:

> …

>> Since some of our own dependencies are on github (at the very least

>> guile-git), we need to come up with a solution.

> …

> Correction: libgit2 is on github, a dependency of guile-git (which is on gitlab).

Sure, see bug#28659 ...possbily this needs to be merged that bug.

janneke

Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org

Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com

Ricardo Wurmus wrote 7 years ago

Recipients:(name . ng0)(address . ng0@infotropique.org)(address . 28745@debbugs.gnu.org)

Message-ID:87y3olmhd3.fsf@elephly.net

ng0 <ng0@infotropique.org> writes:

Toggle quote (3 lines)

> Right now we have around 449 packages with tarball sources from

> github in our gnu/packages.

I assume that this problem does not exist for tarballs that have been

signed and uploaded by the maintainer. This is only a problem for

auto-generated tarballs for tags, so it’s probably less than 449

packages.

Toggle quote (4 lines)

> - Move them all to use git-download and just use

> the commit that has been tagged in the versions that produce

> the tarballs on github.

This doesn’t seem like a bad idea. It’s not great that we’ll have to

bootstrap the build systems for all these packages.

Ricardo

GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC

https://elephly.net

Ludovic Courtès wrote 7 years ago

control message for bug #28745

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87inflbyoo.fsf@gnu.org

severity 28745 important

Maxim Cournoyer wrote 7 years ago

[PATCH] tarballs generated on github are generated on demand (leading to different hash sums)

Recipients:(name . bug#28745)(address . 28745@debbugs.gnu.org)

Message-ID:87k1zv7pos.fsf@gmail.com

Hello,

I could finish a script that helped me finding all of our affected

packages, verify that only the hash but not the content of the archives

had changed, as well as automate the hash update for those safe to

update.

Attached is the patch and the scripts I used. I think we might

want to reuse some of it to extend guix lint to warn packagers that

archives coming from .*github.*archives URL are not guaranteed to be

stable and that it would be better, if available, to use manually

uploaded releases archives.

Thanks!

Maxim

PS: I've also uploaded the scripts here:

https://notabug.org/apteryx/fiascofor ease of cloning. Any comments

about my nascent (ab)use of Scheme are welcome!

Attachment: 0001-gnu-packages-Fix-the-hashes-of-mutated-GitHub-archiv.patch

Attachment: file

;;; Script that detects problematic github packages.
;;; To run, use something like this in the "fiasco" dir:
;;; ~/src/guix/pre-inst-env guile -L . main.scm

(use-modules (fiasco finder)
	     (fiasco fixer))

(define (main)
  ;; You may select a different results-dir by parameterizing it
  ;; differently below. More parameters available to configure can be
  ;; found in (fiasco finder).
  (parameterize ((results-dir (string-append (getenv "HOME")
					     "/src/guile-hacks/fiasco")))
    (find-problematic-packages)
    (fix-packages-hash)))

(main)

(define-module (fiasco fixer)
  #:use-module (fiasco finder)
  #:use-module (guix base32)
  #:use-module (guix upstream)

  #:export (fix-packages-hash))

;;; Commentary:
;;;
;;; Repair the packages whose hash can be safely updated, as found by
;;; the finder script. This should be run from a checkout of the Guix
;;; source tree, e.g. as "./pre-inst-guix guile ~/src/guile-hacks/fiasco/run.scm

(define (result-needs-checking? result)
  (and (not (result-hash-ok? result))
       (not (result-safe-to-update? result))))

(define* (fix-packages-hash #:optional (file (results-file)))
  "Correct the packages whose hash can be safely updated, based on
data in FILE."
  (let* ((results (results-file->results file))
	 (results-to-check (filter result-needs-checking? results))
	 (actionable-results (filter result-safe-to-update? results)))

    (define (update-package-hash result)
      (when (not (null? (result->package result)))
	(let* ((package (result->package result))
	       (name (result-package-name result))
	       (version (result-package-version result))
	       (old-hash (result-guix-hash result))
	       (new-hash (result-upstream-hash result))
	       (new-hash-bv (nix-base32-string->bytevector new-hash)))
	  (format #t "~a: updating hash from ~s to ~s..." name old-hash new-hash)
	  (if (update-package-source package version new-hash-bv)
	      (format #t "   success~%")
	      (format #t "   failed~%")))))

    (format #t "The following packages require manual verification:~%")
    (for-each (lambda (r)
		(format #t "~a version ~a~%"
			(result-package-name r)
			(result-package-version r)))
	      results-to-check)
    (display "\n")

    (format #t "Attempting to repair the hashes of ~a packages...~%"
	    (length actionable-results))

    (for-each update-package-hash actionable-results)))

Ludovic Courtès wrote 7 years ago

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(name . bug#28745)(address . 28745-done@debbugs.gnu.org)

Message-ID:871slxcyz8.fsf@gnu.org

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (5 lines)

> I could finish a script that helped me finding all of our affected

> packages, verify that only the hash but not the content of the archives

> had changed, as well as automate the hash update for those safe to

> update.

Great job!

Toggle quote (6 lines)

> Attached is the patch and the scripts I used. I think we might

> want to reuse some of it to extend guix lint to warn packagers that

> archives coming from .*github.*archives URL are not guaranteed to be

> stable and that it would be better, if available, to use manually

> uploaded releases archives.

Unfortunately, it’s become commonplace to publish nothing else than a

Git tag. Now, in those cases, we could also use ‘git-fetch’, which

wouldn’t be affected by problems with generated tarballs.

Thoughts?

Toggle quote (4 lines)

> PS: I've also uploaded the scripts here:

> https://notabug.org/apteryx/fiasco for ease of cloning. Any comments

> about my nascent (ab)use of Scheme are welcome!

The code looks nice!

Toggle quote (16 lines)> From 774a764149ecb0e234ae09c9a0a273af671c3c86 Mon Sep 17 00:00:00 2001
> From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
> Date: Sun, 15 Oct 2017 22:17:12 -0400
> Subject: [PATCH] gnu: packages: Fix the hashes of mutated GitHub archives.
>
> Fixes bug https://bugs.gnu.org/28745.
>
> * gnu/packages/audio.scm (csound): Fix hash.
> * gnu/packages/engineering.scm (fritzing): Likewise.
> * gnu/packages/erlang.scm (erlang): Likewise.
> * gnu/packages/fonts.scm (font-google-material-design-icons): Likewise.
> * gnu/packages/graphics.scm (ogre): Likewise.
> * gnu/packages/java.scm (java-plexus-interpolation, antlr3): Likewise.
> * gnu/packages/serialization.scm (yaml-cpp): Likewise.
> * gnu/packages/version-control.scm (libgit2): Likewise.

I’ve checked the hashes by running:

./pre-inst-env guix build -S --no-substitutes csound fritzing erlang \

font-google-material-design-icons ogre java-plexus-interpolation \

antlr3 yaml-cpp libgit2 --max-jobs=2

and everything went well.

Pushed as fd75eb6cd4e5c689f9e6ce7dd8d87f423778d308, thanks!

Ludo’.

Closed

Maxim Cournoyer wrote 7 years ago

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(name . bug#28745)(address . 28745-done@debbugs.gnu.org)

Message-ID:87y3o350yn.fsf@gmail.com

ludo@gnu.org (Ludovic Courtès) writes:

Toggle quote (2 lines)

> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

[...]

Toggle quote (13 lines)>
>> Attached is the patch and the scripts I used. I think we might
>> want to reuse some of it to extend guix lint to warn packagers that
>> archives coming from .*github.*archives URL are not guaranteed to be
>> stable and that it would be better, if available, to use manually
>> uploaded releases archives.
>
> Unfortunately, it’s become commonplace to publish nothing else than a
> Git tag.  Now, in those cases, we could also use ‘git-fetch’, which
> wouldn’t be affected by problems with generated tarballs.
>
> Thoughts?

I think the status quo is reasonable for now; if this becomes a recurring

problem we can reopen the issue and do something more about it.

Toggle quote (6 lines)

>> PS: I've also uploaded the scripts here:

>> https://notabug.org/apteryx/fiasco for ease of cloning. Any comments

>> about my nascent (ab)use of Scheme are welcome!

> The code looks nice!

OK, that's reassuring! :)

Toggle quote (29 lines)>
>> From 774a764149ecb0e234ae09c9a0a273af671c3c86 Mon Sep 17 00:00:00 2001
>> From: Maxim Cournoyer <maxim.cournoyer@gmail.com>
>> Date: Sun, 15 Oct 2017 22:17:12 -0400
>> Subject: [PATCH] gnu: packages: Fix the hashes of mutated GitHub archives.
>>
>> Fixes bug https://bugs.gnu.org/28745.
>>
>> * gnu/packages/audio.scm (csound): Fix hash.
>> * gnu/packages/engineering.scm (fritzing): Likewise.
>> * gnu/packages/erlang.scm (erlang): Likewise.
>> * gnu/packages/fonts.scm (font-google-material-design-icons): Likewise.
>> * gnu/packages/graphics.scm (ogre): Likewise.
>> * gnu/packages/java.scm (java-plexus-interpolation, antlr3): Likewise.
>> * gnu/packages/serialization.scm (yaml-cpp): Likewise.
>> * gnu/packages/version-control.scm (libgit2): Likewise.
>
> I’ve checked the hashes by running:
>
>   ./pre-inst-env guix build -S --no-substitutes csound fritzing erlang \
>      font-google-material-design-icons ogre java-plexus-interpolation \
>      antlr3 yaml-cpp libgit2  --max-jobs=2
>
> and everything went well.
>
> Pushed as fd75eb6cd4e5c689f9e6ce7dd8d87f423778d308, thanks!
>
> Ludo’.

Thanks!

Maxim

Closed

Ricardo Wurmus wrote 7 years ago

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:871slvv4pn.fsf@elephly.net

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (6 lines)

> Unfortunately, it’s become commonplace to publish nothing else than a

> Git tag. Now, in those cases, we could also use ‘git-fetch’, which

> wouldn’t be affected by problems with generated tarballs.

> Thoughts?

For a couple of packages I’ve already started using git-fetch with the

tag (instead of the commit hash). I think that’s preferable over using

auto-generated tarballs.

Ricardo

GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC

https://elephly.net

Closed

Your comment

This issue is archived.

To comment on this conversation send an email to 28745@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 28745

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags

mumi command -t +confirmed -t +easy

Or, remove the moreinfo tag and set the help tag

mumi command -t -moreinfo -t +help

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date