ELPA packages are fetched from unstable url -> not reproducible

  • Open
  • quality assurance status badge
Details
2 participants
  • Johannes Rosenberger
  • zimoun
Owner
unassigned
Submitted by
Johannes Rosenberger
Severity
normal
J
J
Johannes Rosenberger wrote on 1 Mar 2021 14:15
(address . bug-guix@gnu.org)
1614603812.pcr83awkcn.jorsn@jorsn.eu
Hey Guixers,

i think guix makes the same mistake Nixpkgs make (at least when I looked up
what guix is doing around two weeks ago):

They fetch the uncompressed tars built by ELPA.

These are only available for the newest version of a package.
ELPA keeps compressed archives only of around 20 hand-selected versions.
All package versions are kept in their git repo, which is a complete archive,
but there you must somehow extract the commit hash of a version.

Details are here:


I proposed possible solutions in the Nixpkgs issue.


Best,

Johannes
Z
Z
zimoun wrote on 20 Mar 2021 23:33
(address . 46849@debbugs.gnu.org)
86h7l5wj44.fsf@gmail.com
Hi,

I did a mistake and the bug report had not been CC.

Cheers,
simon

-------------------- Start of forwarded message --------------------
From: zimoun <zimon.toutoune@gmail.com>
To: Johannes Rosenberger <johannes@jorsn.eu>
Subject: Re: bug#46849: ELPA packages are fetched from unstable url -> not
reproducible
Date: Fri, 05 Mar 2021 02:32:11 +0100

Hi,

Thanks for the notification.

On Mon, 01 Mar 2021 at 14:15, Johannes Rosenberger <johannes@jorsn.eu> wrote:

Toggle quote (5 lines)
> These are only available for the newest version of a package.
> ELPA keeps compressed archives only of around 20 hand-selected versions.
> All package versions are kept in their git repo, which is a complete archive,
> but there you must somehow extract the commit hash of a version.

So it would break the “guix time-machine”, right?

There is 2 solutions:

1- trust the future Tarball Heritage [1]
2- switch to git-fetch all the ELPA packages.

Toggle quote (2 lines)
About #2, I am confused by this quote:

If you can work from the elpa.git instead, then you'll avoid
those problems (but the content is slightly different, so it
might be less convenient).




All the best,
simon
-------------------- End of forwarded message --------------------
Z
Z
zimoun wrote on 20 Mar 2021 23:35
(address . 46849@debbugs.gnu.org)
86eeg9wj0p.fsf@gmail.com
Follow-up.

-------------------- Start of forwarded message --------------------
Date: Fri, 05 Mar 2021 12:56:27 +0100
From: Johannes Rosenberger <johannes@jorsn.eu>
Subject: Re: bug#46849: ELPA packages are fetched from unstable url -> not
reproducible
To: zimoun <zimon.toutoune@gmail.com>

Hi Simon,


Excerpts from zimoun's message of March 5, 2021 2:32 am:

Toggle quote (9 lines)
> On Mon, 01 Mar 2021 at 14:15, Johannes Rosenberger <johannes@jorsn.eu> wrote:
>
>> These are only available for the newest version of a package.
>> ELPA keeps compressed archives only of around 20 hand-selected versions.
>> All package versions are kept in their git repo, which is a complete archive,
>> but there you must somehow extract the commit hash of a version.
>
> So it would break the “guix time-machine”, right?

Not only this. In Nixpkgs it broke the release of auctex in the
stable branch, because this wasn't at the newest version.
The old version was still available lz-compressed, but there is no
guarantee for this.

Toggle quote (5 lines)
> There is 2 solutions:
>
> 1- trust the future Tarball Heritage [1]
> 2- switch to git-fetch all the ELPA packages.

I documented (2) there:


There is one third solution:

3- trust archive.org

In Nixpkgs we also add archive.org urls as secondary source urls for
proprietary printer drivers.

Toggle quote (8 lines)
>
> About #2, I am confused by this quote:
>
> If you can work from the elpa.git instead, then you'll avoid
> those problems (but the content is slightly different, so it
> might be less convenient).

I don't understand this sentence either, because the file


seems to create the packages, so every package in the elpa built from
the git should be the same. One could check whether all packages on ELPA
are also in the git and vice versa. Also, some packages might not be
`external` in the language of ELPA, so not residing in an `external/*`
branch.


Best,

Johannes
-------------------- End of forwarded message --------------------
Z
Z
zimoun wrote on 20 Mar 2021 23:41
868s6hwirn.fsf@gmail.com
Follow up 2.

-------------------- Start of forwarded message --------------------
Date: Fri, 05 Mar 2021 13:08:05 +0100
From: Johannes Rosenberger <johannes@jorsn.eu>
Subject: Re: bug#46849: ELPA packages are fetched from unstable url -> not
reproducible
To: zimoun <zimon.toutoune@gmail.com>

Excerpts from Johannes Rosenberger's message of March 5, 2021 12:56 pm:

Toggle quote (8 lines)
> Excerpts from zimoun's message of March 5, 2021 2:32 am:
>
>> There is 2 solutions:
>>
>> 1- trust the future Tarball Heritage [1]
>> 2- switch to git-fetch all the ELPA packages.
> 3- trust archive.org

and maybe a fourth one:

(Blog entry about Nix & this by Tweag: https://www.softwareheritage.org/)

Best,

Johannes

-------------------- End of forwarded message --------------------
Z
Z
zimoun wrote on 20 Mar 2021 23:41
864kh5wiqb.fsf@gmail.com
Follow up 3.

-------------------- Start of forwarded message --------------------
From: zimoun <zimon.toutoune@gmail.com>
To: Johannes Rosenberger <johannes@jorsn.eu>
Subject: Re: bug#46849: ELPA packages are fetched from unstable url -> not
reproducible
Date: Fri, 05 Mar 2021 13:31:09 +0100

Hi Johannes,

On Fri, 05 Mar 2021 at 13:08, Johannes Rosenberger <johannes@jorsn.eu> wrote:

Toggle quote (6 lines)
>>> There is 2 solutions:
>>>
>>> 1- trust the future Tarball Heritage [1]
>>> 2- switch to git-fetch all the ELPA packages.
>> 3- trust archive.org

About archive.org, I do not know. Currently, there is no fallback in
Guix to it that I am aware, and nothing planned AFAIK.

Toggle quote (5 lines)
> and maybe a fourth one:
>
> 4- https://www.softwareheritage.org/
> (Blog entry about Nix & this by Tweag: https://www.softwareheritage.org/)

Yeah, this is what I called #1. :-) Currently, via the ’nixguix’ SWH
loader [1], packages using url-fetch are archived via the file [2].
However, work remains to have a full robust end-to-end solution:

a) not all the extensions of ’url-fetch’ are archived (and I do not
remember the status about the .el)

b) the fallback is not robust because of inconsistent addresses
between SWH (swh-id) and the-rest-of-the-world (checksum hashes)–to say
it quickly.

The aim of the disarchive’s project [3] is to address b) by creating a
bridge, i.e., stores in a separate database [4] the structure of the
metadata and then rebuild the archive from a checksum using the files
addressed by swh-id.




Cheers,
simon
-------------------- End of forwarded message --------------------
?