Making package transformation options available to 'guix pull'?

OpenSubmitted by Carl Dong.
Details
4 participants
  • Carl Dong
  • Leo Famulari
  • Ludovic Courtès
  • zimoun
Owner
unassigned
Severity
normal
C
C
Carl Dong wrote on 20 Feb 00:24 +0100
Extending package transformation options
(address . bug-guix@gnu.org)
A993FD81-6D70-4834-A124-005BD2AD4E22@carldong.me
Hi Guix!
TL;DR Shall we add a `--without-tests` option (or something similar) for `guix time-machine`?
Our release workflow in Bitcoin is planned to shift over to Guix for the next release, and as such we have a lot more testers who are running Guix as we speak on their machines! Exciting!
Our build workflow basically boils down to something like:
Toggle snippet (5 lines)$ guix time-machine --commit=<pinned-guix> … \ -- environment --manifest=manifest.scm … \ -- <build-command>
The fact that this is even possible (and allows reproducibility across time) speaks to the awesomeness of Guix! :-)
One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:1. Spurious tests2. “Expiring” tests (like the gnutls one)3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)
I think that the package transformation options are the perfect “hack-around” solution for this, namely `--without-tests`. In fact, since those who experience these problems are building from source, it makes no difference that the store item would be change: they would need to build everything anyway!
However, in the case of our workflow above, there is no way to specify `--without-tests=` to `guix time-machine`, which insists on building the broken `gnutls` package and running its tests. I’ve tried isolating the command to just:
Toggle snippet (4 lines)$ guix time-machine --commit=<pinned-guix> … \ -- --help
To make sure that it’s `guix time-machine` which is trying to build gnutls and not the time-machine’d `guix environment`.
I’m not too familiar with how package transformation options work internally, but I’m guessing all we need here is a `--without-tests` option (or something similar) for `guix time-machine`? Is there anything else that’s necessary to successfully skip tests for my workflow?
Cheers,Carl Dongcontact@carldong.me"I fight for the users"
L
L
Leo Famulari wrote on 20 Feb 01:17 +0100
(name . Carl Dong)(address . contact@carldong.me)(address . 46650@debbugs.gnu.org)
YDBVJL/lDRIGLG3T@jasmine.lan
On Fri, Feb 19, 2021 at 06:24:57PM -0500, Carl Dong wrote:
Toggle quote (5 lines)> One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:> 1. Spurious tests> 2. “Expiring” tests (like the gnutls one)> 3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)
I hate to throw a wrench in your plans, but I think that using packagetransformation options on the command-line militates against many of thebenefits you are seeking to take advantage of, in terms of anintrospectable and bootstrappable software system.
Normally when using Guix, our Git tree provides a convenient andcomprehensive view of the entire software system. It also provides anentry point into the system for making changes.
When using these command-line package transformations, you diverge fromthe Guix Git tree and enter uncharted and untracked territory. If youinstall some software or build some system using these transformations,there is no Git tree you can check out and begin exploring. For example,there will be no package definition of GnuTLS-without-tests that can bereferred to and shared.
L
L
Ludovic Courtès wrote on 20 Feb 14:57 +0100
(name . Carl Dong)(address . contact@carldong.me)(address . 46650@debbugs.gnu.org)
87h7m6j0yi.fsf@gnu.org
Hi,
Carl Dong <contact@carldong.me> skribis:
Toggle quote (5 lines)> One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:> 1. Spurious tests> 2. “Expiring” tests (like the gnutls one)> 3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)
Yes, these flaky/uncontrolled problems were one of the motivations for‘--without-tests’.
Toggle quote (9 lines)> However, in the case of our workflow above, there is no way to specify `--without-tests=` to `guix time-machine`, which insists on building the broken `gnutls` package and running its tests. I’ve tried isolating the command to just:>> --8<---------------cut here---------------start------------->8---> $ guix time-machine --commit=<pinned-guix> … \> -- --help> --8<---------------cut here---------------end--------------->8—>> To make sure that it’s `guix time-machine` which is trying to build gnutls and not the time-machine’d `guix environment`.
Package transformation options currently only apply to packagesspecified on the command line of ‘guix install’, ‘guix build’, etc.These options are not available to ‘guix system’, ‘guix pull’, and othernon-package-oriented commands.
Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends onGnuTLS. So here, we have a special case: to build Guix itself, you needto build GnuTLS first, and there’s no way to pass ‘--without-tests’ atthat level.
I’m not sure how to change that because unfortunately, ‘guix pull’ and‘time-machine’ don’t even know that Guix depends on GnuTLS; only thetarget Guix knows about this. :-/
Thanks,Ludo’.
L
L
Ludovic Courtès wrote on 20 Feb 15:00 +0100
(name . Leo Famulari)(address . leo@famulari.name)
87blcej0v1.fsf@gnu.org
Leo Famulari <leo@famulari.name> skribis:
Toggle quote (22 lines)> On Fri, Feb 19, 2021 at 06:24:57PM -0500, Carl Dong wrote:>> One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:>> 1. Spurious tests>> 2. “Expiring” tests (like the gnutls one)>> 3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)>> I hate to throw a wrench in your plans, but I think that using package> transformation options on the command-line militates against many of the> benefits you are seeking to take advantage of, in terms of an> introspectable and bootstrappable software system.>> Normally when using Guix, our Git tree provides a convenient and> comprehensive view of the entire software system. It also provides an> entry point into the system for making changes.>> When using these command-line package transformations, you diverge from> the Guix Git tree and enter uncharted and untracked territory. If you> install some software or build some system using these transformations,> there is no Git tree you can check out and begin exploring. For example,> there will be no package definition of GnuTLS-without-tests that can be> referred to and shared.
I think that’s OK: you’re building a different GnuTLS derivation, true,but how that derivation is computed is fully under control as thetransformation option appears on the command line or in your manifest.
It’s really equivalent to forking Guix and adding #:tests? #f, only moreflexible.
Ludo’.
Z
Z
zimoun wrote on 23 Feb 23:22 +0100
(address . 46650@debbugs.gnu.org)
86pn0qtofg.fsf@gmail.com
Hi,
On Sat, 20 Feb 2021 at 14:57, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (9 lines)> Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends on> GnuTLS. So here, we have a special case: to build Guix itself, you need> to build GnuTLS first, and there’s no way to pass ‘--without-tests’ at> that level.>> I’m not sure how to change that because unfortunately, ‘guix pull’ and> ‘time-machine’ don’t even know that Guix depends on GnuTLS; only the> target Guix knows about this. :-/
If «“guix pull” and ’time-machine’ don’t even know that Guix depends onGnuTLS» and «to build Guix itself, you need to build GnuTLS first», howdoes it work? How does «you» know that they needs to build GnuTLSfirst?
Well, I do not know if it is viable, one hack should to use the packageGuix to collect this missing knowledge. Why is it not possible to buildall the package dependencies of Guix with the option --without-tests?

Cheers,simon
L
L
Ludovic Courtès wrote on 24 Feb 12:11 +0100
(name . zimoun)(address . zimon.toutoune@gmail.com)
87sg5l3ekn.fsf@gnu.org
Hi,
zimoun <zimon.toutoune@gmail.com> skribis:
Toggle quote (16 lines)> On Sat, 20 Feb 2021 at 14:57, Ludovic Courtès <ludo@gnu.org> wrote:>>> Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends on>> GnuTLS. So here, we have a special case: to build Guix itself, you need>> to build GnuTLS first, and there’s no way to pass ‘--without-tests’ at>> that level.>>>> I’m not sure how to change that because unfortunately, ‘guix pull’ and>> ‘time-machine’ don’t even know that Guix depends on GnuTLS; only the>> target Guix knows about this. :-/>> If «“guix pull” and ’time-machine’ don’t even know that Guix depends on> GnuTLS» and «to build Guix itself, you need to build GnuTLS first», how> does it work? How does «you» know that they needs to build GnuTLS> first?
‘guix pull’ builds Guix using the method implemented in (guix self).The entry point there is ‘compiled-guix’, which returns a compiled Guix.
It’s not a package so the package API doesn’t apply.
HTH,Ludo’.
Z
Z
zimoun wrote on 24 Feb 14:43 +0100
(name . Ludovic Courtès)(address . ludo@gnu.org)
CAJ3okZ2xRR5CjcQ7y6jSZ9-XF2qg9cnH4RZZ=U0MJhjJtkr_Xw@mail.gmail.com
Hi Ludo,
On Wed, 24 Feb 2021 at 12:11, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (19 lines)> > On Sat, 20 Feb 2021 at 14:57, Ludovic Courtès <ludo@gnu.org> wrote:> >> >> Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends on> >> GnuTLS. So here, we have a special case: to build Guix itself, you need> >> to build GnuTLS first, and there’s no way to pass ‘--without-tests’ at> >> that level.> >>> >> I’m not sure how to change that because unfortunately, ‘guix pull’ and> >> ‘time-machine’ don’t even know that Guix depends on GnuTLS; only the> >> target Guix knows about this. :-/> >> > If «“guix pull” and ’time-machine’ don’t even know that Guix depends on> > GnuTLS» and «to build Guix itself, you need to build GnuTLS first», how> > does it work? How does «you» know that they needs to build GnuTLS> > first?>> ‘guix pull’ builds Guix using the method implemented in (guix self).> The entry point there is ‘compiled-guix’, which returns a compiled Guix.
Thanks. It perfectly answer to my question. :-)
Toggle quote (2 lines)> It’s not a package so the package API doesn’t apply.
I am confused. The 'specification->package' returns a regularpackage, right?. And then,
Toggle snippet (10 lines) (define gnutls (specification->package "gnutls"))
(define dependencies (append-map transitive-package-dependencies (list guile-gcrypt gnutls guile-git guile-avahi guile-json guile-semver guile-ssh guile-sqlite3 guile-zlib guile-lzlib guile-zstd)))
So here, a transformation could be applied. For example, build allthe dependencies without testing them; at the 'define gnutls' (&co)step or in the map. Why not?
Well, just to be able to travel back in time if some expiration changed.

Cheers,simon
L
L
Ludovic Courtès wrote on 24 Feb 18:21 +0100
(name . zimoun)(address . zimon.toutoune@gmail.com)
87h7m12xfw.fsf@gnu.org
Hi,
zimoun <zimon.toutoune@gmail.com> skribis:
Toggle quote (16 lines)> I am confused. The 'specification->package' returns a regular> package, right?. And then,>> (define gnutls> (specification->package "gnutls"))>> (define dependencies> (append-map transitive-package-dependencies> (list guile-gcrypt gnutls guile-git guile-avahi> guile-json guile-semver guile-ssh guile-sqlite3> guile-zlib guile-lzlib guile-zstd)))>> So here, a transformation could be applied. For example, build all> the dependencies without testing them; at the 'define gnutls' (&co)> step or in the map. Why not?
Sure, but how do you get a user-specified transformation to this line?
Ludo’.
Z
Z
zimoun wrote on 26 Feb 22:44 +0100
(name . Ludovic Courtès)(address . ludo@gnu.org)
86czwm1p3y.fsf@gmail.com
Hi,
On Wed, 24 Feb 2021 at 18:21, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (6 lines)>> So here, a transformation could be applied. For example, build all>> the dependencies without testing them; at the 'define gnutls' (&co)>> step or in the map. Why not?>> Sure, but how do you get a user-specified transformation to this line?
I do not know (yet?). :-)
First, we are only speaking about “guix time-machine” and second, onlyabout corner cases which could break in the future, as expiration.Right?We are not talking about general transformations applied to self (evenif it could be great).
So we could imagine something like ’set-build-from-command-line’ or’set-build-option’ to tweak self only for the inferior. I have nottraced (yet) on the stack of calls from “guix time-machine” to self viathe inferiors, but somehow it looks like passing arguments.
Well, be able to tweak how to build Guix via the time-machine appears tome better than configuring a build machine ahead of time. It is likepredicting the future and instead we should try to have a flexiblepresent in order to fix the past once in the unpredictable future. :-)

Cheers,simon
L
L
Ludovic Courtès wrote on 18 Mar 14:41 +0100
control message for bug #46650
(address . control@debbugs.gnu.org)
875z1opoit.fsf@gnu.org
retitle 46650 Making package transformation options available to 'guix pull'?quit
?