Making package transformation options available to 'guix pull'?

  • Open
  • quality assurance status badge
Details
4 participants
  • Carl Dong
  • Leo Famulari
  • Ludovic Courtès
  • zimoun
Owner
unassigned
Submitted by
Carl Dong
Severity
normal
C
C
Carl Dong wrote on 20 Feb 2021 00:24
Extending package transformation options
(address . bug-guix@gnu.org)
A993FD81-6D70-4834-A124-005BD2AD4E22@carldong.me
Hi Guix!

TL;DR Shall we add a `--without-tests` option (or something similar) for `guix time-machine`?

Our release workflow in Bitcoin is planned to shift over to Guix for the next release, and as such we have a lot more testers who are running Guix as we speak on their machines! Exciting!

Our build workflow basically boils down to something like:

Toggle snippet (5 lines)
$ guix time-machine --commit=<pinned-guix> … \
-- environment --manifest=manifest.scm … \
-- <build-command>

The fact that this is even possible (and allows reproducibility across time) speaks to the awesomeness of Guix! :-)

One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:
1. Spurious tests
2. “Expiring” tests (like the gnutls one)
3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)

I think that the package transformation options are the perfect “hack-around” solution for this, namely `--without-tests`. In fact, since those who experience these problems are building from source, it makes no difference that the store item would be change: they would need to build everything anyway!

However, in the case of our workflow above, there is no way to specify `--without-tests=` to `guix time-machine`, which insists on building the broken `gnutls` package and running its tests. I’ve tried isolating the command to just:

Toggle snippet (4 lines)
$ guix time-machine --commit=<pinned-guix> … \
-- --help

To make sure that it’s `guix time-machine` which is trying to build gnutls and not the time-machine’d `guix environment`.

I’m not too familiar with how package transformation options work internally, but I’m guessing all we need here is a `--without-tests` option (or something similar) for `guix time-machine`? Is there anything else that’s necessary to successfully skip tests for my workflow?

Cheers,
Carl Dong
contact@carldong.me
"I fight for the users"
L
L
Leo Famulari wrote on 20 Feb 2021 01:17
(name . Carl Dong)(address . contact@carldong.me)(address . 46650@debbugs.gnu.org)
YDBVJL/lDRIGLG3T@jasmine.lan
On Fri, Feb 19, 2021 at 06:24:57PM -0500, Carl Dong wrote:
Toggle quote (5 lines)
> One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:
> 1. Spurious tests
> 2. “Expiring” tests (like the gnutls one)
> 3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)

I hate to throw a wrench in your plans, but I think that using package
transformation options on the command-line militates against many of the
benefits you are seeking to take advantage of, in terms of an
introspectable and bootstrappable software system.

Normally when using Guix, our Git tree provides a convenient and
comprehensive view of the entire software system. It also provides an
entry point into the system for making changes.

When using these command-line package transformations, you diverge from
the Guix Git tree and enter uncharted and untracked territory. If you
install some software or build some system using these transformations,
there is no Git tree you can check out and begin exploring. For example,
there will be no package definition of GnuTLS-without-tests that can be
referred to and shared.
L
L
Ludovic Courtès wrote on 20 Feb 2021 14:57
(name . Carl Dong)(address . contact@carldong.me)(address . 46650@debbugs.gnu.org)
87h7m6j0yi.fsf@gnu.org
Hi,

Carl Dong <contact@carldong.me> skribis:

Toggle quote (5 lines)
> One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:
> 1. Spurious tests
> 2. “Expiring” tests (like the gnutls one)
> 3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)

Yes, these flaky/uncontrolled problems were one of the motivations for
‘--without-tests’.

Toggle quote (9 lines)
> However, in the case of our workflow above, there is no way to specify `--without-tests=` to `guix time-machine`, which insists on building the broken `gnutls` package and running its tests. I’ve tried isolating the command to just:
>
> --8<---------------cut here---------------start------------->8---
> $ guix time-machine --commit=<pinned-guix> … \
> -- --help
> --8<---------------cut here---------------end--------------->8—
>
> To make sure that it’s `guix time-machine` which is trying to build gnutls and not the time-machine’d `guix environment`.

Package transformation options currently only apply to packages
specified on the command line of ‘guix install’, ‘guix build’, etc.
These options are not available to ‘guix system’, ‘guix pull’, and other
non-package-oriented commands.

Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends on
GnuTLS. So here, we have a special case: to build Guix itself, you need
to build GnuTLS first, and there’s no way to pass ‘--without-tests’ at
that level.

I’m not sure how to change that because unfortunately, ‘guix pull’ and
‘time-machine’ don’t even know that Guix depends on GnuTLS; only the
target Guix knows about this. :-/

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 20 Feb 2021 15:00
(name . Leo Famulari)(address . leo@famulari.name)
87blcej0v1.fsf@gnu.org
Leo Famulari <leo@famulari.name> skribis:

Toggle quote (22 lines)
> On Fri, Feb 19, 2021 at 06:24:57PM -0500, Carl Dong wrote:
>> One of the main reasons for switching over to Guix is for its bootstrappbility and the ability to perform `--bootstrap --no-substitutes` builds. As mentioned in the gnutls ticket, performing these source-based builds can sometimes reveal problems with tests suites:
>> 1. Spurious tests
>> 2. “Expiring” tests (like the gnutls one)
>> 3. Foreign distro problems (0zmcam9sdiag9s1qx3xvlf8a3jnm53x2-sed-4.8.drv fails to build on SELinux-based distros because of a test failure in testsuite/inplace-selinux)
>
> I hate to throw a wrench in your plans, but I think that using package
> transformation options on the command-line militates against many of the
> benefits you are seeking to take advantage of, in terms of an
> introspectable and bootstrappable software system.
>
> Normally when using Guix, our Git tree provides a convenient and
> comprehensive view of the entire software system. It also provides an
> entry point into the system for making changes.
>
> When using these command-line package transformations, you diverge from
> the Guix Git tree and enter uncharted and untracked territory. If you
> install some software or build some system using these transformations,
> there is no Git tree you can check out and begin exploring. For example,
> there will be no package definition of GnuTLS-without-tests that can be
> referred to and shared.

I think that’s OK: you’re building a different GnuTLS derivation, true,
but how that derivation is computed is fully under control as the
transformation option appears on the command line or in your manifest.

It’s really equivalent to forking Guix and adding #:tests? #f, only more
flexible.

Ludo’.
Z
Z
zimoun wrote on 23 Feb 2021 23:22
(address . 46650@debbugs.gnu.org)
86pn0qtofg.fsf@gmail.com
Hi,

On Sat, 20 Feb 2021 at 14:57, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (9 lines)
> Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends on
> GnuTLS. So here, we have a special case: to build Guix itself, you need
> to build GnuTLS first, and there’s no way to pass ‘--without-tests’ at
> that level.
>
> I’m not sure how to change that because unfortunately, ‘guix pull’ and
> ‘time-machine’ don’t even know that Guix depends on GnuTLS; only the
> target Guix knows about this. :-/

If «“guix pull” and ’time-machine’ don’t even know that Guix depends on
GnuTLS» and «to build Guix itself, you need to build GnuTLS first», how
does it work? How does «you» know that they needs to build GnuTLS
first?

Well, I do not know if it is viable, one hack should to use the package
Guix to collect this missing knowledge. Why is it not possible to build
all the package dependencies of Guix with the option --without-tests?


Cheers,
simon
L
L
Ludovic Courtès wrote on 24 Feb 2021 12:11
(name . zimoun)(address . zimon.toutoune@gmail.com)
87sg5l3ekn.fsf@gnu.org
Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (16 lines)
> On Sat, 20 Feb 2021 at 14:57, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends on
>> GnuTLS. So here, we have a special case: to build Guix itself, you need
>> to build GnuTLS first, and there’s no way to pass ‘--without-tests’ at
>> that level.
>>
>> I’m not sure how to change that because unfortunately, ‘guix pull’ and
>> ‘time-machine’ don’t even know that Guix depends on GnuTLS; only the
>> target Guix knows about this. :-/
>
> If «“guix pull” and ’time-machine’ don’t even know that Guix depends on
> GnuTLS» and «to build Guix itself, you need to build GnuTLS first», how
> does it work? How does «you» know that they needs to build GnuTLS
> first?

‘guix pull’ builds Guix using the method implemented in (guix self).
The entry point there is ‘compiled-guix’, which returns a compiled Guix.

It’s not a package so the package API doesn’t apply.

HTH,
Ludo’.
Z
Z
zimoun wrote on 24 Feb 2021 14:43
(name . Ludovic Courtès)(address . ludo@gnu.org)
CAJ3okZ2xRR5CjcQ7y6jSZ9-XF2qg9cnH4RZZ=U0MJhjJtkr_Xw@mail.gmail.com
Hi Ludo,

On Wed, 24 Feb 2021 at 12:11, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (19 lines)
> > On Sat, 20 Feb 2021 at 14:57, Ludovic Courtès <ludo@gnu.org> wrote:
> >
> >> Guix as returned by ‘guix pull’ and ‘guix time-machine’ depends on
> >> GnuTLS. So here, we have a special case: to build Guix itself, you need
> >> to build GnuTLS first, and there’s no way to pass ‘--without-tests’ at
> >> that level.
> >>
> >> I’m not sure how to change that because unfortunately, ‘guix pull’ and
> >> ‘time-machine’ don’t even know that Guix depends on GnuTLS; only the
> >> target Guix knows about this. :-/
> >
> > If «“guix pull” and ’time-machine’ don’t even know that Guix depends on
> > GnuTLS» and «to build Guix itself, you need to build GnuTLS first», how
> > does it work? How does «you» know that they needs to build GnuTLS
> > first?
>
> ‘guix pull’ builds Guix using the method implemented in (guix self).
> The entry point there is ‘compiled-guix’, which returns a compiled Guix.

Thanks. It perfectly answer to my question. :-)

Toggle quote (2 lines)
> It’s not a package so the package API doesn’t apply.

I am confused. The 'specification->package' returns a regular
package, right?. And then,

Toggle snippet (10 lines)
(define gnutls
(specification->package "gnutls"))

(define dependencies
(append-map transitive-package-dependencies
(list guile-gcrypt gnutls guile-git guile-avahi
guile-json guile-semver guile-ssh guile-sqlite3
guile-zlib guile-lzlib guile-zstd)))

So here, a transformation could be applied. For example, build all
the dependencies without testing them; at the 'define gnutls' (&co)
step or in the map. Why not?

Well, just to be able to travel back in time if some expiration changed.


Cheers,
simon
L
L
Ludovic Courtès wrote on 24 Feb 2021 18:21
(name . zimoun)(address . zimon.toutoune@gmail.com)
87h7m12xfw.fsf@gnu.org
Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (16 lines)
> I am confused. The 'specification->package' returns a regular
> package, right?. And then,
>
> (define gnutls
> (specification->package "gnutls"))
>
> (define dependencies
> (append-map transitive-package-dependencies
> (list guile-gcrypt gnutls guile-git guile-avahi
> guile-json guile-semver guile-ssh guile-sqlite3
> guile-zlib guile-lzlib guile-zstd)))
>
> So here, a transformation could be applied. For example, build all
> the dependencies without testing them; at the 'define gnutls' (&co)
> step or in the map. Why not?

Sure, but how do you get a user-specified transformation to this line?

Ludo’.
Z
Z
zimoun wrote on 26 Feb 2021 22:44
(name . Ludovic Courtès)(address . ludo@gnu.org)
86czwm1p3y.fsf@gmail.com
Hi,

On Wed, 24 Feb 2021 at 18:21, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (6 lines)
>> So here, a transformation could be applied. For example, build all
>> the dependencies without testing them; at the 'define gnutls' (&co)
>> step or in the map. Why not?
>
> Sure, but how do you get a user-specified transformation to this line?

I do not know (yet?). :-)

First, we are only speaking about “guix time-machine” and second, only
about corner cases which could break in the future, as expiration.
Right?
We are not talking about general transformations applied to self (even
if it could be great).

So we could imagine something like ’set-build-from-command-line’ or
’set-build-option’ to tweak self only for the inferior. I have not
traced (yet) on the stack of calls from “guix time-machine” to self via
the inferiors, but somehow it looks like passing arguments.

Well, be able to tweak how to build Guix via the time-machine appears to
me better than configuring a build machine ahead of time. It is like
predicting the future and instead we should try to have a flexible
present in order to fix the past once in the unpredictable future. :-)


Cheers,
simon
L
L
Ludovic Courtès wrote on 18 Mar 2021 14:41
control message for bug #46650
(address . control@debbugs.gnu.org)
875z1opoit.fsf@gnu.org
retitle 46650 Making package transformation options available to 'guix pull'?
quit
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 46650@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 46650
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch