Constructing hg-fetch fixed-output derivation requires Mercurial

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Simon Tournier
Owner
unassigned
Submitted by
Simon Tournier
Severity
normal
S
S
Simon Tournier wrote on 10 Apr 18:18 +0200
(address . bug-guix@gnu.org)
87h6g9w5rt.fsf@gmail.com
Hi,

For instance,

Toggle snippet (11 lines)
$ guix build -S -d hg-commitsigs
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
3,7 MB will be downloaded:
/gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
mercurial-6.2.2 3.5MiB 529KiB/s 00:07 ???????????????????? 100.0%

/gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv

and it is unexpected. The construction of the fixed-output derivation
does not need to download stuff; it only needs to compose stuff
detailing how to do. Any download (or build) must happen when running
the derivation itself.

The issue: later – say 1 or 2 years from now – the command:

guix time-machone --commit=929ddec -- build -S -d hg-commitsigs

will start to build all what Mercurial needs (python etc.). If for some
reasons*, only one of Mercurial dependencies fails then we are doomed.

( Context: I am working on a fixed-output translator; rely on
current strategies for downloading and do not require all the past
stack just for downloading source code. )

I think it comes from this part:

Toggle snippet (6 lines)
(hg-fetch '#$(hg-reference-url ref)
'#$(hg-reference-changeset ref)
#$output
#:hg-command (string-append #+hg "/bin/hg")))

from ’hg-fetch’ in (guix hg-download). Here the #+hg is not required
because just before there is:

(set-path-environment-variable "PATH" '("bin")
(match '#+inputs
(((names dirs outputs ...) ...)
dirs)))

So relying on string "hg" should be enough; as it is done in
’git-fetch/in-band*’ for one example.

Do I miss something?


Cheers,
simon

*reasons of failures: See
L
L
Ludovic Courtès wrote on 12 Apr 11:30 +0200
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 70339@debbugs.gnu.org)
87r0fbexnr.fsf@gnu.org
Hello!

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (13 lines)
> $ guix build -S -d hg-commitsigs
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> 3,7 MB will be downloaded:
> /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
> substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
> downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
> mercurial-6.2.2 3.5MiB 529KiB/s 00:07 ???????????????????? 100.0%
>
> /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
>
>
> and it is unexpected.

That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
me. :-)

Toggle quote (15 lines)
> I think it comes from this part:
>
> (hg-fetch '#$(hg-reference-url ref)
> '#$(hg-reference-changeset ref)
> #$output
> #:hg-command (string-append #+hg "/bin/hg")))
>
> from ’hg-fetch’ in (guix hg-download). Here the #+hg is not required
> because just before there is:
>
> (set-path-environment-variable "PATH" '("bin")
> (match '#+inputs
> (((names dirs outputs ...) ...)
> dirs)))

Maybe, but one way or another, Mercurial is necessary.

Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
could create a different fixed-output derivation producing the same
result but without using Mercurial; typically, the builder of that
derivation would download from SWH.

Does that make sense?

HTH,
Ludo’.
S
S
Simon Tournier wrote on 12 Apr 14:42 +0200
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 70339@debbugs.gnu.org)
CAJ3okZ1BgEqBKGAH2p+kJtTmFsBPqGZUsJea3TPo5qgo6ROHBg@mail.gmail.com
Hi Ludo,

On Fri, 12 Apr 2024 at 11:30, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

Toggle quote (15 lines)
> > $ guix build -S -d hg-commitsigs
> > substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> > 3,7 MB will be downloaded:
> > /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
> > substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
> > downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
> > mercurial-6.2.2 3.5MiB 529KiB/s 00:07 ???????????????????? 100.0%
> >
> > /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
> >
> > and it is unexpected.
>
> That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
> me. :-)

There is a misunderstanding, I guess.

Running 'hg clone' requires to have a local copy of Mercurial, yes for sure. :-)

However, just ask what it will run (please note the dash d in guix
build -S -d hg-commitsigs) must not require to have a local copy of
Mercurial (binary). If you still think yes, why is it not the case
for fixed-output derivations relying on the old Git builder?

Toggle quote (9 lines)
> > I think it comes from this part:
> >
> > (hg-fetch '#$(hg-reference-url ref)
> > '#$(hg-reference-changeset ref)
> > #$output
> > #:hg-command (string-append #+hg "/bin/hg")))
> >
> > from ’hg-fetch’ in (guix hg-download). Here the #+hg is not required
> > because just before there is:
[...]
Toggle quote (2 lines)
> Maybe, but one way or another, Mercurial is necessary.

Again, I think it is a bug from #+hg instead of plain "hg". Having a
local copy of Mercurial (binary) must not be required to just display
the fixed-output derivation. For running this fixed-output
derivation, yes for sure.

Toggle quote (7 lines)
> Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
> could create a different fixed-output derivation producing the same
> result but without using Mercurial; typically, the builder of that
> derivation would download from SWH.
>
> Does that make sense?

Yes, it makes sense; see my very first attempt in [1] :-).

But you cannot apply this strategy for fixed-output derivations
relying on Mercurial. You need first to build Mercurial (and thus all
the Python stack) just to display the fixed-output derivation. Then,
yes once you have this fixed-output derivation, it is possible to
manipulate it for getting another one.

This report is about #+hg that needs to be fixed for the future.
And because of that, the strategy above for fixed-output derivations
relying on Mercurial is doomed for the past, IMHO. Except if you have
an idea. ;-)

Cheers,
simon


L
L
Ludovic Courtès wrote on 12 Apr 16:05 +0200
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 70339@debbugs.gnu.org)
87il0m65i4.fsf@inria.fr
Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (28 lines)
> Hi Ludo,
>
> On Fri, 12 Apr 2024 at 11:30, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> > $ guix build -S -d hg-commitsigs
>> > substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>> > 3,7 MB will be downloaded:
>> > /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
>> > substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
>> > downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
>> > mercurial-6.2.2 3.5MiB 529KiB/s 00:07 ???????????????????? 100.0%
>> >
>> > /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
>> >
>> > and it is unexpected.
>>
>> That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
>> me. :-)
>
> There is a misunderstanding, I guess.
>
> Running 'hg clone' requires to have a local copy of Mercurial, yes for sure. :-)
>
> However, just ask what it will run (please note the dash d in guix
> build -S -d hg-commitsigs) must not require to have a local copy of
> Mercurial (binary). If you still think yes, why is it not the case
> for fixed-output derivations relying on the old Git builder?

Oh sorry, I had missed the ‘-d’ bit.

In this case, what’s happening is grafts: Guix downloads (or builds)
Mercurial so it can compute its grafted derivation.

Toggle quote (9 lines)
>> Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
>> could create a different fixed-output derivation producing the same
>> result but without using Mercurial; typically, the builder of that
>> derivation would download from SWH.
>>
>> Does that make sense?
>
> Yes, it makes sense; see my very first attempt in [1] :-).

[...]

Toggle quote (2 lines)
Nice!

Thanks,
Ludo’.
S
S
Simon Tournier wrote on 22 Apr 02:22 +0200
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 70339-done@debbugs.gnu.org)
87bk626yc7.fsf@gmail.com
Hi,

On ven., 12 avril 2024 at 16:05, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

Toggle quote (3 lines)
> In this case, what’s happening is grafts: Guix downloads (or builds)
> Mercurial so it can compute its grafted derivation.

Ah indeed! Damned. :-)

Closing.

Cheers,
simon
Closed
?