Constructing hg-fetch fixed-output derivation requires Mercurial

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Simon Tournier
Owner
unassigned
Submitted by
Simon Tournier
Severity
normal

Debbugs page

Simon Tournier wrote 11 months ago
(address . bug-guix@gnu.org)
87h6g9w5rt.fsf@gmail.com
Hi,

For instance,

Toggle snippet (11 lines)
$ guix build -S -d hg-commitsigs
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
3,7 MB will be downloaded:
/gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
mercurial-6.2.2 3.5MiB 529KiB/s 00:07 ▕██████████████████▏ 100.0%

/gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv

and it is unexpected. The construction of the fixed-output derivation
does not need to download stuff; it only needs to compose stuff
detailing how to do. Any download (or build) must happen when running
the derivation itself.

The issue: later – say 1 or 2 years from now – the command:

guix time-machone --commit=929ddec -- build -S -d hg-commitsigs

will start to build all what Mercurial needs (python etc.). If for some
reasons*, only one of Mercurial dependencies fails then we are doomed.

( Context: I am working on a fixed-output translator; rely on
current strategies for downloading and do not require all the past
stack just for downloading source code. )

I think it comes from this part:

Toggle snippet (6 lines)
(hg-fetch '#$(hg-reference-url ref)
'#$(hg-reference-changeset ref)
#$output
#:hg-command (string-append #+hg "/bin/hg")))

from ’hg-fetch’ in (guix hg-download). Here the #+hg is not required
because just before there is:

(set-path-environment-variable "PATH" '("bin")
(match '#+inputs
(((names dirs outputs ...) ...)
dirs)))

So relying on string "hg" should be enough; as it is done in
’git-fetch/in-band*’ for one example.

Do I miss something?


Cheers,
simon

*reasons of failures: See
Ludovic Courtès wrote 11 months ago
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 70339@debbugs.gnu.org)
87r0fbexnr.fsf@gnu.org
Hello!

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (13 lines)
> $ guix build -S -d hg-commitsigs
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> 3,7 MB will be downloaded:
> /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
> substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
> downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
> mercurial-6.2.2 3.5MiB 529KiB/s 00:07 ▕██████████████████▏ 100.0%
>
> /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
>
>
> and it is unexpected.

That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
me. :-)

Toggle quote (15 lines)
> I think it comes from this part:
>
> (hg-fetch '#$(hg-reference-url ref)
> '#$(hg-reference-changeset ref)
> #$output
> #:hg-command (string-append #+hg "/bin/hg")))
>
> from ’hg-fetch’ in (guix hg-download). Here the #+hg is not required
> because just before there is:
>
> (set-path-environment-variable "PATH" '("bin")
> (match '#+inputs
> (((names dirs outputs ...) ...)
> dirs)))

Maybe, but one way or another, Mercurial is necessary.

Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
could create a different fixed-output derivation producing the same
result but without using Mercurial; typically, the builder of that
derivation would download from SWH.

Does that make sense?

HTH,
Ludo’.
Simon Tournier wrote 11 months ago
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 70339@debbugs.gnu.org)
CAJ3okZ1BgEqBKGAH2p+kJtTmFsBPqGZUsJea3TPo5qgo6ROHBg@mail.gmail.com
Hi Ludo,

On Fri, 12 Apr 2024 at 11:30, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

Toggle quote (15 lines)
> > $ guix build -S -d hg-commitsigs
> > substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> > 3,7 MB will be downloaded:
> > /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
> > substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
> > downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
> > mercurial-6.2.2 3.5MiB 529KiB/s 00:07 ▕██████████████████▏ 100.0%
> >
> > /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
> >
> > and it is unexpected.
>
> That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
> me. :-)

There is a misunderstanding, I guess.

Running 'hg clone' requires to have a local copy of Mercurial, yes for sure. :-)

However, just ask what it will run (please note the dash d in guix
build -S -d hg-commitsigs) must not require to have a local copy of
Mercurial (binary). If you still think yes, why is it not the case
for fixed-output derivations relying on the old Git builder?

Toggle quote (9 lines)
> > I think it comes from this part:
> >
> > (hg-fetch '#$(hg-reference-url ref)
> > '#$(hg-reference-changeset ref)
> > #$output
> > #:hg-command (string-append #+hg "/bin/hg")))
> >
> > from ’hg-fetch’ in (guix hg-download). Here the #+hg is not required
> > because just before there is:
[...]
Toggle quote (2 lines)
> Maybe, but one way or another, Mercurial is necessary.

Again, I think it is a bug from #+hg instead of plain "hg". Having a
local copy of Mercurial (binary) must not be required to just display
the fixed-output derivation. For running this fixed-output
derivation, yes for sure.

Toggle quote (7 lines)
> Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
> could create a different fixed-output derivation producing the same
> result but without using Mercurial; typically, the builder of that
> derivation would download from SWH.
>
> Does that make sense?

Yes, it makes sense; see my very first attempt in [1] :-).

But you cannot apply this strategy for fixed-output derivations
relying on Mercurial. You need first to build Mercurial (and thus all
the Python stack) just to display the fixed-output derivation. Then,
yes once you have this fixed-output derivation, it is possible to
manipulate it for getting another one.

This report is about #+hg that needs to be fixed for the future.
And because of that, the strategy above for fixed-output derivations
relying on Mercurial is doomed for the past, IMHO. Except if you have
an idea. ;-)

Cheers,
simon


Ludovic Courtès wrote 11 months ago
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)(address . 70339@debbugs.gnu.org)
87il0m65i4.fsf@inria.fr
Simon Tournier <zimon.toutoune@gmail.com> skribis:

Toggle quote (28 lines)
> Hi Ludo,
>
> On Fri, 12 Apr 2024 at 11:30, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:
>
>> > $ guix build -S -d hg-commitsigs
>> > substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>> > 3,7 MB will be downloaded:
>> > /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
>> > substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
>> > downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
>> > mercurial-6.2.2 3.5MiB 529KiB/s 00:07 ▕██████████████████▏ 100.0%
>> >
>> > /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
>> >
>> > and it is unexpected.
>>
>> That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
>> me. :-)
>
> There is a misunderstanding, I guess.
>
> Running 'hg clone' requires to have a local copy of Mercurial, yes for sure. :-)
>
> However, just ask what it will run (please note the dash d in guix
> build -S -d hg-commitsigs) must not require to have a local copy of
> Mercurial (binary). If you still think yes, why is it not the case
> for fixed-output derivations relying on the old Git builder?

Oh sorry, I had missed the ‘-d’ bit.

In this case, what’s happening is grafts: Guix downloads (or builds)
Mercurial so it can compute its grafted derivation.

Toggle quote (9 lines)
>> Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
>> could create a different fixed-output derivation producing the same
>> result but without using Mercurial; typically, the builder of that
>> derivation would download from SWH.
>>
>> Does that make sense?
>
> Yes, it makes sense; see my very first attempt in [1] :-).

[...]

Toggle quote (2 lines)
Nice!

Thanks,
Ludo’.
Simon Tournier wrote 11 months ago
(name . Ludovic Courtès)(address . ludovic.courtes@inria.fr)(address . 70339-done@debbugs.gnu.org)
87bk626yc7.fsf@gmail.com
Hi,

On ven., 12 avril 2024 at 16:05, Ludovic Courtès <ludovic.courtes@inria.fr> wrote:

Toggle quote (3 lines)
> In this case, what’s happening is grafts: Guix downloads (or builds)
> Mercurial so it can compute its grafted derivation.

Ah indeed! Damned. :-)

Closing.

Cheers,
simon
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 70339@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 70339
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch
You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags
mumi command -t +confirmed -t +easy
Or, remove the moreinfo tag and set the help tag
mumi command -t -moreinfo -t +help