[PATCH] Parallelize `guix package`

Hi Guix!

Yesterday I had an interesting conversation on IRC about the behaviour

of multiple `guix package` processes running in parallel.

Specifically, when two transactions target the same profile (usually

/var/guix/profiles/per-user/$USER/guix-profile) at the same time, one

of them will fail to claim the lock and abort. 0001 makes it so that

the process waits for the lock. 0002 makes it so that packages

specified via -i can be built in parallel.

Regards,

Leo

Dec 17, 2019 8:19:14 AM Leo Prikler :

Can we extend this to include things like environment --ad-hoc?

Hi Leo,

(Cc: Julien, who worked on this part.)

Leo Prikler <leo.prikler@student.tugraz.at> skribis:

I actually like the current behavior, FWIW. Julien came up with this

locking mostly so that people do not inadvertently attempt to perform

several operations concurrently.

The key word here is “inadvertently”: IMO, there’s no reason to run

multiple ‘guix package’ on the same profile concurrently. With a

wait-for-lock policy, the result would be non-deterministic: you cannot

tell which one of the two processes will complete first.

WDYT?

Thanks,

Ludo’.

Am Dienstag, den 17.12.2019, 14:20 +0000 schrieb Brett Gilio:

`guix environment` does not claim any locks, so it does not suffer from

the problem that this patch tries to address. Perhaps my wording was

bad: By "can be built in parallel", I meant that if one starts two

processes, e.g. `guix install emacs` and `guix install ffmpeg`, emacs

and ffmpeg are built in parallel. This does not mean, that

dependencies of emacs are built in parallel – for that you'd have to

dig closer to the core.

Regards,

Leo

Dec 17, 2019 8:34:25 AM Leo Prikler :

Ah right. My mistake. I just woke up, so I think I need more coffee. :)

Brett Gilio

Hi Ludo’,

Am Dienstag, den 17.12.2019, 15:32 +0100 schrieb Ludovic Courtès:

Fair enough and that is an improvement over non-locking behaviour,

where one could spawn multiple profile generations from one, neither of

which is complete. Perhaps my attempt at doing this in a somewhat

controlled manner is equally harmful, but I will still try my best

arguing for it, as I believe it can make a positive impact.

I think the current policy is wait-for-lock deferred to the user. The

user has to let the first task complete before they can start the

second. In this setup, the user can simply launch the setup and trust,

that it will complete later while taking into account the changes the

first one has made.

Let's talk about three classes of operations – installations, removals

and upgrades – and their interactions. I will not take into account

roll-back, switch-generation and delete-generation, as it is

nonsensical to perform these in parallel to any other action. Perhaps

we could check for their presence first and acquire the lock with no-

wait semantics in that case.

- any operation on different packages: Either succeeds first and the

other builds on the profile it generates. As there is no collision in

the packages themselves, there will be no harm.

- install same package twice: Either succeeds first, the other will be

a no-op.

- install vs. remove same package: Non-deterministic, but why would you

do that?

- install vs. upgrade same package: Upgrade will be a no-op in either

case.

- remove vs. upgrade same package: Upgrade may inadvertently upgrade

the old package if it happens to come first, but in the final package

it will be removed either way.

Of course, any operation can also fail midway due to some step not

succeeding. In that case it would be as if one had issued the other

command right after that, which may perhaps not be what one wanted to

do (assuming I install package A, and some guide suggests to also build

related, but not dependency-connected package B, so I end up installing

B without A). However, such cases can easily be fixed by either

installing a fixed version of A later, using B on its own if it can be,

or rolling back.

Of course, both solutions are flawed in the way that they assume user

intent either way. Perhaps a better one would be to let the user

specify whether they want to wait or not through a command line

parameter, using the current behaviour as the default approach.

WDYT?

Regards,

Leo

Le 17 décembre 2019 16:19:34 GMT+01:00, Leo Prikler <leo.prikler@student.tugraz.at> a écrit :

I might be missing something. Guix install etc act on a "hidden" descripcion of the profile. Tgey take the current profile, modify it as specified (adding a package, renovinh another or upgrading some). When you run two guix package in parallel, they both work on the same profile, which creates unexpected results.

The expectation behind tge lock is that users will cancel tge ocher command and fix it before re-running it (e.g. instead of guix install foo & guix install bar, run guix install foo bar).

Am Dienstag, den 17.12.2019, 16:50 +0100 schrieb Julien Lepiller:

That's why the lock is claimed first. This way, the second process

acts on the profile that the first generated. I've tested this by

installing cowsay in parallel to lolcat, but it should work for bigger

packages in much the same way.

That is perhaps a reasonable expectation in most cases, but may be

annoying in others. Take any package with an absurdly long build time

(e.g. icecat) and then think "Oh, but I also wanted this" while it is

building. Now you have to either actively wait for icecat to complete

or stop it, add the other package and suffer the same build time again,

(whereas in the other way, you can wait for icecat to complete and

still launch a second process).

With the parallel builds of 0002, thing become even better, as you can

use bar even before foo is completed in case it manages to grab the

lock first. With the long build of icecat against a package with a

relatively short build, this could very well be the case and might end

up being a game changer. Of course, one could abuse ad-hoc

environments as well while waiting for the first process to finish, but

I don't think that's how people running into this problem expect to be

solving it (especially if they do want both foo and bar in their

profiles).

Regards,

Leo

Hi Leo,

Leo Prikler <leo.prikler@student.tugraz.at> skribis:

I cannot think of a useful behavior if wait-for-lock were implemented.

Really, as a user, you’d be unable to know what the end result is. I

don’t see that as very useful. :-)

What you describe above as potential mitigation is just that,

mitigation, and it could easily become complex (as a maintainer, I

woudn’t want to be responsible for this kind of complexity :-)), and

again, for very questionable “gains”.

Thoughts? Julien?

Thanks,

Ludo’.

close 38649

thanks

Am Mittwoch, den 18.12.2019, 15:37 +0100 schrieb Ludovic Courtès:

It took a while, but I feel you've convinced me.