Request for merging "core-updates" branch

Open

Details

9 participants

Efraim Flashner
Guillaume Le Vaillant
???
Lars-Dominik Braun
Ludovic Courtès
Christopher Baines
Maxim Cournoyer
Steve George
tumashu

Owner: unassigned

Submitted by: Steve George

Severity: normal

Blocked by

Steve George wrote on 18 Apr 16:56 +0200

Request for merging core-updates branch

Recipients:(address . bug-guix@gnu.org)

Message-ID:ZiE0qcjXe5H_3XLT@dragon2

Let's see where we are with the branch currently.

Thanks,

Steve / Futurile

Christopher Baines wrote on 18 Apr 20:51 +0200

(no subject)

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87ttjy4i8l.fsf@cbaines.net

retitle 70456 Request for merging "core-updates" branch

thanks

Christopher Baines wrote on 19 Apr 16:42 +0200

Re: Request for merging "core-updates" branch

Recipients:

Message-ID:87il0d4dn0.fsf@cbaines.net

Hey,

Thanks for raising this issue Steve, given the branch has been going for

around 9 months (since [1]) now, I think it's well overdue to start

looking at building and merging it.

1: https://lists.gnu.org/archive/html/guix-commits/2023-07/msg00332.html

I pushed a single commit plus a merge from master today, and that was

pretty difficult. There was plenty of conflicts, and I probably have

resolved some wrongly, and there's potentially some things that Git

didn't raise as conflicts but might have broken with merging in master.

I'm also really confused by what commits appear to be on the branch,

take 12b15585a75062f3fba09d82861c6fae9a7743b2 which appears to be one

core-updates, but it's a duplicate of

e2a7c227dea5b361e2ebdbba24b923d1922a79d0 which was pushed to

master. Same with this commit 28d14130953d868d4848540d9de8e1ae4a01a467,

which is different to f29f80c194d0c534a92354b2bc19022a9b70ecf8 on

master.

Putting aside the functional changes on core-updates, it's doesn't seem

good to merge these seemingly duplicate commits on to master. I'm not

sure how this happened though, or how to fix it.

Chris

Christopher Baines wrote on 19 Apr 19:00 +0200

Re: bug#70456: Request for merging "core-updates" branch

Recipients:(address . 70456@debbugs.gnu.org)

Message-ID:877cgt47a1.fsf@cbaines.net

Christopher Baines <mail@cbaines.net> writes:

Toggle quote (8 lines)

> I'm also really confused by what commits appear to be on the branch,

> take 12b15585a75062f3fba09d82861c6fae9a7743b2 which appears to be one

> core-updates, but it's a duplicate of

> e2a7c227dea5b361e2ebdbba24b923d1922a79d0 which was pushed to

> master. Same with this commit 28d14130953d868d4848540d9de8e1ae4a01a467,

> which is different to f29f80c194d0c534a92354b2bc19022a9b70ecf8 on

> master.

I've worked out at least when these two werid commits turned up on

core-updates.

12b15585a7 is mentioned here:

https://lists.gnu.org/archive/html/guix-commits/2023-09/msg00955.html

and 28d1413095 is mentioned here:

https://lists.gnu.org/archive/html/guix-commits/2024-03/msg00381.html

With the changes last month in March, I was going to suggest deleting

the branch and then re-creating from f205179ed2 and trying to re-apply

the changes that should be on core-updates, while avoiding any

"duplicate" commits. However, I'm not even sure where to being with the

~5000 commits pushed in September, at least one of them is a duplicate

of a commit on master, but I'm not sure how many of the other ~5000 are.

For comparison, I did a merge of master in to core-updates today, and

this is what it shows up like on guix-commits:

https://lists.gnu.org/archive/html/guix-commits/2024-04/msg01209.html

There are only two new revisions, the ed update I pushed, and the merge

commit, which is what a merge should look like as far as I'm aware.

Christopher Baines wrote on 20 Apr 13:14 +0200

Re: Status of ‘core-updates’

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:87mspo2sme.fsf@cbaines.net

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (7 lines)

> What’s the status of ‘core-updates’? What are the areas where help is

> needed?

> I know a lot has happened since the last update¹, which is roughly when

> I dropped the ball due to other commitments, but I’m not sure where we

> are now.

I haven't really been following core-updates, but I have had a look

since there's a request to merge it now [1].

I'm really concerned by the commits on the branch though, assuming I'm

using Git right, there are 6351 commits on the branch:

git log --pretty=oneline core-updates ^master | wc -l

Somehow, I think there's been a couple of pushes of commits to

core-updates that have partially duplicated lots of commits from master,

I put some more details in:

https://issues.guix.gnu.org/70456#3

I think keeping the Git commit history clean and representative is

really important, so to me at least this means core-updates can't be

merged to master in it's current form, even if the changes overall from

these 6351 commits are reasonable.

I'm really not sure how to move forward though, I had a go at trying to

rebuild the branch without introducing the thousands of duplicate

commits and that produced a branch with 765 commits over master, which

still seems a lot, but a big improvement over 6351:

https://git.cbaines.net/guix/log/?h=chris-core-updates-no-duplicates-attempt

That was really hard going though, as there's plenty of merge conflicts

along the way, and I'm pretty sure I solved some of them

incorrectly. The resulting branch also differs from core-updates.

Maybe someone with more time, care and attention could do a better job,

but it might be more worthwhile just starting fresh and rather than

trying to produce a like for like branch just without the thousands of

duplicate commits, effectively manually rebase the branch (without the

duplicate commits) on master and try to get the commits in to a usable

state.

Any ideas?

Thanks,

Chris

Maxim Cournoyer wrote on 20 Apr 18:16 +0200

Re: bug#70456: Request for merging "core-updates" branch

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:87bk64j9h8.fsf@gmail.com

Hi,

Christopher Baines <mail@cbaines.net> writes:

Toggle quote (35 lines)> Christopher Baines <mail@cbaines.net> writes:
>
>> I'm also really confused by what commits appear to be on the branch,
>> take 12b15585a75062f3fba09d82861c6fae9a7743b2 which appears to be one
>> core-updates, but it's a duplicate of
>> e2a7c227dea5b361e2ebdbba24b923d1922a79d0 which was pushed to
>> master. Same with this commit 28d14130953d868d4848540d9de8e1ae4a01a467,
>> which is different to f29f80c194d0c534a92354b2bc19022a9b70ecf8 on
>> master.
>
> I've worked out at least when these two werid commits turned up on
> core-updates.
>
> 12b15585a7 is mentioned here:
>   https://lists.gnu.org/archive/html/guix-commits/2023-09/msg00955.html
>
> and 28d1413095 is mentioned here:
>   https://lists.gnu.org/archive/html/guix-commits/2024-03/msg00381.html
>
>
> With the changes last month in March, I was going to suggest deleting
> the branch and then re-creating from f205179ed2 and trying to re-apply
> the changes that should be on core-updates, while avoiding any
> "duplicate" commits. However, I'm not even sure where to being with the
> ~5000 commits pushed in September, at least one of them is a duplicate
> of a commit on master, but I'm not sure how many of the other ~5000 are.
>
> For comparison, I did a merge of master in to core-updates today, and
> this is what it shows up like on guix-commits:
>
>   https://lists.gnu.org/archive/html/guix-commits/2024-04/msg01209.html
>
> There are only two new revisions, the ed update I pushed, and the merge
> commit, which is what a merge should look like as far as I'm aware.

I think probably what happened is that in the middle of a merge of
master -> core-updates (which entails sometimes painful conflicts
resolution), a new commit pushed to core-updates, and to be able to push
the resulting local branch (including the thousands of commits from the
merge commit) got rebased on the remote core-updates.

Perhaps another merge commit appeared on the remote around the same
time, which would explain the duplicates.

While I agree it's messy to have 5000 of duplicated commits, I'm not
sure attempting to rewrite the branch, which has seen a lot of original
commits, is a good idea (it'd be easy to have some good commits fall
into cracks, leading to lost of work).

I'd rather we take this experience as a strong reminding that rebasing
merge commits should be avoided at all costs (git already issues a
warning, IIRC).  As you suggested, the next time a situation like this
happens (locally prepared merge commit with new commits made to the
remote branch), merging the remote into the local branch is probably a
nicer solution.

-- 
Thanks,
Maxim

Christopher Baines wrote on 20 Apr 20:08 +0200

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)

Message-ID:87h6fv3o0t.fsf@cbaines.net

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (53 lines)> Hi,
>
> Christopher Baines <mail@cbaines.net> writes:
>
>> Christopher Baines <mail@cbaines.net> writes:
>>
>>> I'm also really confused by what commits appear to be on the branch,
>>> take 12b15585a75062f3fba09d82861c6fae9a7743b2 which appears to be one
>>> core-updates, but it's a duplicate of
>>> e2a7c227dea5b361e2ebdbba24b923d1922a79d0 which was pushed to
>>> master. Same with this commit 28d14130953d868d4848540d9de8e1ae4a01a467,
>>> which is different to f29f80c194d0c534a92354b2bc19022a9b70ecf8 on
>>> master.
>>
>> I've worked out at least when these two werid commits turned up on
>> core-updates.
>>
>> 12b15585a7 is mentioned here:
>>   https://lists.gnu.org/archive/html/guix-commits/2023-09/msg00955.html
>>
>> and 28d1413095 is mentioned here:
>>   https://lists.gnu.org/archive/html/guix-commits/2024-03/msg00381.html
>>
>>
>> With the changes last month in March, I was going to suggest deleting
>> the branch and then re-creating from f205179ed2 and trying to re-apply
>> the changes that should be on core-updates, while avoiding any
>> "duplicate" commits. However, I'm not even sure where to being with the
>> ~5000 commits pushed in September, at least one of them is a duplicate
>> of a commit on master, but I'm not sure how many of the other ~5000 are.
>>
>> For comparison, I did a merge of master in to core-updates today, and
>> this is what it shows up like on guix-commits:
>>
>>   https://lists.gnu.org/archive/html/guix-commits/2024-04/msg01209.html
>>
>> There are only two new revisions, the ed update I pushed, and the merge
>> commit, which is what a merge should look like as far as I'm aware.
>
> I think probably what happened is that in the middle of a merge of
> master -> core-updates (which entails sometimes painful conflicts
> resolution), a new commit pushed to core-updates, and to be able to push
> the resulting local branch (including the thousands of commits from the
> merge commit) got rebased on the remote core-updates.
>
> Perhaps another merge commit appeared on the remote around the same
> time, which would explain the duplicates.
>
> While I agree it's messy to have 5000 of duplicated commits, I'm not
> sure attempting to rewrite the branch, which has seen a lot of original
> commits, is a good idea (it'd be easy to have some good commits fall
> into cracks, leading to lost of work).

I think it's important to weigh up the cost and risks associated with
either merging these commits, or somehow avoiding doing so. I think the
potential impact is more than just a bit of messy Git history.

Assuming we merge core-updates without doing anything about these
duplicate commits, and taking the cwltool package as a semi-random
example, if you do:

  git log -p gnu/packages/bioinformatics.scm

You're going to see two commits for the update to 3.1.20240112164112,
that's maybe confusing, but not a big issue I guess since they look the
same, just different hashes.

But say you're looking at the Git history because you want that specific
version of cwltool and you're going to use guix time-machine or an
inferior looking at that revision. Well, it's a lucky dip. If you pick
the original master commit, you're in luck, you'll probably get
substitutes for cwltool. But if you pick the other seemingly identical
commit, you're effectively checking out core-updates as it was last
month and the chance of substitutes is much less likely. I also can't
really think how you'd work out which commit is best to use once
core-updates is merged? The easiest way would probably be to check the
signature, but that will only work most of the time.

This isn't a new issue, it's already problematic for substitute
availability to use intermediate commits (commits that weren't directly
pointed to by master). But there are over 1000 packages who's versions
are being changed on core-updates currently, or at least it looks like
this because of the duplicate commits, and if I'm correct about how
people are using the git history to find commits for specific versions
of packages, then having these duplicates in the Git history for master
forever more is going to catch people out for as long as those versions
remain relevant.

Maxim Cournoyer wrote on 20 Apr 23:15 +0200

Re: Status of ‘core-updates’

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:87mspnivms.fsf@gmail.com

Hi Christopher,

Christopher Baines <mail@cbaines.net> writes:

Toggle quote (39 lines)> Ludovic Courtès <ludo@gnu.org> writes:
>
>> What’s the status of ‘core-updates’?  What are the areas where help is
>> needed?
>>
>> I know a lot has happened since the last update¹, which is roughly when
>> I dropped the ball due to other commitments, but I’m not sure where we
>> are now.
>
> I haven't really been following core-updates, but I have had a look
> since there's a request to merge it now [1].
>
> I'm really concerned by the commits on the branch though, assuming I'm
> using Git right, there are 6351 commits on the branch:
>
>   git log --pretty=oneline core-updates ^master | wc -l
>
> Somehow, I think there's been a couple of pushes of commits to
> core-updates that have partially duplicated lots of commits from master,
> I put some more details in:
>
>   https://issues.guix.gnu.org/70456#3
>
> I think keeping the Git commit history clean and representative is
> really important, so to me at least this means core-updates can't be
> merged to master in it's current form, even if the changes overall from
> these 6351 commits are reasonable.
>
> I'm really not sure how to move forward though, I had a go at trying to
> rebuild the branch without introducing the thousands of duplicate
> commits and that produced a branch with 765 commits over master, which
> still seems a lot, but a big improvement over 6351:
>
>   https://git.cbaines.net/guix/log/?h=chris-core-updates-no-duplicates-attempt
>
> That was really hard going though, as there's plenty of merge conflicts
> along the way, and I'm pretty sure I solved some of them
> incorrectly. The resulting branch also differs from core-updates.

I also think Git commit history is important, but in this case I weigh

the value of removing ~5000 duplicated rust commits against the risks of

resolving merge conflicts wrong or forgetting commits upon attempting to

recreate the branch from scratch lower than the benefit.

Toggle quote (7 lines)

> Maybe someone with more time, care and attention could do a better job,

> but it might be more worthwhile just starting fresh and rather than

> trying to produce a like for like branch just without the thousands of

> duplicate commits, effectively manually rebase the branch (without the

> duplicate commits) on master and try to get the commits in to a usable

> state.

Given the little attention core-updates is currently receiving, I doubt
someone is willing to put the effort to recreate the branch from scratch
to clean its git history; at least speaking for myself I'd rather spend
the little hack time I have to work on it toward getting it finalized.

I believe how these duplicates came to exist was probably two separate
master -> core-updates merge commits, with one of them ending up being
rebased on top of the other, probably so that it could be pushed.
Perhaps we could capture in our contribution guidelines that rebasing a
merge commit should never be done to keep the history clean, and that in
a situation where:

1. a merge has been prepared locally (with conflicts resolved and all)
2. a new commit has appeared on the remote branch

the solution should be to merge the remote branch into the local one
instead of rebasing the local one on the remote one (as is usually
done).  Disclaimer: I haven't actually tried this suggested approach,
which should be done before documenting it, if there's a consensus to do
so.

In other words, I suggest we document what *not* to do to avoid
repeating the same mistake in the future, and move on.

-- 
Thanks,
Maxim

Steve George wrote on 22 Apr 11:39 +0200

block 70456 with 67973

Recipients:(address . control@debbugs.gnu.org)

Message-ID:1713778748-3175-bts-steve@futurile.net

block 70456 with 67973

thanks

Steve George wrote on 22 Apr 11:41 +0200

block 70456 with 45885

Recipients:(address . control@debbugs.gnu.org)

Message-ID:1713778908-2675-bts-steve@futurile.net

block 70456 with 45885

thanks

Steve George wrote on 22 Apr 11:48 +0200

block 70456 with 40316

Recipients:(address . control@debbugs.gnu.org)

Message-ID:1713779317-2200-bts-steve@futurile.net

block 70456 with 40316

thanks

Steve George wrote on 22 Apr 11:51 +0200

block 70456 with 68270

Recipients:(address . control@debbugs.gnu.org)

Message-ID:1713779459-1387-bts-steve@futurile.net

block 70456 with 68270

thanks

Maxim Cournoyer wrote on 22 Apr 19:31 +0200

Re: bug#70456: Request for merging "core-updates" branch

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:87sezdgv8a.fsf@gmail.com

Hi Christopher,

Christopher Baines <mail@cbaines.net> writes:

[...]

Toggle quote (6 lines)

> Assuming we merge core-updates without doing anything about these

> duplicate commits, and taking the cwltool package as a semi-random

> example, if you do:

> git log -p gnu/packages/bioinformatics.scm

I trust the 'newest' (appearing first in 'git log --grep='cwltool:

Update') would yield the commit having substitutes?

If so, the inconvenience is somewhat mitigated, as long as you know to

use the newest of duplicated commits.

Thanks,

Maxim

Steve George wrote on 23 Apr 15:07 +0200

block 70456 with 46442

Recipients:(address . control@debbugs.gnu.org)

Message-ID:1713877622-2297-bts-steve@futurile.net

block 70456 with 46442

thanks

Steve George wrote on 23 Apr 17:23 +0200

block 70456 with 70537

Recipients:(address . control@debbugs.gnu.org)

Message-ID:1713885799-1346-bts-steve@futurile.net

block 70456 with 70537

thanks

Steve George wrote on 23 Apr 18:32 +0200

block 70456 with 39415

Recipients:(address . control@debbugs.gnu.org)

Message-ID:1713889973-3525-bts-steve@futurile.net

block 70456 with 39415

thanks

tumashu wrote on 26 Apr 16:44 +0200

Request for merging "core-updates" branch

Recipients:(address . 70456@debbugs.gnu.org)

Message-ID:24c1b4b1.25ee.18f1add5df8.Coremail.tumashu@163.com

emacs has a script gitmerge.el, it can skip some commit when merge with different merge rule (ours), maybe can make life easier:

https://git.savannah.gnu.org/cgit/emacs.git/tree/admin/gitmerge.el

https://git.savannah.gnu.org/cgit/emacs.git/tree/admin/notes/git-workflow

?????????????

Attachment: file

Ludovic Courtès wrote on 2 May 09:53 +0200

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:8734r03b11.fsf@inria.fr

Hi Chris and all,

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (16 lines)> I think keeping the Git commit history clean and representative is
> really important, so to me at least this means core-updates can't be
> merged to master in it's current form, even if the changes overall from
> these 6351 commits are reasonable.
>
> I'm really not sure how to move forward though, I had a go at trying to
> rebuild the branch without introducing the thousands of duplicate
> commits and that produced a branch with 765 commits over master, which
> still seems a lot, but a big improvement over 6351:
>
>   https://git.cbaines.net/guix/log/?h=chris-core-updates-no-duplicates-attempt
>
> That was really hard going though, as there's plenty of merge conflicts
> along the way, and I'm pretty sure I solved some of them
> incorrectly. The resulting branch also differs from core-updates.

Woow, impressive. How did you go about finding which commits were

duplicates/cherry-picked from master? Which commit did you start from?

Given everything you’ve explained, it seems to me it’s worth trying to

start from a clean branch like this.

I checked it out (commit da77ea23daa0bfa4a73290dff99b22d6825ff80b) to

get an idea of where we are and got this:

Toggle snippet (4 lines)

make[2]: *** No rule to make target 'gnu/packages/patches/glib-networking-gnutls-binding.patch', needed by 'all-am'.

make[2]: *** No rule to make target 'gnu/packages/patches/librecad-support-for-boost-1.76.patch', needed by 'all-am'.

It stopped at:

Toggle snippet (3 lines)

gnu/packages/sdl.scm:72:2: error: (package (name "sdl2") (version "2.30.1") (source (origin (method url-fetch) (uri (string-append "https://libsdl.org/release/SDL2-" version ".tar.gz")) (sha256 (base32 "0fj7gxc7rlzzrafnx9nmf7ws3paxy583fmx7bcbavi6gr3xmy881")))) (arguments (list #:tests? #f #:configure-flags (gexp (append (quote ("--disable-wayland-shared" "--enable-video-kmsdrm" "--disable-kmsdrm-shared")) (quote ("--disable-alsa-shared" "--disable-pulseaudio-shared" "--disable-x11-shared" "LDFLAGS=-lGL")))) #:make-flags (gexp (cons* (string-append "LDFLAGS=-Wl,-rpath," (ungexp (this-package-input "eudev")) "/lib" ",-rpath," (ungexp (this-package-input "vulkan-loader")) "/lib") (quote ("V=1")))))) (propagated-inputs (list libx11 libcap mesa)) (native-inputs (list pkg-config)) (inputs (list libxrandr glu alsa-lib pulseaudio dbus eudev glib ibus-minimal libxkbcommon libxcursor vulkan-loader wayland wayland-protocols)) (outputs (quote ("out" "debug"))) (synopsis "Cross platform game development library") (description "Simple DirectMedia Layer is a cross-platform development library designed to\nprovide low level access to audio, keyboard, mouse, joystick, and graphics\nhardware.") (home-page "https://libsdl.org/") (license license:bsd-3)): missing field initializers (build-system)

I guess these are merge conflicts that weren’t correctly resolved.

This branch rewrites the entire ‘core-updates’ history. What about

rewriting starting from the first series of “duplicate” commits? That

should solve the immediate issue while keeping the “known good” history?

Thanks,

Ludo’.

Christopher Baines wrote on 8 May 14:03 +0200

Process gnome-team before core-updates

Recipients:(address . control@debbugs.gnu.org)

Message-ID:87pltwfr3i.fsf@cbaines.net

block 70456 by 70766

thanks

I think being able to merge core-updates is still a few weeks away, so I

think there's time to build and merge gnome-team without delaying

core-updates.

If it does become a problem, we can always switch approach and wait

until after core-updates is merged to look at gnome-team.

Efraim Flashner wrote on 9 Jun 11:54 +0200

Re: branch core-updates updated (c8c6883398 -> 0e06c9697a)

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:ZmV706i6uTb69V1Q@3900XT

On Sun, Jun 09, 2024 at 10:46:10AM +0100, Christopher Baines wrote:

Toggle quote (29 lines)> guix-commits@gnu.org writes:
> 
> > efraim pushed a change to branch core-updates
> > in repository guix.
> >
> >     from c8c6883398 gnu: dico: Add libxcrypt dependency.
> >      new 9804f8c149 gnu: coeurl: Update to 0.3.1.
> >      new 51c7b6d76f gnu: font-gnu-freefont: Build with newer fontforge.
> >      new 0e06c9697a gnu: Remove fontforge-20190801.
> >
> > The 3 revisions listed above as "new" are entirely new to this
> > repository and will be described in separate emails.  The revisions
> > listed as "add" were already present in the repository and have only
> > been added to this reference.
> 
> These changes confused me as I was looking at the trying to work out why
> they needed to be pushed to core-updates. Eventually I figured out that
> Git is right, these commits are entirely new, but they duplicate
> existing commits already pushed to master (e.g. 0e06c9697a is a
> duplicate of 3d5f4b2d7dda).
> 
> I know the new guidance says to "Avoid merging master in to the branch",
> but one of the reasons for that is to avoid situations just like this
> where merges are done incorrectly and commits are duplicated between
> branches.
> 
> To fix this, I think we should rebase core-updates on master and drop
> these commits.

I wasn't aware I was "doing it wrong" with this, I saw that coeurl 0.3.0
failed on core-updates but a bump to 0.3.1 fixed the build, and it could
go to master also. Similar story with working to remove
fontforge-20190801 which no longer built on core-updates. I figured that
applying the patch to both branches would make it easier to merge master
into core-updates since there would be less drift between the two.

What is the correct way to apply a patch to multiple branches?

-- 
Efraim Flashner   <efraim@flashner.co.il>   ????? ?????
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

Christopher Baines wrote on 9 Jun 11:46 +0200

Recipients:

Message-ID:87jziy1mal.fsf@cbaines.net

guix-commits@gnu.org writes:

Toggle quote (13 lines)> efraim pushed a change to branch core-updates
> in repository guix.
>
>     from c8c6883398 gnu: dico: Add libxcrypt dependency.
>      new 9804f8c149 gnu: coeurl: Update to 0.3.1.
>      new 51c7b6d76f gnu: font-gnu-freefont: Build with newer fontforge.
>      new 0e06c9697a gnu: Remove fontforge-20190801.
>
> The 3 revisions listed above as "new" are entirely new to this
> repository and will be described in separate emails.  The revisions
> listed as "add" were already present in the repository and have only
> been added to this reference.

These changes confused me as I was looking at the trying to work out why
they needed to be pushed to core-updates. Eventually I figured out that
Git is right, these commits are entirely new, but they duplicate
existing commits already pushed to master (e.g. 0e06c9697a is a
duplicate of 3d5f4b2d7dda).

I know the new guidance says to "Avoid merging master in to the branch",
but one of the reasons for that is to avoid situations just like this
where merges are done incorrectly and commits are duplicated between
branches.

To fix this, I think we should rebase core-updates on master and drop
these commits.

Christopher Baines wrote on 9 Jun 12:20 +0200

Recipients:(address . 70456@debbugs.gnu.org)

Message-ID:87frtm1koq.fsf@cbaines.net

Efraim Flashner <efraim@flashner.co.il> writes:

Toggle quote (37 lines)> On Sun, Jun 09, 2024 at 10:46:10AM +0100, Christopher Baines wrote:
>> guix-commits@gnu.org writes:
>> 
>> > efraim pushed a change to branch core-updates
>> > in repository guix.
>> >
>> >     from c8c6883398 gnu: dico: Add libxcrypt dependency.
>> >      new 9804f8c149 gnu: coeurl: Update to 0.3.1.
>> >      new 51c7b6d76f gnu: font-gnu-freefont: Build with newer fontforge.
>> >      new 0e06c9697a gnu: Remove fontforge-20190801.
>> >
>> > The 3 revisions listed above as "new" are entirely new to this
>> > repository and will be described in separate emails.  The revisions
>> > listed as "add" were already present in the repository and have only
>> > been added to this reference.
>> 
>> These changes confused me as I was looking at the trying to work out why
>> they needed to be pushed to core-updates. Eventually I figured out that
>> Git is right, these commits are entirely new, but they duplicate
>> existing commits already pushed to master (e.g. 0e06c9697a is a
>> duplicate of 3d5f4b2d7dda).
>> 
>> I know the new guidance says to "Avoid merging master in to the branch",
>> but one of the reasons for that is to avoid situations just like this
>> where merges are done incorrectly and commits are duplicated between
>> branches.
>> 
>> To fix this, I think we should rebase core-updates on master and drop
>> these commits.
>
> I wasn't aware I was "doing it wrong" with this, I saw that coeurl 0.3.0
> failed on core-updates but a bump to 0.3.1 fixed the build, and it could
> go to master also. Similar story with working to remove
> fontforge-20190801 which no longer built on core-updates. I figured that
> applying the patch to both branches would make it easier to merge master
> into core-updates since there would be less drift between the two.

You're right regarding drift, but unfortunately I think this duplication

of commits creates merge conflicts, or at least makes them much more

likely.

Toggle quote (2 lines)

> What is the correct way to apply a patch to multiple branches?

I'd frame the problem differently, we don't want multiple branches, we
want everything on master. Unfortunately for some changes that is hard
to test and creates too much churn, so for those changes they go to a
branch where they can be built and tested prior to pushing/merging them
to master.

To avoid changes happening on a topic branch and master, I think it's
important that any change that can be made on master, is made on
master. That should avoid the problem where someone else comes along and
doesn't realise a change has been made on core-updates, and duplicates
that on master. In this instance, you pushed the changes to master,
which is great.

I realise it does require more up front effort, but if you see a failing
build on a topic branch that's fixed by some change on master, the topic
branch should be rebased to include those changes. Providing it's done
correctly, merging should be fine as well, but the docs now say to avoid
that and prefer rebasing (mostly because of how merging has introduced
duplicate commits in to core-updates)

This rebasing or merging will minimise the drift between the two
branches as well, while avoiding Git conflicts and commit duplication.

So I don't think we want to be applying patches to multiple branches. If
it can be applied to master, it should be applied there, then all the
branches should be rebased to include it.

Does that make sense? This isn't really a "doing it wrong" thing, more
that we need to try and optimise the process for managing branches. I
haven't got a concise definition of what we're trying to optimise for,
but in this instance we want to avoid Git issues like merge conflicts
and avoid duplication and complexity by making changes that can be made
on master, just on master.

Christopher Baines wrote on 9 Jun 13:34 +0200

Recipients:(address . 70456@debbugs.gnu.org)

Message-ID:87bk4a1hav.fsf@cbaines.net

Christopher Baines <mail@cbaines.net> writes:

Toggle quote (32 lines)> Efraim Flashner <efraim@flashner.co.il> writes:
>
>> On Sun, Jun 09, 2024 at 10:46:10AM +0100, Christopher Baines wrote:
>>> guix-commits@gnu.org writes:
>>> 
>>> > efraim pushed a change to branch core-updates
>>> > in repository guix.
>>> >
>>> >     from c8c6883398 gnu: dico: Add libxcrypt dependency.
>>> >      new 9804f8c149 gnu: coeurl: Update to 0.3.1.
>>> >      new 51c7b6d76f gnu: font-gnu-freefont: Build with newer fontforge.
>>> >      new 0e06c9697a gnu: Remove fontforge-20190801.
>>> >
>>> > The 3 revisions listed above as "new" are entirely new to this
>>> > repository and will be described in separate emails.  The revisions
>>> > listed as "add" were already present in the repository and have only
>>> > been added to this reference.
>>> 
>>> These changes confused me as I was looking at the trying to work out why
>>> they needed to be pushed to core-updates. Eventually I figured out that
>>> Git is right, these commits are entirely new, but they duplicate
>>> existing commits already pushed to master (e.g. 0e06c9697a is a
>>> duplicate of 3d5f4b2d7dda).
>>> 
>>> I know the new guidance says to "Avoid merging master in to the branch",
>>> but one of the reasons for that is to avoid situations just like this
>>> where merges are done incorrectly and commits are duplicated between
>>> branches.
>>> 
>>> To fix this, I think we should rebase core-updates on master and drop
>>> these commits.

I've gone ahead and done this now.

Lars-Dominik Braun wrote on 14 Jun 08:30 +0200

Re: Request for merging core-updates branch

Recipients:(name . Steve George)(address . steve@futurile.net)

Message-ID:ZmvjnvQ9K37K938Q@noor.fritz.box

Hi,

it seems the core-updates branch is first in the merge-queue. haskell-team

was successfully built by the CI and is ready to be merged. Since there

does not seem to be an ETA for core-updates, can I skip the queue and

go ahead with merging haskell-team?

Lars

Guillaume Le Vaillant wrote on 14 Jun 09:55 +0200

Re: bug#70456: Request for merging core-updates branch

Recipients:(name . Lars-Dominik Braun)(address . lars@6xq.net)

Message-ID:87cyokt0v1.fsf@kitej

Lars-Dominik Braun <lars@6xq.net> skribis:

Toggle quote (9 lines)

> Hi,

> it seems the core-updates branch is first in the merge-queue. haskell-team

> was successfully built by the CI and is ready to be merged. Since there

> does not seem to be an ETA for core-updates, can I skip the queue and

> go ahead with merging haskell-team?

> Lars

Hi.

The lisp-team branch is also in a good shape and ready to be merged.

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCZmv3Yg8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j+6xAD/X6AXpCySJa55Cn848tSt3i01ZqEbyfwEFTka

VQ9NlA4A/2UyU10yXlMr95Ybrn+JHOLcNalAvS5hGk84BlAUXX94

=zc2X

-----END PGP SIGNATURE-----

Maxim Cournoyer wrote on 17 Jun 21:53 +0200

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)

Message-ID:87jzinwdl5.fsf@gmail.com

Hi,

Guillaume Le Vaillant <glv@posteo.net> writes:

Toggle quote (14 lines)> Lars-Dominik Braun <lars@6xq.net> skribis:
>
>> Hi,
>>
>> it seems the core-updates branch is first in the merge-queue. haskell-team
>> was successfully built by the CI and is ready to be merged. Since there
>> does not seem to be an ETA for core-updates, can I skip the queue and
>> go ahead with merging haskell-team?
>>
>> Lars
>
> Hi.
> The lisp-team branch is also in a good shape and ready to be merged.

I think it's fine to merge these first; perhaps the core-updates merge
request should be removed if it was preposterous (usually we issue the
merge request when we are confident the branch is ready to me merged).

-- 
Thanks,
Maxim

Guillaume Le Vaillant wrote on 17 Jun 22:19 +0200

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)

Message-ID:874j9rgw4e.fsf@kitej

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (22 lines)> Hi,
>
> Guillaume Le Vaillant <glv@posteo.net> writes:
>
>> Lars-Dominik Braun <lars@6xq.net> skribis:
>>
>>> Hi,
>>>
>>> it seems the core-updates branch is first in the merge-queue. haskell-team
>>> was successfully built by the CI and is ready to be merged. Since there
>>> does not seem to be an ETA for core-updates, can I skip the queue and
>>> go ahead with merging haskell-team?
>>>
>>> Lars
>>
>> Hi.
>> The lisp-team branch is also in a good shape and ready to be merged.
>
> I think it's fine to merge these first; perhaps the core-updates merge
> request should be removed if it was preposterous (usually we issue the
> merge request when we are confident the branch is ready to me merged).

Hi.

It might be more logical to have two steps. First a "work started on

xyz-team branch" message to indicate to the QA to make the stats for

this branch, and then the "request for merging xyz-team branch" message

to put the branch in the merge queue.

I'll try to merge the lisp-team branch tomorrow (UTC+2 timezone).

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCZnCaYQ8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j849AD+PKnovStS23LN8FMbYiKcBMT5kRopWgbmI636

YqPoEHQA/2S+0bkCsm14khKIcoTLNosuJxqzo2EXWx9pMJrtSq6Q

=ERdq

-----END PGP SIGNATURE-----

??? wrote on 18 Jun 04:14 +0200

core-updates failed package: time

Recipients:(address . 70456@debbugs.gnu.org)

Message-ID:87bk3zgfpm.fsf@envs.net

time-1.9 fail its test on core-updates:

https://ci.guix.gnu.org/build/4587082/log/raw

time(1) failed to detect 5MB allcoation.

mem-baseline(kb): 768

mem-5MB(kb): 5720

delta(kb): 4952

FAIL tests/time-max-rss.sh (exit status: 1)

delta is 4952, but it expect 5000-6000.

No idea what's the reason, maybe skip it?

??? wrote on 18 Jun 09:49 +0200

core-updates failed package: libfaketime for i686-linux

Recipients:(address . 70456@debbugs.gnu.org)

Message-ID:875xu63d2a.fsf_-_@envs.net

??? <iyzsong@envs.net> writes:

libfaketime (dependency of nss, poppler, cairo) failed for i686-linux

https://ci.guix.gnu.org/build/4555668/log/raw

out=1717606732 (secs since Epoch) expected=1 - bad

out=1717606732 (secs since Epoch) expected=2 - bad

out=1717606732 (secs since Epoch) expected=4 - bad

that mean FAKETIME=1 date doesn't report 1, but 1717606732.

I think that due to libfaketime doesn't work with _TIME_BITS=64 on 32bit

systems. Maybe this issue

https://github.com/wolfcw/libfaketime/issues/418.

I have no idea how to fix this, help need!

Maxim Cournoyer wrote on 18 Jun 14:15 +0200

Re: bug#70456: Request for merging core-updates branch

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)

Message-ID:87cyoee9am.fsf@gmail.com

Hi,

Guillaume Le Vaillant <glv@posteo.net> writes:

Toggle quote (31 lines)> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Hi,
>>
>> Guillaume Le Vaillant <glv@posteo.net> writes:
>>
>>> Lars-Dominik Braun <lars@6xq.net> skribis:
>>>
>>>> Hi,
>>>>
>>>> it seems the core-updates branch is first in the merge-queue. haskell-team
>>>> was successfully built by the CI and is ready to be merged. Since there
>>>> does not seem to be an ETA for core-updates, can I skip the queue and
>>>> go ahead with merging haskell-team?
>>>>
>>>> Lars
>>>
>>> Hi.
>>> The lisp-team branch is also in a good shape and ready to be merged.
>>
>> I think it's fine to merge these first; perhaps the core-updates merge
>> request should be removed if it was preposterous (usually we issue the
>> merge request when we are confident the branch is ready to me merged).
>
> Hi.
>
> It might be more logical to have two steps. First a "work started on
> xyz-team branch" message to indicate to the QA to make the stats for
> this branch, and then the "request for merging xyz-team branch" message
> to put the branch in the merge queue.

That'd be neat. Some other flow/UI idea:

From the QA interface, have a "Request to build" button action attached

to a branch.

If the branch could be built successfully (with all checks OK), enable a

"Request to merge" button, that could send an email with the patches to

review to guix-patches.

Thanks,

Maxim

Christopher Baines wrote on 21 Jun 17:13 +0200

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)

Message-ID:87msne2ury.fsf@cbaines.net

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (20 lines)> Guillaume Le Vaillant <glv@posteo.net> writes:
>
>> Lars-Dominik Braun <lars@6xq.net> skribis:
>>
>>> Hi,
>>>
>>> it seems the core-updates branch is first in the merge-queue. haskell-team
>>> was successfully built by the CI and is ready to be merged. Since there
>>> does not seem to be an ETA for core-updates, can I skip the queue and
>>> go ahead with merging haskell-team?
>>>
>>> Lars
>>
>> Hi.
>> The lisp-team branch is also in a good shape and ready to be merged.
>
> I think it's fine to merge these first; perhaps the core-updates merge
> request should be removed if it was preposterous (usually we issue the
> merge request when we are confident the branch is ready to me merged).

Can you clarify what you mean by preposterous?

While the guidance did previously say to raise an issue when you wanted

to merge the branch, it's now changed to when you create the branch in

an attempt to avoid this of situation of long running stateful branches

in the future.

I fail to see how merging core-updates is going to get easier if we

wait.

Christopher Baines wrote on 22 Jun 21:33 +0200

Re: Request for merging "core-updates" branch

Recipients:(address . 70456@debbugs.gnu.org)

Message-ID:87msnc6ac8.fsf@cbaines.net

Hey,

I've spent a bunch of time in the last few days trying to see if I can

get the bordeaux build farm moving on core-updates and I think things

are moving at pace now.

Builds are happening for 6 systems, with the only major omission being

i586-gnu, I think there are existing issues with the guix-daemon in the

childhurds not being able to stop builds which timeout, which leads to

them getting stuck on builds. I'm not sure if there's an open bug about

this.

The other major issue that comes up every time there's a core-updates

round is that the x86_64-linux bootstrap doesn't seem to build on btrfs,

or at least milano-guix-1 which uses btrfs. I have to work around this

by scheduling builds with --tag=filesystem=ext4 to have them run on

other machines. There isn't a good bug for this, but #53416 probably

applies.

data.qa.guix.gnu.org seemed to be timing out more than usual for both

patches and branches, so I've made some changes there and in the

qa-frontpage to mitigate that. The core-updates page should now always

load, although something needs adding so you can see how up to date the

data is.

1: https://qa.guix.gnu.org/branch/core-updates

Andreas has also been getting some additional x86_64-linux/i686-linux

agents up and running which should help to speed up the build

throughput.

Chris

Maxim Cournoyer wrote on 24 Jun 06:07 +0200

Re: bug#70456: Request for merging core-updates branch

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:87sex30yrv.fsf@gmail.com

Hi Christopher,

Christopher Baines <mail@cbaines.net> writes:

Toggle quote (29 lines)> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
>> Guillaume Le Vaillant <glv@posteo.net> writes:
>>
>>> Lars-Dominik Braun <lars@6xq.net> skribis:
>>>
>>>> Hi,
>>>>
>>>> it seems the core-updates branch is first in the merge-queue. haskell-team
>>>> was successfully built by the CI and is ready to be merged. Since there
>>>> does not seem to be an ETA for core-updates, can I skip the queue and
>>>> go ahead with merging haskell-team?
>>>>
>>>> Lars
>>>
>>> Hi.
>>> The lisp-team branch is also in a good shape and ready to be merged.
>>
>> I think it's fine to merge these first; perhaps the core-updates merge
>> request should be removed if it was preposterous (usually we issue the
>> merge request when we are confident the branch is ready to me merged).
>
> Can you clarify what you mean by preposterous?
>
> While the guidance did previously say to raise an issue when you wanted
> to merge the branch, it's now changed to when you create the branch in
> an attempt to avoid this of situation of long running stateful branches
> in the future.

OK, sorry, I somehow had forgotten/missed that. Some branches might

require more work post merge-request than was expected; it's hard to say

without trying it first.

Toggle quote (3 lines)

> I fail to see how merging core-updates is going to get easier if we

> wait.

It's not.  But while things are still being worked on, if other branches
are done and ready, they shouldn't be in held in limbo.

-- 
Thanks,
Maxim

Christopher Baines wrote on 27 Jun 15:12 +0200

Re: Request for merging "core-updates" branch

Recipients:(address . 70456@debbugs.gnu.org)

Message-ID:875xtu4jix.fsf@cbaines.net

Christopher Baines <mail@cbaines.net> writes:

Toggle quote (4 lines)

> I've spent a bunch of time in the last few days trying to see if I can

> get the bordeaux build farm moving on core-updates and I think things

> are moving at pace now.

Following on from this, builds have mostly paused for the last few days

as many changes have landed on the master branch.

There's a few impactful changes in this range [1] (subversion,

git-minimal, qt-build-system, ...) but also going back just over a week

there were the lisp-team changes pushed to master (ecl and sbcl updates)

which I don't think are in core-updates yet.

1: https://git.savannah.gnu.org/cgit/guix.git/log/?qt=range&q=c5fc11488c..831001c581

It's hard to tell how these changes will interact with those on

core-updates but I think we need to just rebase the branch and find out.

I've now done that and pushed it as core-updates-next. This is mostly to

allow others to make changes to core-updates-next today, before swapping

it out with core-updates tomorrow.

This core-updates-next also includes a few additional patches, plus the

contents of tex-team, since that was next in the queue, and the changes

shouldn't be risky.

Some good progress was made with building core-updates, so I'm hoping

that with this next iteration (and hopefully a quiet period on the

master branch) we can get to the point where we can merge.

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 70456@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 70456

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date