[PATCH 0/2] Disable keyword expansion for Subversion downloads

Hello Guix!

As discussed in https://issues.guix.gnu.org/43442, SVN keyword

expansion during checkout is problematic because it depends on local

timezone. This in itself isn't really a problem in Guix because

the timezone in the environment of fixed-output derivations is

always UTC. However, SWH disables keyword expansion (for this

reason), and thus it makes sense to follow suit.

Disabling keyword expansion entails many rebuilds. However, these

are "safe", so I'd like to have them on a branch (or simply handled

by qa.guix) that we'd merge as soon as it's built.

The second patch makes ‘recursive?’ default to #f, which AFAICS

is what all the SVN-using packages that were not explicit expected

(no changes required).

Thoughts?

Ludo’.

Ludovic Courtès (2):

svn-download: Disable keyword expansion.

svn-download: Default to non-recursive checkouts.

gnu/packages/java.scm | 2 +-

gnu/packages/machine-learning.scm | 2 +-

gnu/packages/tex.scm | 16 ++++++++--------

gnu/packages/video.scm | 4 ++--

guix/build/svn.scm | 9 ++++++++-

guix/svn-download.scm | 6 +++---

6 files changed, 23 insertions(+), 16 deletions(-)

base-commit: 47fe66586e0fbc177288ec91cc88ea0b081505b0

--

2.39.2

As it turns out, all packages that fetch code from Subversion expect it

to be non-recursive by default. Clarify that.

Reported by Timothy Sample <samplet@ngyro.com>.

(<svn-multi-reference>)[recursive?]: Likewise.

---

guix/svn-download.scm | 6 +++---

1 file changed, 3 insertions(+), 3 deletions(-)

Subversion keyword expansion is potentially non-reproducible as some of

them expand time strings relative to the local time zone:

https://issues.guix.gnu.org/43442#18

In practice this is not a problem in Guix since Subversion checkouts

happen in an isolated environment using the "default timezone" (UTC).

However, Software Heritage disables keyword expansion for this very

reason. By following suit, we make sure content can be retrieved from

there.

Reported by Timothy Sample <samplet@ngyro.com>.

accordingly.

(texlive-dvips): Likewise.

(texlive-cm): Likewise.

(texlive-tex-plain): Likewise.

(texlive-kpathsea): Likewise

(texlive-latexconfig): Likewise.

(texlive-tetex): Likewise.

(texlive-pdftex): Likewise.

(libsmpeg-with-sdl1): Likewise.

---

gnu/packages/java.scm | 2 +-

gnu/packages/machine-learning.scm | 2 +-

gnu/packages/tex.scm | 16 ++++++++--------

gnu/packages/video.scm | 4 ++--

guix/build/svn.scm | 9 ++++++++-

5 files changed, 20 insertions(+), 13 deletions(-)

Hi Ludo,

Whoops! I guess I was a little slow getting my patch in. I’ve

highlighted a few sources you missed below, as well as attached my patch

(for comparison and for listing the new hashes of the missing sources).

Ludovic Courtès <ludo@gnu.org> writes:

Also java-geronimo-genesis-2.1.

Also texlive-dvipdfmx.

Also texlive-xetex.

There’s also the “texlive-scripts” input in texlive-bin, and the

“texlive-luatexconfig” native input in texlive-latex-base.

I generated a big diff for all 17 sources that I found (mostly because

it excites me that Guix can do stuff like that). Basically all the

changes were boring (as we expected).

For “texlive-scripts”, I added a phase that approximates keyword

expansion for the few scripts that use it to output version information.

It‘s not exactly the same as what Subversion does. Subversion uses the

last revision that updated the given file, and I just use

‘%texlive-revision’ for every file. I’m not too attached to this

approach, but it felt like the right thing to do.

I also carefully checked turning recursion off by default. I’m

confident that nothing changes.

-- Tim

Hi Timothy!

Timothy Sample <samplet@ngyro.com> skribis:

No problem! I had it on my to-do list.

Oh, good that you were able to find some more!

Excellent.

I wouldn’t mind doing nothing, but what you did sounds good.

Excellent. I’ll push it to ‘master’.

For posterity, you can add something along the lines of what I had:

Subversion keyword expansion is potentially non-reproducible as some of

them expand time strings relative to the local time zone:

https://issues.guix.gnu.org/43442#18

In practice this is not a problem in Guix since Subversion checkouts

happen in an isolated environment using the "default timezone" (UTC).

However, Software Heritage disables keyword expansion for this very

reason. By following suit, we make sure content can be retrieved from

there.

Likewise:

+ ;; Disable keyword substitutions (keywords are CVS-like strings

+ ;; like "$Date$", "$Id$", and so on) for two reasons: (1) some

+ ;; expansions depend on the local time zone, and (2) SWH disables

+ ;; it in its archive for this very reason.

+ "--ignore-keywords"

Chris, should we get it built on qa.guix? It’s on the order of 7K

rebuilds per architecture since ‘texlive-bin’ is modified.

Alternatively we can set up ci.guix for it, if you think it’s more

appropriate resource-wise.

WDYT?

Ludo’.

PS: I’ll be AFK for the next few days.

Ludovic Courtès <ludo@gnu.org> writes:

The data service comparison is available now which is good.

The builds won't be submitted automatically since there's so many, but

I've shuffled the code around to allow manually submitting them, I'm

running the following command in a screen session on bayfront:

root@bayfront /var/lib/qa-frontpage# sudo -u qa-frontpage /gnu/store/lv4014zwba44fzf035pafba228lypqyl-guix-qa-frontpage-0-7.53bd333/bin/guix-qa-frontpage submit-issue-builds 62712 --priority=600

It will be some time before the builds are completed though, I also need

to try and sort out something for the ARM build machines, as they seem

to be really struggling with slow upload speeds at the moment.

Christopher Baines <mail@cbaines.net> writes:

This has now finished running (I don't exactly know when), so the

package changes information on [1] is now showing up.

All the x86_64-linux builds have already completed, and builds for other

systems are happening.

1: https://qa.guix.gnu.org/issue/62712

Hi Chris,

Christopher Baines <mail@cbaines.net> skribis:

Thanks a lot!

With ‘core-updates’ coming up, I’m not sure what the best strategy is.

The data at qa.guix should give us confidence, so perhaps we can rebase

it after the ‘core-updates’ merge and apply it at that point?

Ludo’.

Hi,

Oh cool! Thanks.

On Thu, 20 Apr 2023 at 11:57, Ludovic Courtès <ludo@gnu.org> wrote:

Yes, it looks good to me.

Cheers,

simon

Hi Ludo,

On jeu., 20 avril 2023 at 11:57, Ludovic Courtès <ludo@gnu.org> wrote:

What is the status of this patch? The merge of ’core-updates’ is behind

us now \o/ yeah! and all seems green for installing this patch, no?

Cheers,

simon

Hello,

Simon Tournier <zimon.toutoune@gmail.com> writes:

It was applied in "tex-team" branch, which is currently being built by

the CI.

Regards,

--

Nicolas Goaziou

Hi,

On sam., 06 mai 2023 at 02:22, Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

Cool! This branch seems green, isn’t it? Is it merged?

Well, since the patch is installed, what about closing this submission?

Cheers,

simon

Hi!

Simon Tournier <zimon.toutoune@gmail.com> skribis:

Nope. Unfortunately, due to the recent ‘rust-team’ merge, we’ll have to

merge back and wait for CI to build it again.

Nicolas, can you take care of merging ‘master’ into the branch? (Or

rebasing it.)

Ludo’.

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

Done. I re-created `tex-team' on top of `master'. It can be built again.

Regards,

--

Nicolas Goaziou

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

Given this might be the next branch to try and merge to master, would

you be interested in trying out the proecss I set out here [1]?

1: https://issues.guix.gnu.org/63459

All that would involve is creating a guix-patches issue saying you're

looking to merge the tex-team branch.

I'm interested in this since I want to look at getting the qa-frontpage

to spot these issues for branch merges, and then use that information to

manage builds and testing.

Chris

Hello,

Christopher Baines <mail@cbaines.net> writes:

Done in https://issues.guix.gnu.org/63521

Regards,

--

Nicolas Goaziou

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

Awesome, thanks. I'll have a look at getting the qa-frontpage to detect

that issue and start submitting builds for the tex-team branch.

block 62712 by 63521

quit

This was merged last week (see commit

e6da6e3152b01b36d925d8670f1b8c3e1a39ef4b).

Closing!

Ludo’.