[PATCH] gnu: Make Mesa use zstd compression for shader cache instead of zlib.

Done

Details

4 participants

???
John Kehayias
Christopher Baines
Sigve Sudland

Owner: unassigned

Submitted by: Sigve Sudland

Severity: normal

Sigve Sudland wrote on 19 Jul 2023 23:02

Recipients:(address . guix-patches@gnu.org)(name . Sigve Sudland)(address . sigve_sudland@hotmail.com)

Message-ID:AS8P250MB07186E45F45ABAC876FE11629B39A@AS8P250MB0718.EURP250.PROD.OUTLOOK.COM

* gnu/packages/gl.scm (mesa): Enable zstd compression.

---

gnu/packages/gl.scm | 6 +++++-

1 file changed, 5 insertions(+), 1 deletion(-)

Toggle diff (26 lines)diff --git a/gnu/packages/gl.scm b/gnu/packages/gl.scm
index 1691086e1a..2f54d8553d 100644
--- a/gnu/packages/gl.scm
+++ b/gnu/packages/gl.scm
@@ -299,7 +299,8 @@ (define-public mesa
            libxvmc
            llvm-for-mesa
            wayland
-           wayland-protocols))
+           wayland-protocols
+           `(,zstd "lib")))
     (native-inputs
      (list bison
            flex
@@ -362,6 +363,9 @@ (define-public mesa
          ;; Enable the codecs that were built by default as part of the
          ;; 21.3.x releases to avoid functionality regressions.
          "-Dvideo-codecs=vc1dec,h264dec,h264enc,h265dec,h265enc"
+
+         ;; Enable ZSTD compression for shader cache
+         "-Dzstd=enabled"

          ;; Also enable the tests.
          "-Dbuild-tests=true"

base-commit: ebb54e6a5fba36a571ed239398ee3e0307a91c26

2.41.0

??? wrote on 20 Jul 2023 12:22

Recipients:(name . Sigve Sudland)(address . sigve_sudland@hotmail.com)(address . 64738@debbugs.gnu.org)

Message-ID:87r0p2opb7.fsf@envs.net

Sigve Sudland <sigve_sudland@hotmail.com> writes:

Toggle quote (2 lines)

> * gnu/packages/gl.scm (mesa): Enable zstd compression.

Good idea! Follow our commit log style, I adjust it to:

gnu: mesa: Enable zstd compression for shader cache.

* gnu/packages/gl.scm (mesa)[inputs]: Add zstd:lib.

[arguments]: Add '-Dzstd=enabled' to configure-flags.

And apply it on the mesa-updates branch locally, wait it building, once

done I'll pushed it to the mesa-updates branch. I think this issue

should be closed when it got merged into master.

Thank you!

??? wrote on 20 Jul 2023 12:58

Recipients:(name . Sigve Sudland)(address . sigve_sudland@hotmail.com)

Message-ID:87o7k6onmw.fsf_-_@envs.net

Toggle quote (6 lines)

> gnu: mesa: Enable zstd compression for shader cache.

> * gnu/packages/gl.scm (mesa)[inputs]: Add zstd:lib.

> [arguments]: Add '-Dzstd=enabled' to configure-flags.

Pushed to the mesa-updates branch, merging progress tracked in

https://issues.guix.gnu.org/64369.

Thanks.

John Kehayias wrote on 20 Jul 2023 17:54

Re: bug#64369: Request for merging "mesa-updates" branch

Recipients:(name . ???)(address . iyzsong@envs.net)

Message-ID:87ilaezigl.fsf_-_@protonmail.com

Hi,

On Thu, Jul 20, 2023 at 06:58 PM, ??? wrote:

Toggle quote (11 lines)>>
>>     gnu: mesa: Enable zstd compression for shader cache.
>>
>>     * gnu/packages/gl.scm (mesa)[inputs]: Add zstd:lib.
>>     [arguments]: Add '-Dzstd=enabled' to configure-flags.
>
> Pushed to the mesa-updates branch, merging progress tracked in
> https://issues.guix.gnu.org/64369.
>
> Thanks.

I just saw this as the message only went to the bug number. I would not

have merged this yet without checking about build status after the big

texlive updates. I was hoping to avoid a full rebuild so we could merge

to master with substitutes already, after checking post-texlive and the

graft of mesa on master.

However, doesn't seem like Cuirass has picked up this change and hasn't

built yet. Any idea why?

My plan right now was going to be to merge master into mesa-updates, see

what the build status looks like, and then decide from there. If there

were not too many rebuilds (say less than 1000) from the updates on

master, we should be able to go ahead soon. And then start on the next

round of patches/updates for mesa-updates.

On the other hand, if there were lots of rebuilds there are other

patches waiting that should be used also (libdrm, libva if I remember)

as well as ungrafting the change on master. So, preventing waiting and

doing another big rebuild with the changes that have been waiting since

the first build of this branch.

What do people think? My thought right now was to revert this change

(since it rebuilds all of mesa dependents), merge master in, and check

the status. But I'm confused why Cuirass wasn't building..

John

Christopher Baines wrote on 20 Jul 2023 18:03

Recipients:(name . John Kehayias)(address . john.kehayias@protonmail.com)

Message-ID:871qh2y2yk.fsf@cbaines.net

John Kehayias <john.kehayias@protonmail.com> writes:

Toggle quote (24 lines)> Hi,
>
> On Thu, Jul 20, 2023 at 06:58 PM, ??? wrote:
>
>>>
>>>     gnu: mesa: Enable zstd compression for shader cache.
>>>
>>>     * gnu/packages/gl.scm (mesa)[inputs]: Add zstd:lib.
>>>     [arguments]: Add '-Dzstd=enabled' to configure-flags.
>>
>> Pushed to the mesa-updates branch, merging progress tracked in
>> https://issues.guix.gnu.org/64369.
>>
>> Thanks.
>
> I just saw this as the message only went to the bug number. I would not
> have merged this yet without checking about build status after the big
> texlive updates. I was hoping to avoid a full rebuild so we could merge
> to master with substitutes already, after checking post-texlive and the
> graft of mesa on master.
>
> However, doesn't seem like Cuirass has picked up this change and hasn't
> built yet. Any idea why?

Cuirass at ci.guix.gnu.org seems to be having problems noticing new

revisions. At least it's about a day behind for master branch revisions.

Toggle quote (16 lines)> My plan right now was going to be to merge master into mesa-updates, see
> what the build status looks like, and then decide from there. If there
> were not too many rebuilds (say less than 1000) from the updates on
> master, we should be able to go ahead soon. And then start on the next
> round of patches/updates for mesa-updates.
>
> On the other hand, if there were lots of rebuilds there are other
> patches waiting that should be used also (libdrm, libva if I remember)
> as well as ungrafting the change on master. So, preventing waiting and
> doing another big rebuild with the changes that have been waiting since
> the first build of this branch.
>
> What do people think? My thought right now was to revert this change
> (since it rebuilds all of mesa dependents), merge master in, and check
> the status. But I'm confused why Cuirass wasn't building.

The mesa package wasn't affected by tex-team-next, so unless there are
other inputs to most/all off the packages to which mesa is an input,
there shouldn't be many rebuilds. I'd still rebase or merge master in to
the branch and see what QA says though.

Whether you want to revert this latest commit and try and merge sooner,
or wait for things to be built and merge later is up to you though.

John Kehayias wrote on 20 Jul 2023 18:23

Recipients:(name . Christopher Baines)(address . mail@cbaines.net)

Message-ID:87h6pyzh4s.fsf@protonmail.com

Hi Chris,

On Thu, Jul 20, 2023 at 05:03 PM, Christopher Baines wrote:

Toggle quote (9 lines)

> John Kehayias <john.kehayias@protonmail.com> writes:

>> However, doesn't seem like Cuirass has picked up this change and hasn't

>> built yet. Any idea why?

> Cuirass at ci.guix.gnu.org seems to be having problems noticing new

> revisions. At least it's about a day behind for master branch revisions.

Okay, thanks for the info. So either way will let things build through

the weekend probably.

Toggle quote (22 lines)>> My plan right now was going to be to merge master into mesa-updates, see
>> what the build status looks like, and then decide from there. If there
>> were not too many rebuilds (say less than 1000) from the updates on
>> master, we should be able to go ahead soon. And then start on the next
>> round of patches/updates for mesa-updates.
>>
>> On the other hand, if there were lots of rebuilds there are other
>> patches waiting that should be used also (libdrm, libva if I remember)
>> as well as ungrafting the change on master. So, preventing waiting and
>> doing another big rebuild with the changes that have been waiting since
>> the first build of this branch.
>>
>> What do people think? My thought right now was to revert this change
>> (since it rebuilds all of mesa dependents), merge master in, and check
>> the status. But I'm confused why Cuirass wasn't building.
>
> The mesa package wasn't affected by tex-team-next, so unless there are
> other inputs to most/all off the packages to which mesa is an input,
> there shouldn't be many rebuilds. I'd still rebase or merge master in to
> the branch and see what QA says though.
>

Right, wasn't sure what the number of rebuilds the texlive updates

caused beyond tex-related stuff, so I didn't want to assume.

Are we able to rebase and then force push on savannah? I thought not,

or is that just for the master branch? If I can force push (I suppose

I could just try later), I would leave off 64738 (zstd in mesa) and

rebase on master. Otherwise I guess revert and merge master to

mesa-updates.

Toggle quote (4 lines)

> Whether you want to revert this latest commit and try and merge sooner,

> or wait for things to be built and merge later is up to you though.

Then either way see what rebuilding looks like. If there isn't much

I'd opt for going sooner while we have substitutes already and then

gather these other patches for the next merge. Likely waiting for a

new mesa update which is probably due very soon. But if things will be

rebuilding a lot anyway, I might as well get all these other related

changes in to just build once.

Thanks Chris, I'll see what to do and keep an eye on the builds.

John

John Kehayias wrote on 20 Jul 2023 21:35

Re: bug#64738: [PATCH] gnu: Make Mesa use zstd compression for shader cache instead of zlib.

Recipients:(name . ???)(address . iyzsong@envs.net)

Message-ID:87cz0mz89r.fsf_-_@protonmail.com

On Thu, Jul 20, 2023 at 06:58 PM, ??? wrote:

Toggle quote (11 lines)>>
>>     gnu: mesa: Enable zstd compression for shader cache.
>>
>>     * gnu/packages/gl.scm (mesa)[inputs]: Add zstd:lib.
>>     [arguments]: Add '-Dzstd=enabled' to configure-flags.
>
> Pushed to the mesa-updates branch, merging progress tracked in
> https://issues.guix.gnu.org/64369.
>
> Thanks.

An update here: I reset mesa-updates to just be the 2 commits from

before this patch (that were already built on Cuirass) but rebased on

master. Based on how many builds may be new, we can see what to do. I

hope it is just a small amount of rebuilds so mesa-updates can be merged

as is, and then include this and other patches for the next round.

Hope that is okay, especially as I think this is a minor change to mesa,

rather than a bug or missing hardware support.

John

??? wrote on 21 Jul 2023 12:15

Recipients:(name . John Kehayias)(address . john.kehayias@protonmail.com)

Message-ID:87wmytlgdp.fsf@envs.net

John Kehayias <john.kehayias@protonmail.com> writes:

Toggle quote (23 lines)> On Thu, Jul 20, 2023 at 06:58 PM, ??? wrote:
>
>>>
>>>     gnu: mesa: Enable zstd compression for shader cache.
>>>
>>>     * gnu/packages/gl.scm (mesa)[inputs]: Add zstd:lib.
>>>     [arguments]: Add '-Dzstd=enabled' to configure-flags.
>>
>> Pushed to the mesa-updates branch, merging progress tracked in
>> https://issues.guix.gnu.org/64369.
>>
>> Thanks.
>
> An update here: I reset mesa-updates to just be the 2 commits from
> before this patch (that were already built on Cuirass) but rebased on
> master. Based on how many builds may be new, we can see what to do. I
> hope it is just a small amount of rebuilds so mesa-updates can be merged
> as is, and then include this and other patches for the next round.
>
> Hope that is okay, especially as I think this is a minor change to mesa,
> rather than a bug or missing hardware support.
>

That's fine, thank you for take care of it.

John Kehayias wrote on 25 Jul 2023 23:27

Recipients:(name . ???)(address . iyzsong@envs.net)

Message-ID:87r0oviswg.fsf@protonmail.com

Hello,

On Fri, Jul 21, 2023 at 06:15 PM, ??? wrote:

Toggle quote (27 lines)> John Kehayias <john.kehayias@protonmail.com> writes:
>
>> On Thu, Jul 20, 2023 at 06:58 PM, ??? wrote:
>>
>>>>
>>>>     gnu: mesa: Enable zstd compression for shader cache.
>>>>
>>>>     * gnu/packages/gl.scm (mesa)[inputs]: Add zstd:lib.
>>>>     [arguments]: Add '-Dzstd=enabled' to configure-flags.
>>>
>>> Pushed to the mesa-updates branch, merging progress tracked in
>>> <https://issues.guix.gnu.org/64369>.
>>>
>>> Thanks.
>>
>> An update here: I reset mesa-updates to just be the 2 commits from
>> before this patch (that were already built on Cuirass) but rebased on
>> master. Based on how many builds may be new, we can see what to do. I
>> hope it is just a small amount of rebuilds so mesa-updates can be merged
>> as is, and then include this and other patches for the next round.
>>
>> Hope that is okay, especially as I think this is a minor change to mesa,
>> rather than a bug or missing hardware support.
>>
>
> That's fine, thank you for take care of it.

I ended up adding this (and other changes) to mesa-updates along with
a mesa version bump and some other patches. Hopefully merging to
master in the next few days once substitutes look good.

Thanks!
John

John Kehayias wrote on 31 Jul 2023 17:27

Recipients:(name . ???)(address . iyzsong@envs.net)

Message-ID:87pm48gkyw.fsf_-_@protonmail.com

On Tue, Jul 25, 2023 at 09:27 PM, John Kehayias wrote:

Toggle quote (38 lines)> Hello,
>
> On Fri, Jul 21, 2023 at 06:15 PM, ??? wrote:
>
>> John Kehayias <john.kehayias@protonmail.com> writes:
>>
>>> On Thu, Jul 20, 2023 at 06:58 PM, ??? wrote:
>>>
>>>>>
>>>>>     gnu: mesa: Enable zstd compression for shader cache.
>>>>>
>>>>>     * gnu/packages/gl.scm (mesa)[inputs]: Add zstd:lib.
>>>>>     [arguments]: Add '-Dzstd=enabled' to configure-flags.
>>>>
>>>> Pushed to the mesa-updates branch, merging progress tracked in
>>>> <https://issues.guix.gnu.org/64369>.
>>>>
>>>> Thanks.
>>>
>>> An update here: I reset mesa-updates to just be the 2 commits from
>>> before this patch (that were already built on Cuirass) but rebased on
>>> master. Based on how many builds may be new, we can see what to do. I
>>> hope it is just a small amount of rebuilds so mesa-updates can be merged
>>> as is, and then include this and other patches for the next round.
>>>
>>> Hope that is okay, especially as I think this is a minor change to mesa,
>>> rather than a bug or missing hardware support.
>>>
>>
>> That's fine, thank you for take care of it.
>
> I ended up adding this (and other changes) to mesa-updates along with
> a mesa version bump and some other patches. Hopefully merging to
> master in the next few days once substitutes look good.
>
> Thanks!
> John

Pushed as 090c254fe7713db330636dae2c204c8282207cc8. Thanks!

Closed

Your comment

This issue is archived.

To comment on this conversation send an email to 64738@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 64738

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date