video acceleration/libva segfaults caused by stale mesa shader cache

Open

Details

2 participants

Giovanni Biscuolo
Maxim Cournoyer

Owner: unassigned

Submitted by: Maxim Cournoyer

Severity: normal

Maxim Cournoyer wrote on 1 May 2023 04:42

Recipients:(name . bug-guix)(address . bug-guix@gnu.org)

Message-ID:875y9c7owu.fsf@gmail.com

Hi,

After reinstalling someone's desktop which has support for VA-API,

'vainfo' from 'libva-utils' would consume all the memory then crash.

Other applications relying on libva would crash as well, e.g. ffmpeg (or

its users, such as vlc/jami). Here's a sample output from VLC:

Toggle snippet (9 lines)

vlc received_605209834855384.mp4

VLC media player 3.0.18 Vetinari (revision 3.0.13-8-g41878ff4f2)

[000000000109d770] main libvlc: Lancement de vlc avec l'interface par défaut. Utiliser « cvlc » pour démarrer VLC sans interface.

libva info: VA-API version 1.17.0

libva info: Trying to open /gnu/store/9pypr3c3y379shbwm9ilb4pik9mkfd83-mesa-22.2.4/lib/dri/radeonsi_drv_video.so

libva info: Found init function __vaDriverInit_1_17

Erreur de segmentation

After tracing the process, I noticed that the last thing it did was

loading its mesa shader cache, stored under:

Toggle snippet (3 lines)

~/.cache/mesa_shader_cache

Deleting that directory resolved the issue.

It seems that'd be a bug in Mesa (for failing to determine that it

should have invalidated its cache going from version 21 to 22 post

core-updates merge).

Thanks,

Maxim

Maxim Cournoyer wrote on 1 May 2023 04:58

Recipients:(address . 63197@debbugs.gnu.org)

Message-ID:871qk07o7a.fsf@gmail.com

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (27 lines)> Hi,
>
> After reinstalling someone's desktop which has support for VA-API,
> 'vainfo' from 'libva-utils' would consume all the memory then crash.
> Other applications relying on libva would crash as well, e.g. ffmpeg (or
> its users, such as vlc/jami).  Here's a sample output from VLC:
>
> vlc received_605209834855384.mp4 
> VLC media player 3.0.18 Vetinari (revision 3.0.13-8-g41878ff4f2)
> [000000000109d770] main libvlc: Lancement de vlc avec l'interface par défaut. Utiliser « cvlc » pour démarrer VLC sans interface.
> libva info: VA-API version 1.17.0
> libva info: Trying to open /gnu/store/9pypr3c3y379shbwm9ilb4pik9mkfd83-mesa-22.2.4/lib/dri/radeonsi_drv_video.so
> libva info: Found init function __vaDriverInit_1_17
> Erreur de segmentation
>
>
> After tracing the process, I noticed that the last thing it did was
> loading its mesa shader cache, stored under:
>
> ~/.cache/mesa_shader_cache
>
> Deleting that directory resolved the issue.
>
> It seems that'd be a bug in Mesa (for failing to determine that it
> should have invalidated its cache going from version 21 to 22 post
> core-updates merge).

I've forwarded this report upstream here:
https://gitlab.freedesktop.org/mesa/mesa/-/issues/8937

-- 
Thanks,
Maxim

Giovanni Biscuolo wrote on 15 Jun 2023 15:49

Recipients:

Message-ID:87pm5w3ke9.fsf@xelera.eu

Hi Maxim,

I learned about this issue today

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

Toggle quote (11 lines)>> After tracing the process, I noticed that the last thing it did was
>> loading its mesa shader cache, stored under:
>>
>> ~/.cache/mesa_shader_cache
>>
>> Deleting that directory resolved the issue.
>>
>> It seems that'd be a bug in Mesa (for failing to determine that it
>> should have invalidated its cache going from version 21 to 22 post
>> core-updates merge).

AFAIU this issue is still present using mesa 23 since Guillaume Le

Vaillant had to use this workaround yesterday [1] and reported his

backtrace upstream [2]

If I'm not wrong (i.e. vlc et al are now using mesa 23) this should also

be reported upstream (I can do it if needed).

AFAIU the only thing we can do to fix this bug is to disable the shader

cache (MESA_SHADER_CACHE_DISABLE=true) until a proper fix is found

upstream.

...or apply a patch to rename "~/.cache/mesa_shader_cache" to

"~/.cache/mesa<version>_shader_cache"

Alternatively, we should find a way to make Guix users aware of this

kind of problems and possible workarounds they can apply (it's not

related to this specific bug)

WDYT?

Thanks! Gio'

[1] id:871qify1i8.fsf@kitej

[2] https://gitlab.freedesktop.org/mesa/mesa/-/issues/8937#note_1960628

Giovanni Biscuolo

Xelera IT Infrastructures

Maxim Cournoyer wrote on 17 Jun 2023 02:36

Recipients:(name . Giovanni Biscuolo)(address . g@xelera.eu)(address . 63197@debbugs.gnu.org)

Message-ID:878rcivs9q.fsf@gmail.com

Hello,

Giovanni Biscuolo <g@xelera.eu> writes:

Toggle quote (26 lines)> Hi Maxim,
>
> I learned about this issue today
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>
> [...]
>
>>> After tracing the process, I noticed that the last thing it did was
>>> loading its mesa shader cache, stored under:
>>>
>>> ~/.cache/mesa_shader_cache
>>>
>>> Deleting that directory resolved the issue.
>>>
>>> It seems that'd be a bug in Mesa (for failing to determine that it
>>> should have invalidated its cache going from version 21 to 22 post
>>> core-updates merge).
>
> AFAIU this issue is still present using mesa 23 since Guillaume Le
> Vaillant had to use this workaround yesterday [1] and reported his
> backtrace upstream [2]
>
> If I'm not wrong (i.e. vlc et al are now using mesa 23) this should also
> be reported upstream (I can do it if needed).

Which upstream are you thinking about? My understanding is that this

problem is a Mesa problem, and it's already reported there (the issue

linked in [2]).

Toggle quote (4 lines)

> AFAIU the only thing we can do to fix this bug is to disable the shader

> cache (MESA_SHADER_CACHE_DISABLE=true) until a proper fix is found

> upstream.

Disabling the shader cache sounds like a decent workaround or even
definitive solution.  One less stale cache to worry about...  If it's
like the Qt shader cache, the performance hit is probably too small to
be noticeable (maybe just slower startup times of complicated opengl
applications such as games?).

Toggle quote (3 lines)

> ...or apply a patch to rename "~/.cache/mesa_shader_cache" to

> "~/.cache/mesa<version>_shader_cache"

That's another good idea.

Toggle quote (4 lines)

> Alternatively, we should find a way to make Guix users aware of this

> kind of problems and possible workarounds they can apply (it's not

> related to this specific bug)

I would rather pursue the other above options you suggest, so that it

doesn't happen in the first place!

Thank you for sharing these ideas.

Maxim

Giovanni Biscuolo wrote on 17 Jun 2023 12:14

Recipients:(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 63197@debbugs.gnu.org)

Message-ID:878rci2y6u.fsf@xelera.eu

Hi Maxim

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

Toggle quote (9 lines)

>> AFAIU this issue is still present using mesa 23 since Guillaume Le

>> Vaillant had to use this workaround yesterday [1] and reported his

>> backtrace upstream [2]

>> If I'm not wrong (i.e. vlc et al are now using mesa 23) this should also

>> be reported upstream (I can do it if needed).

> Which upstream are you thinking about?

mesa

Toggle quote (3 lines)

> My understanding is that this problem is a Mesa problem, and it's

> already reported there (the issue linked in [2]).

yes but the original bug report mentions Mesa 22.2.4 and M. Briar asked:

Toggle snippet (5 lines)

Mesa 22.2.x is already end-of-life and won't receive any fixes

anymore. Does this also happen on newer versions?

(https://gitlab.freedesktop.org/mesa/mesa/-/issues/8937#note_1891435)

IMHO there is no clear answer to that question in the bug thread, maybe

mesa developers still think it's just 22.2.X related

Now we have Mesa 23.0.3 in Giux, probably the one used by vlc when

Guillaume reported his issue upstream (mesa) on June 15

Toggle quote (7 lines)

>> AFAIU the only thing we can do to fix this bug is to disable the shader

>> cache (MESA_SHADER_CACHE_DISABLE=true) until a proper fix is found

>> upstream.

> Disabling the shader cache sounds like a decent workaround or even

> definitive solution. One less stale cache to worry about...

oh yes! Unfortunately cache management is not so robust... sometimes :-(

Toggle quote (4 lines)

> If it's like the Qt shader cache, the performance hit is probably too

> small to be noticeable (maybe just slower startup times of complicated

> opengl applications such as games?).

I don't know the cost in term of performance, I'm not a 3D expert at
all; from what I read on the web about shader chaches I guess it's a
real problem almost only for games and I guess it's not a problem at
all for media players like vlc et al: I'm just brainstorming but what
about having a mesa-with-cache-enabled version just for the games, if it
is really needed?

I should be able to propose a patch to disable the mesa shader cache,
but since I'm not an expert in this field I prefer to leave this
decision (to disable the cache, I mean) to someone else

Toggle quote (5 lines)

>> ...or apply a patch to rename "~/.cache/mesa_shader_cache" to

>> "~/.cache/mesa<version>_shader_cache"

> That's another good idea.

I was just doing guesswork but the bug caused by this mesa upgrade

smells like a binary incompatibility between two versions (or just major

versions)... so a versioned shader cache makes sense to me

I'm not able to propose (I mean to code) such a patch, anyway

Anyway, users should know that they have to periodically clean unused

shader caches, since from what I read on the net the shader cache tends

to really /explode/ in terms of size, in some cases

Toggle quote (7 lines)

>> Alternatively, we should find a way to make Guix users aware of this

>> kind of problems and possible workarounds they can apply (it's not

>> related to this specific bug)

> I would rather pursue the other above options you suggest, so that it

> doesn't happen in the first place!

I agree

Toggle quote (2 lines)

> Thank you for sharing these ideas.

Thank you for your attention!

Happy hacking, Gio'

Giovanni Biscuolo

Xelera IT Infrastructures

Maxim Cournoyer wrote on 26 Jun 2023 18:21

Recipients:(name . Giovanni Biscuolo)(address . g@xelera.eu)(address . 63197@debbugs.gnu.org)

Message-ID:87r0py19fj.fsf@gmail.com

Hi Giovanni,

Giovanni Biscuolo <g@xelera.eu> writes:

[...]

Toggle quote (22 lines)>>> ...or apply a patch to rename "~/.cache/mesa_shader_cache" to
>>> "~/.cache/mesa<version>_shader_cache"
>>
>> That's another good idea.
>
> I was just doing guesswork but the bug caused by this mesa upgrade
> smells like a binary incompatibility between two versions (or just major
> versions)... so a versioned shader cache makes sense to me
>
> I'm not able to propose (I mean to code) such a patch, anyway
>
> Anyway, users should know that they have to periodically clean unused
> shader caches, since from what I read on the net the shader cache tends
> to really /explode/ in terms of size, in some cases
>
>>> Alternatively, we should find a way to make Guix users aware of this
>>> kind of problems and possible workarounds they can apply (it's not
>>> related to this specific bug)
>>
>> I would rather pursue the other above options you suggest, so that it
>> doesn't happen in the first place!

I've ping'd upstream with
https://gitlab.freedesktop.org/mesa/mesa/-/issues/8937#note_1975560.
Let's see what they say!

-- 
Thanks,
Maxim

Maxim Cournoyer wrote on 25 Apr 20:47 +0200

Recipients:(address . 63197@debbugs.gnu.org)

Message-ID:875xw5ffdz.fsf@gmail.com

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (15 lines)> Hi,
>
> After reinstalling someone's desktop which has support for VA-API,
> 'vainfo' from 'libva-utils' would consume all the memory then crash.
> Other applications relying on libva would crash as well, e.g. ffmpeg (or
> its users, such as vlc/jami).  Here's a sample output from VLC:
>
> vlc received_605209834855384.mp4 
> VLC media player 3.0.18 Vetinari (revision 3.0.13-8-g41878ff4f2)
> [000000000109d770] main libvlc: Lancement de vlc avec l'interface par défaut. Utiliser « cvlc » pour démarrer VLC sans interface.
> libva info: VA-API version 1.17.0
> libva info: Trying to open /gnu/store/9pypr3c3y379shbwm9ilb4pik9mkfd83-mesa-22.2.4/lib/dri/radeonsi_drv_video.so
> libva info: Found init function __vaDriverInit_1_17
> Erreur de segmentation

The same issue was reproduced with vlc and Totem after the latest

upgrades, which brought mesa from version 23 to 24. The same solution:

Toggle snippet (3 lines)

$ rm -rf ./.cache/mesa_shader_cache

still works.

I've sent upstream a fresh snapshot of the 23 -> 24 corrupted cache [0].

[0] https://gitlab.freedesktop.org/mesa/mesa/-/issues/8937#note_2388471

Thanks,

Maxim

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 63197@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 63197

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date