ugrep: reproducibility issues

  • Open
  • quality assurance status badge
Details
6 participants
  • Christopher Howard
  • kiasoc5
  • Leo Famulari
  • raingloom
  • Vagrant Cascadian
  • zimoun
Owner
unassigned
Submitted by
Christopher Howard
Severity
normal
C
C
Christopher Howard wrote on 22 May 2021 22:19
ugrep: illegal instruction
(name . Guix Bug Reports)(address . bug-guix@gnu.org)
14ed9ebaa4c203c17aacbd514597c0030a07a44f.camel@librehacker.com
Hi, I found another package, ugrep, which is evidently using native
instructions that do not work on my processor. When I run ugrep (e.g.,
`ugrep -Q') it immediately dies with error 'illegal instruction'.
However, if I pull the source with `guix build -S' and build and
install it to a local directory, it runs without trouble.

I'm sure this will also be a reproducibility bug, and plan on
submitting guix challenge results. However, the command I am running to
rebuild ugrep without grafts or substitutes is rebuilding everything
all the way back to bootstrap, so it probably will not be completed for
a day or two:

guix gc -D $(guix build --no-grafts ugrep); guix build --no-substitutes
ugrep

Please compare similar bug 48373. It didn't seem worth the trouble of
running an object dump and figuring out which instruction is illegal,
but I could do that if required.

My system information:


christopher@nightshade ~$ grep 'flags' /proc/cpuinfo | head -n 1
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext
fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl
nonstop_tsc cpuid extd_apicid pni monitor cx16 popcnt lahf_lm
cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch
osvw ibs skinit wdt hw_pstate vmmcall npt lbrv svm_lock nrip_save

christopher@nightshade ~$ neofetch --stdout
christopher@nightshade
----------------------
OS: Guix System d4ffa9630277fa8699c783c08381d688626d4bc3 x86_64
Host: GA-880GM-UD2H
Kernel: 5.12.5-gnu
Uptime: 1 hour, 30 mins
Packages: 107 (guix-system), 93 (guix-user)
Shell: bash 5.0.16
Resolution: 1920x1200
DE: GNOME 3.34.5
Theme: Adwaita [GTK2/3]
Icons: Adwaita [GTK2/3]
Terminal: kitty
CPU: AMD Athlon II X3 455 (3) @ 3.300GHz
GPU: NVIDIA GeForce 8400 GS Rev. 3
Memory: 1573MiB / 7957MiB

christopher@nightshade ~$ guix describe
Generation 5 May 22 2021 09:43:30 (current)
guix d4ffa96
branch: master
commit: d4ffa9630277fa8699c783c08381d688626d4bc3


--
Christopher Howard
Attachment: file
C
C
Christopher Howard wrote on 23 May 2021 00:46
(address . 48591@debbugs.gnu.org)
f0d7658e13c42b64794712598504b47038ee9fb6.camel@librehacker.com
christopher@nightshade ~$ guix challenge ugrep
/gnu/store/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12 contents
differ:
local hash: 1yf9752hprihmhwnqlhaffkmwmkd3scwzr8dqy7d78vcy70qjkn6
: 1ifdjmscb07pacxwcs2408zi12yrxxf0zdwzz800xlccp5x53m7r
differing file:
/bin/ugrep

1 store items were analyzed:
- 0 (0.0%) were identical
- 1 (100.0%) differed
- 0 (0.0%) were inconclusive

--
Christopher Howard
Attachment: file
L
L
Leo Famulari wrote on 24 May 2021 19:22
(name . Christopher Howard)(address . christopher@librehacker.com)(address . 48591@debbugs.gnu.org)
YKvgz5XUkPoL9EMO@jasmine.lan
On Sat, May 22, 2021 at 12:19:30PM -0800, Christopher Howard wrote:
Toggle quote (8 lines)
> However, the command I am running to
> rebuild ugrep without grafts or substitutes is rebuilding everything
> all the way back to bootstrap, so it probably will not be completed for
> a day or two:
>
> guix gc -D $(guix build --no-grafts ugrep); guix build --no-substitutes
> ugrep

Do this instead:

$ guix environment --no-grafts ugrep -- guix build --no-grafts --no-substitutes ugrep

That way, you can use substitutes for ugrep's dependencies, but then
build ugrep from source.

I hope it helps!
R
R
raingloom wrote on 30 Jun 2021 23:41
(name . Leo Famulari)(address . leo@famulari.name)
20210630234111.26a7e7f7@riseup.net
On Mon, 24 May 2021 13:22:23 -0400
Leo Famulari <leo@famulari.name> wrote:

Toggle quote (22 lines)
> On Sat, May 22, 2021 at 12:19:30PM -0800, Christopher Howard wrote:
> > However, the command I am running to
> > rebuild ugrep without grafts or substitutes is rebuilding everything
> > all the way back to bootstrap, so it probably will not be completed
> > for a day or two:
> >
> > guix gc -D $(guix build --no-grafts ugrep); guix build
> > --no-substitutes ugrep
>
> Do this instead:
>
> $ guix environment --no-grafts ugrep -- guix build --no-grafts
> --no-substitutes ugrep
>
> That way, you can use substitutes for ugrep's dependencies, but then
> build ugrep from source.
>
> I hope it helps!
>
>
>

TIL about that. There really ought to be a more obvious way to do that.
L
L
Leo Famulari wrote on 2 Jul 2021 04:23
(name . raingloom)(address . raingloom@riseup.net)
YN54kgFT9LGxD4VO@jasmine.lan
On Wed, Jun 30, 2021 at 11:41:11PM +0200, raingloom wrote:
Toggle quote (2 lines)
> TIL about that. There really ought to be a more obvious way to do that.

Agreed! It's a common use case.
Z
Z
zimoun wrote on 2 Jul 2021 13:24
(name . Christopher Howard)(address . christopher@librehacker.com)(address . 48591@debbugs.gnu.org)
87o8blm00y.fsf@gmail.com
Hi,

On Sat, 22 May 2021 at 12:19, Christopher Howard <christopher@librehacker.com> wrote:

Toggle quote (2 lines)
> I'm sure this will also be a reproducibility bug,

Indeed, the package ’ugrep’ is not reproducible:

guix build ugrep --no-grafts -q
guix build ugrep --no-grafts -q --check -K
diff --no-dereference -r /gnu/store/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12{,-check}
Binary files /gnu/store/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12/bin/ugrep and /gnu/store/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12-check/bin/ugrep differ

The next step is to investigate with diffoscope.


Well, another data point – maybe Berlin has an issue – because I get:

guix challenge ugrep --substitute-urls="https://ci.guix.gnu.orghttps://bordeaux.guix.gnu.org"
/gnu/store/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12 contents differ:
local hash: 1ifdjmscb07pacxwcs2408zi12yrxxf0zdwzz800xlccp5x53m7r
https://ci.guix.gnu.org/nar/lzip/a6x7rv7xi3d2pw205znmnha9qfjrvs7k-ugrep-3.1.12:1ifdjmscb07pacxwcs2408zi12yrxxf0zdwzz800xlccp5x53m7r

guix hash -r $(guix build --no-grafts ugrep)-check
1yf9752hprihmhwnqlhaffkmwmkd3scwzr8dqy7d78vcy70qjkn6

I fetch by default the substitutes from ci that why “local hash“ and ci
are the same; but my local build with ‘guix build --check’ gives the
same hash as bordeaux.


All the best,
simon
Z
Z
zimoun wrote on 13 Jul 2021 10:45
control message for bug #48591
(address . control@debbugs.gnu.org)
86zguqob4l.fsf@gmail.com
retitle 48591 ugrep: reproducibility issues
quit
K
K
kiasoc5 wrote on 17 Jan 2023 02:37
Re: ugrep: reproducibility issues
(address . 48591@debbugs.gnu.org)
f88e8548-3555-372a-9043-2c03484b64be@disroot.org
Is this bug fixed yet?

If not, seeing that this issue was motivated by an illegal instruction,
we can use compile flags to disable SSE and other platform-dependent
optimizations and investigate further.

WDYT?

V
V
Vagrant Cascadian wrote on 24 Oct 21:04 +0200
Re: bug#48591: ugrep: reproducibility issues
(address . 48591@debbugs.gnu.org)
87y12d8gqa.fsf@wireframe
On 2023-01-16, kiasoc5 wrote:
Toggle quote (10 lines)
> Is this bug fixed yet?
>
> If not, seeing that this issue was motivated by an illegal instruction,
> we can use compile flags to disable SSE and other platform-dependent
> optimizations and investigate further.
>
> WDYT?
>
> https://github.com/Genivia/ugrep#other-platforms-step-3-build

--disable-sse2 disable SSE2 and AVX optimizations
--disable-avx2 disable AVX2 and AVX512BW optimizations, but compile with SSE2 when supported
--disable-neon disable ARM NEON/AArch64 optimizations

Seems like --disable-sse2 on x86_64 and x86 (32-bit) and --disable-neon
(armhf, maybe aarch64) would be needed to make the build independent of
which cpu it runs on.


FWIW, debian disables several of these (--disable-sse2 and
--disable-neon), although also passes --disable-avx which is not mentioned in
the upstream documentation, although is present in configure.ac...



live well,
vagrant
-----BEGIN PGP SIGNATURE-----

iHUEARYKAB0WIQRlgHNhO/zFx+LkXUXcUY/If5cWqgUCZxqaTgAKCRDcUY/If5cW
qrSCAQDZacQxjSHwtsdUf4W1AJUiushWSf8MukG6l8ScMbCRNgEAo8qrWXJMyBtN
RgGG3rNtQacLJT3oghbHIEjRguWimgM=
=DK8c
-----END PGP SIGNATURE-----

?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 48591@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 48591
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch