ci.guix.gnu.org narinfos with excessive NarSize

DoneSubmitted by Christopher Baines.
Details
4 participants
  • Brendan Tildesley
  • Ludovic Courtès
  • Christopher Baines
  • Mathieu Othacehe
Owner
unassigned
Severity
important
C
C
Christopher Baines wrote on 31 Jan 2021 15:47
(address . bug-guix@gnu.org)
87zh0pf9ip.fsf@cbaines.net
I noticed through the Guix Data Service that some narinfo files from
ci.guix.gnu.org have an excessive NarSize.

These are the three that I've found, but there could be more.


/gnu/store/qln574djfgl8h9glij9id8jips7nnrlw-flightgear-2018.3.5
NarSize: 18446744072099351584

/gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1
NarSize: 18446744071612438544

/gnu/store/wd9z64xpck56xzf52jwlpg8vb610b0ym-repeat-masker-4.1.1
NarSize: 18446744071612438544


There's additional information on IRC: http://logs.guix.gnu.org/guix/2021-01-31.log#152751

Cc'ing Mathieu in case this is related to the new offloading mechanism.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmAWww5fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XcpZxAAkBR7gl7a2vCi2Ag6iEZIGQrrDkWnE8KW
9IkLhfuDPJxvxo59Cu7D2EZ5cKCifC3xOtns3j76FngsbQOmMNXs2A6XzvNQA0jd
HiVLg/JF/MthrOupDo2mfflbkIu0wOxei5MS2JP3dVXHzlvsP4Tw1hsyydI/67Hw
dXa6r9ILvC61xjjx3Cq8kC4drQefpbtWWA3kMym0NJO7vLYGgKXAadZdJ6qsc9Zn
ZffA9xixua56tTy/nceRNyaAaXbSG/FLKEhLMGug8ryDq0NqR7WODzAzRa7RxPyy
wJAqQrGPtJ8BPnXJJNNc1L9QY6OnKT0C/Ayyi2AToE/nXp4DvC7iz+MQ6FnLK/7I
esHhzFEnAJW8f2XvKQVcmgnSEolJqgc8zo4WqjFzEsfiPJsFhovJO8kjNrTj4VY4
W7qkexm/65k4Sosjp7PimSaLgKWJcG2jUKwmGJUlUGMDD3RSSGV6gLtVxM4eG+lB
hbuvD0aW7wtzG15BORKCdv34RydvurKF1BHa80WbnTI8SNVmj5mfeFYVoS1Dq1rR
4K1fzEC6oworx34uyIvQMuDcE/Lq5DdqihjMYwFsrLbQHF2vdaeZa3enFwRK5xiH
3CtyrUmZmC0B3RHS5Kr8tRHkQAiYWFf6dO0sV2LQSBq5uhp9vuauCeHYd3o0zedv
0gVX/8By0q0=
=qZAJ
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 31 Jan 2021 16:17
control message for bug #46212
(address . control@debbugs.gnu.org)
874kixp240.fsf@gnu.org
severity 46212 important
quit
L
L
Ludovic Courtès wrote on 31 Jan 2021 16:20
Re: bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
(name . Christopher Baines)(address . mail@cbaines.net)
87y2g9nneh.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (9 lines)
> /gnu/store/qln574djfgl8h9glij9id8jips7nnrlw-flightgear-2018.3.5
> NarSize: 18446744072099351584
>
> /gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1
> NarSize: 18446744071612438544
>
> /gnu/store/wd9z64xpck56xzf52jwlpg8vb610b0ym-repeat-masker-4.1.1
> NarSize: 18446744071612438544

The key point here is that ‘narSize’ in the database is negative:

Toggle snippet (4 lines)
sqlite> select * from validpaths where path = '/gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1';
43262123|/gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1|sha256:33328e16d8d83dcf1a6e031598dbc517aff18e6c7ccd55f7894102bab55fcdb9|1611849907|/gnu/store/rr532q5fmwj1gafdgk6nhxg9khnbsw3z-repeat-masker-4.1.1.drv|-2097113072

The actual nar size in this case is just above 2³¹ so most likely we’re
seeing a signed integer wrapping error.

I believe this is a very recent regression (the registration data for
the item above is Jan. 28th); we have older store items with a correct
‘narSize’, such as

Thoughts?

Ludo’.
L
L
Ludovic Courtès wrote on 1 Feb 2021 10:15
(name . Christopher Baines)(address . mail@cbaines.net)
87mtwoi1wz.fsf@gnu.org
Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (2 lines)
> The key point here is that ‘narSize’ in the database is negative:

With commit 13a7d2a538b00aa0a8cf9b999f1a4ff3e5959af9, ‘register-items’ &
co. will now detect invalid nar size values early on.

Ludo’.
C
C
Christopher Baines wrote on 1 Feb 2021 20:57
Re: ci.guix.gnu.org narinfos with excessive NarSize
(address . 46212@debbugs.gnu.org)
87r1lzftnx.fsf@cbaines.net
Christopher Baines <mail@cbaines.net> writes:

Toggle quote (15 lines)
> I noticed through the Guix Data Service that some narinfo files from
> ci.guix.gnu.org have an excessive NarSize.
>
> These are the three that I've found, but there could be more.
>
>
> /gnu/store/qln574djfgl8h9glij9id8jips7nnrlw-flightgear-2018.3.5
> NarSize: 18446744072099351584
>
> /gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1
> NarSize: 18446744071612438544
>
> /gnu/store/wd9z64xpck56xzf52jwlpg8vb610b0ym-repeat-masker-4.1.1
> NarSize: 18446744071612438544

Guix gives the following error when it encounters one of these bad
narinfos:

error: integer expected from stream
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmAYXRJfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XfnohAAuNiVFQGrybaWryk6qXtAq2nfelX0wx29
Ew5acRsqz3X8lXLQhsZqKM3JLK3jpuDfsgkhI5m2HLlILkwbq4VgMAgyHNCyASZU
tjcghSEZ6inUmBe5S5V+GrCR7cvrh3pwL6X/XFxACdQzaRTtc5BfjS/tzt/gRIbN
Yc4V/Avq4d7UhkNBTKsghiYeJz/hl1KDCvpEwFqdS1nc57RlaPBEfJ6LJ7geervD
LRqg7dGx19A8lULFE2myJEyF9rkuNOgFSlMR4hoxwbi3qURyad77roFC6qf70zzj
y4nGeQLQf6hFQPyLJusRraQc0KZsQWrBoutAcjcnE0Dkn30nEIDw2BsxsWk/ttUx
xgZF+f7pWbrYNHxtakRoaO/Y0HRxfFkcq8VnHPJmiZJFLWPA7XwBOVmw8gG17Vh5
YfNvIT7y5xvaI0FvFFMeRPiLCy2pjEfMRWE46HHaP2HJmCobLy/UinuB24kGVIHS
no8WCgn3rvb9ABWnKygHkT1Du+mHd2ldh27uQvxvUJltcYChrgQWm+C1gUUplwg4
J6VHeh+J/uqAvWQD9QlLjCsh4TODm1YT36E4leWvRHhvpkgoJgpd1GfltmBhpClr
FeXDtKzFzjlcFj62a6qJP3KU5sPbGjxAjILAWMBCIQVteYHa2iP6Wl/Q8LRhhk8q
XgNowRKvtto=
=TA04
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 2 Feb 2021 22:48
Re: bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
(name . Christopher Baines)(address . mail@cbaines.net)(address . 46212@debbugs.gnu.org)
87eehycf9w.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (22 lines)
> Christopher Baines <mail@cbaines.net> writes:
>
>> I noticed through the Guix Data Service that some narinfo files from
>> ci.guix.gnu.org have an excessive NarSize.
>>
>> These are the three that I've found, but there could be more.
>>
>>
>> /gnu/store/qln574djfgl8h9glij9id8jips7nnrlw-flightgear-2018.3.5
>> NarSize: 18446744072099351584
>>
>> /gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1
>> NarSize: 18446744071612438544
>>
>> /gnu/store/wd9z64xpck56xzf52jwlpg8vb610b0ym-repeat-masker-4.1.1
>> NarSize: 18446744071612438544
>
> Guix gives the following error when it encounters one of these bad
> narinfos:
>
> error: integer expected from stream

I guess ‘guix substitute --query’ reads the narinfo, passes the negative
integer as is to the daemon (for ‘query-path-info’ RPCs), which
rightfully complains.

It would be nice for ‘guix publish’ to not emit broken narinfos, and
perhaps we can also add extra checks in the (guix narinfo) reader.

Ludo’.
L
L
Ludovic Courtès wrote on 19 Feb 2021 16:11
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87blcgm6st.fsf@gnu.org
Hi Mathieu,

Did you eventually find out where the negative size comes from?


What should we do in your opinion with database entries that have a
negative size?

Thanks,
Ludo’.

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (27 lines)
> Christopher Baines <mail@cbaines.net> skribis:
>
>> /gnu/store/qln574djfgl8h9glij9id8jips7nnrlw-flightgear-2018.3.5
>> NarSize: 18446744072099351584
>>
>> /gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1
>> NarSize: 18446744071612438544
>>
>> /gnu/store/wd9z64xpck56xzf52jwlpg8vb610b0ym-repeat-masker-4.1.1
>> NarSize: 18446744071612438544
>
> The key point here is that ‘narSize’ in the database is negative:
>
> sqlite> select * from validpaths where path = '/gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1';
> 43262123|/gnu/store/qhix6afvy2a6n7hlx4qgdns461p8kdnv-repeat-masker-4.1.1|sha256:33328e16d8d83dcf1a6e031598dbc517aff18e6c7ccd55f7894102bab55fcdb9|1611849907|/gnu/store/rr532q5fmwj1gafdgk6nhxg9khnbsw3z-repeat-masker-4.1.1.drv|-2097113072
>
> The actual nar size in this case is just above 2³¹ so most likely we’re
> seeing a signed integer wrapping error.
>
> I believe this is a very recent regression (the registration data for
> the item above is Jan. 28th); we have older store items with a correct
> ‘narSize’, such as
> <https://ci.guix.gnu.org/nm6w84c9zj3yiylal3dk1sqzxq11sjzw.narinfo>.
>
> Thoughts?
>
> Ludo’.
M
M
Mathieu Othacehe wrote on 22 Feb 2021 09:59
(name . Ludovic Courtès)(address . ludo@gnu.org)
87blcciike.fsf@gnu.org
Hey Ludo,

Toggle quote (7 lines)
> Did you eventually find out where the negative size comes from?
>
> https://issues.guix.gnu.org/46212
>
> What should we do in your opinion with database entries that have a
> negative size?

I didn't look closely to this problem yet. However, I fixed an issue
with locales in the remote building mechanism that caused publish server
crashes:

That's maybe somehow related.

Thanks,

Mathieu
L
L
Ludovic Courtès wrote on 22 Feb 2021 14:03
(name . Mathieu Othacehe)(address . othacehe@gnu.org)
87a6rwb6go.fsf@gnu.org
Hi,

Mathieu Othacehe <othacehe@gnu.org> skribis:

Toggle quote (12 lines)
>> Did you eventually find out where the negative size comes from?
>>
>> https://issues.guix.gnu.org/46212
>>
>> What should we do in your opinion with database entries that have a
>> negative size?
>
> I didn't look closely to this problem yet. However, I fixed an issue
> with locales in the remote building mechanism that caused publish server
> crashes:
> https://lists.gnu.org/archive/html/bug-guix/2021-02/msg00231.html.

Hmm I don’t think so.

The bug here is likely due to 32-bit signed integer wrapping. That can
only happen in C code, so to me possible culprits would be
guile-simple-zmq or the layer above it (if there’s a binary protocol
involved) or the postgresql interface. Only a vague intuition, though.

Ludo’.
M
M
Mathieu Othacehe wrote on 24 Feb 2021 11:26
(name . Ludovic Courtès)(address . ludo@gnu.org)
87blc9oj70.fsf@gnu.org
Hey,

Toggle quote (5 lines)
> The bug here is likely due to 32-bit signed integer wrapping. That can
> only happen in C code, so to me possible culprits would be
> guile-simple-zmq or the layer above it (if there’s a binary protocol
> involved) or the postgresql interface. Only a vague intuition, though.

Hmm, looks like you're right! There's a memory corruption in the
remote-server process that's really hard to reproduce. I suspect the ZMQ
library or its Guile bindings.

I'm trying to valgrind the process to identify the issue, without much
success for now.

Thanks,

Mathieu
L
L
Ludovic Courtès wrote on 21 Mar 2021 15:38
(name . Christopher Baines)(address . mail@cbaines.net)
874kh43725.fsf@gnu.org
Hi there!

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (5 lines)
> Guix gives the following error when it encounters one of these bad
> narinfos:
>
> error: integer expected from stream

I get this while attempting to install flightgear from
12c17b46f1464db5591572754c814c845a2057ad.

It has the scary NarSize:

Toggle snippet (12 lines)
$ wget -qO - https://ci.guix.gnu.org/k094mjy3m1cyh85vd27l2gn0w5ikscya.narinfo | head -9
StorePath: /gnu/store/k094mjy3m1cyh85vd27l2gn0w5ikscya-flightgear-2018.3.5
URL: nar/gzip/k094mjy3m1cyh85vd27l2gn0w5ikscya-flightgear-2018.3.5
Compression: gzip
FileSize: 1797450320
URL: nar/lzip/k094mjy3m1cyh85vd27l2gn0w5ikscya-flightgear-2018.3.5
Compression: lzip
FileSize: 1605734577
NarHash: sha256:0bwyhvqr7l5sk6fckymj41bivm6ws09aid8lg6lg0bmy513kn0aw
NarSize: 18446744072099352280

The nar was baked on March 17th:

Toggle snippet (4 lines)
ludo@berlin ~$ ls -l /var/cache/guix/publish/lzip/k094mjy3m1cyh85vd27l2gn0w5ikscya-flightgear-2018.3.5.narinfo
-rw-r--r-- 2 guix-publish guix-publish 2232 Mar 17 22:20 /var/cache/guix/publish/lzip/k094mjy3m1cyh85vd27l2gn0w5ikscya-flightgear-2018.3.5.narinfo

The nar size is indeed negative again:

Toggle snippet (7 lines)
ludo@berlin ~$ sudo sqlite3 /var/guix/db/db.sqlite
SQLite version 3.28.0 2019-04-16 19:49:53
Enter ".help" for usage hints.
sqlite> select * from validpaths where path='/gnu/store/k094mjy3m1cyh85vd27l2gn0w5ikscya-flightgear-2018.3.5';
45484766|/gnu/store/k094mjy3m1cyh85vd27l2gn0w5ikscya-flightgear-2018.3.5|sha256:5c013b4728be2ef0a87914b5a812d0dcd41d5720b2fac99c99bad093f1869e2f|1615825221|/gnu/store/r995zx36fm4k4r1mwyy22mg2syjdc0fj-flightgear-2018.3.5.drv|-1610199336

Registration date is:

Toggle snippet (4 lines)
scheme@(guile-user)> (date->string (time-utc->date (make-time time-utc 0 1615825221) 3600))
$11 = "Mon Mar 15 17:20:21+0100 2021"

Right now berlin is running guix-1.2.0-13.a53f711, which AFAICS includes
the narsize bound checking added in 13a7d2a538b00aa0a8cf9b999f1a4ff3e5959af9.

This only ensures that narsize is positive. Could it be that wrapping
happens in sqlite itself? Or could it be that Cuirass itself uses an
older version of (guix store database)?

Ludo’.
L
L
Ludovic Courtès wrote on 21 Mar 2021 15:55
(name . Christopher Baines)(address . mail@cbaines.net)
87y2eg1rqy.fsf@gnu.org
Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (3 lines)
> I get this while attempting to install flightgear from
> 12c17b46f1464db5591572754c814c845a2057ad.

Ah ha! A newer flightgear, baked today, doesn’t have this problem (from
commit 3c74dbb35f1ea0f43b02fd81e5afd24608f71c45):

Toggle snippet (15 lines)
$ wget -qO - https://ci.guix.gnu.org/58rpdzfh42xqq36dg25cgxzfw0alsm7w.narinfo | head -12
StorePath: /gnu/store/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5
URL: nar/gzip/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5
Compression: gzip
FileSize: 1797450279
URL: nar/lzip/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5
Compression: lzip
FileSize: 1605732816
URL: nar/zstd/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5
Compression: zstd
FileSize: 1668702713
NarHash: sha256:09804lw134f16k4pwwm8jpwiy9dvr4hjvz64wx74bl8g1jjf9i9w
NarSize: 2684767960

And on berlin:

Toggle snippet (8 lines)
ludo@berlin ~$ sudo sqlite3 /var/guix/db/db.sqlite
Password:
SQLite version 3.28.0 2019-04-16 19:49:53
Enter ".help" for usage hints.
sqlite> select * from validpaths where path='/gnu/store/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5';
45741398|/gnu/store/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5|sha256:3cc5e4a40c0fd1454ee7c4fc2d21c9bb251ff995a8727ec934c1911138250025|1616252839|/gnu/store/rzcviskbmy0x4hz15dki4a2aci9wrjxv-flightgear-2018.3.5.drv|2684767960

Registration date:

Toggle snippet (4 lines)
scheme@(guile-user)> (date->string (time-utc->date (make-time time-utc 0 1616252839) 3600))
$15 = "Sat Mar 20 16:07:19+0100 2021"

Could it be that the bug was fixed in the meantime? Or that this one
was, say, built directly via Guix whereas the other one was built
through Cuirass? Mystery!

Ludo’.
L
L
Ludovic Courtès wrote on 21 Mar 2021 15:55
(name . Christopher Baines)(address . mail@cbaines.net)
87wnu01rq6.fsf@gnu.org
Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (3 lines)
> I get this while attempting to install flightgear from
> 12c17b46f1464db5591572754c814c845a2057ad.

Ah ha! A newer flightgear, baked today, doesn’t have this problem (from
commit 3c74dbb35f1ea0f43b02fd81e5afd24608f71c45):

Toggle snippet (15 lines)
$ wget -qO - https://ci.guix.gnu.org/58rpdzfh42xqq36dg25cgxzfw0alsm7w.narinfo | head -12
StorePath: /gnu/store/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5
URL: nar/gzip/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5
Compression: gzip
FileSize: 1797450279
URL: nar/lzip/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5
Compression: lzip
FileSize: 1605732816
URL: nar/zstd/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5
Compression: zstd
FileSize: 1668702713
NarHash: sha256:09804lw134f16k4pwwm8jpwiy9dvr4hjvz64wx74bl8g1jjf9i9w
NarSize: 2684767960

And on berlin:

Toggle snippet (8 lines)
ludo@berlin ~$ sudo sqlite3 /var/guix/db/db.sqlite
Password:
SQLite version 3.28.0 2019-04-16 19:49:53
Enter ".help" for usage hints.
sqlite> select * from validpaths where path='/gnu/store/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5';
45741398|/gnu/store/58rpdzfh42xqq36dg25cgxzfw0alsm7w-flightgear-2018.3.5|sha256:3cc5e4a40c0fd1454ee7c4fc2d21c9bb251ff995a8727ec934c1911138250025|1616252839|/gnu/store/rzcviskbmy0x4hz15dki4a2aci9wrjxv-flightgear-2018.3.5.drv|2684767960

Registration date:

Toggle snippet (4 lines)
scheme@(guile-user)> (date->string (time-utc->date (make-time time-utc 0 1616252839) 3600))
$15 = "Sat Mar 20 16:07:19+0100 2021"

Could it be that the bug was fixed in the meantime? Or that this one
was, say, built directly via Guix whereas the other one was built
through Cuirass? Mystery!

Ludo’.
M
M
Mathieu Othacehe wrote on 29 Mar 2021 09:45
(name . Ludovic Courtès)(address . ludo@gnu.org)
8735we2yj5.fsf@gnu.org
Hey,

Toggle quote (4 lines)
> Could it be that the bug was fixed in the meantime? Or that this one
> was, say, built directly via Guix whereas the other one was built
> through Cuirass? Mystery!

That's strange. There's nothing really special about how Cuirass builds
its stuff. It's a plain "build-derivations" call in the "cuirass
remote-worker" process. This process only builds stuff and report it
using (simple-zmq).

A memory corruption in that module also seems unlikely, as there are
almost 30 instances of this process running nicely for days.

Thanks,

Mathieu
B
B
Brendan Tildesley wrote on 7 Apr 2021 11:09
ci.guix.gnu.org narinfos with excessive NarSize
(name . 46212@debbugs.gnu.org)(address . 46212@debbugs.gnu.org)(name . ludo@gnu.org)(address . ludo@gnu.org)
839241594.45677.1617786564789@office.mailbox.org
I just had the same bug with flightgear as Ludo.

Running `guix build flightgear', guix showed it was downloading substitute information, and then errored with

guix build: error: integer expected from stream

Possibly helpful strace info:

write(14, "\36\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0?\0\0\0\0\0\0\0/gnu/store/ssdka48kwx9f2z54j63mjxvn6bg502mm-flightgear-2018.3.5\0@\0\0\0\0\0\0\0/gnu/store/pr6xy604hg7yhcinnlymrkc958kl0bnn-openscenegraph-3.4.1<\0\0\0\0\0\0\0/gnu/store/g8g4paw3q3z8hkl441ddhdn0in54gi3f-simgear-2018.3.5\0\0\0\0", 232) = 232
read(14, "ptxc\0\0\0\0", 8) = 8
read(14, "\34\0\0\0\0\0\0\0", 8) = 8
read(14, "integer expected from stream\0\0\0\0", 32) = 32
read(14, "\1", 1) = 1
read(14, "\0\0\0\0\0\0\0", 7) = 7
close(14)
Attachment: file
C
C
Christopher Baines wrote on 11 Jun 2021 14:04
(name . Christopher Baines)(address . mail@cbaines.net)
87tum4sj45.fsf@cbaines.net
Christopher Baines <mail@cbaines.net> writes:

Toggle quote (3 lines)
> I noticed through the Guix Data Service that some narinfo files from
> ci.guix.gnu.org have an excessive NarSize.

Since I've scanned all the narinfos for a recent revision, I thought I'd
update this issue with the broken narinfos that ci.guix.gnu.org is
currently serving (just for a recent revision):

/gnu/store/s18pfbv3a9sf61rh29m1i8vdznwjlny7-repeat-masker-4.1.1
/gnu/store/4lm1wa8m9n84vh29d6pa3ygy29iwyz2v-flightgear-2018.3.5
/gnu/store/q6qa5s2z2l20q25qpxyfv1wni5wwhk24-flightgear-2018.3.5

I think there were safeguards put in place to avoid bad data making it's
way in to the store database? If so, maybe the store item could be gc'd
on berlin, and the cached narinfos deleted, and that would clean things
up?
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmDDUTpfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xf/jg//WBQ5JtPXC6XYVRxbYXcPPsukdSYd8OhH
+eJlTfsat/xcX60Ny1qsOrIwwzxqEml1re01Dt2u3pvG4Hv9fUnMlKH6Utolxt71
LYw+X4LEdIERjHKWVweAEHVG6OzffKWvhMDU5ZS2TeS9Y1JXaNeYhU3/mYbBh9au
drtgsaG6iQh1i9b3XLaCxD+Z0mToQmuRD8o5afGovmp4gefkU42znhn25qdBuKwm
hx1GS+lRBzR3CKIRoejWYNA4fP25x52gFwycmqw3LYLsKFjxoBB9q9YS0qwWuHfN
WhkH/jgbrH9RJI5XYpbFI+snSq6Xa5MagMXNEE0GXBapJGFGuLTSk77zC6R0TYM8
baobKJfrB2C9YqVo98BONL6cATDzXuD6zat3BUTVNHy+Ka+pbytCt8DEeil+QWh9
l1kN/bHOxdObk7kPy+c97e3F6uXDsb7QOW6bm5SwyPLkhlaAwgb8YqW8yGeqoWpE
zFxhjZcMyvKqMq6x03FMAwyP/2G0HOrr6WEsiouktRbggxt3ANc2BMwJOXj/HPgH
uh+I+jsqPPD/DRZX1oF/V+DuRT9EbC/TuLydjwKQ5B38rJq6B+OigLVWcG/93k8H
7Umb2tXZwgoIbmYvtpfdB4zVEoOauuAmCGhblFjTITW6EebTIWOfKSa0ehnS7l12
7u7zoC5+MXs=
=us0O
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 18 Jun 2021 11:48
(name . Christopher Baines)(address . mail@cbaines.net)
87pmwjh5ao.fsf@gnu.org
Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (13 lines)
> Christopher Baines <mail@cbaines.net> writes:
>
>> I noticed through the Guix Data Service that some narinfo files from
>> ci.guix.gnu.org have an excessive NarSize.
>
> Since I've scanned all the narinfos for a recent revision, I thought I'd
> update this issue with the broken narinfos that ci.guix.gnu.org is
> currently serving (just for a recent revision):
>
> /gnu/store/s18pfbv3a9sf61rh29m1i8vdznwjlny7-repeat-masker-4.1.1
> /gnu/store/4lm1wa8m9n84vh29d6pa3ygy29iwyz2v-flightgear-2018.3.5
> /gnu/store/q6qa5s2z2l20q25qpxyfv1wni5wwhk24-flightgear-2018.3.5

This is what I get for the last one:

Toggle snippet (4 lines)
sqlite> select * from validpaths where path = '/gnu/store/q6qa5s2z2l20q25qpxyfv1wni5wwhk24-flightgear-2018.3.5';
48732920|/gnu/store/q6qa5s2z2l20q25qpxyfv1wni5wwhk24-flightgear-2018.3.5|sha256:86426115e1f49835a24612ae89a180e87ac835332b4e83d19ecbd43211336cd2|1623360391|/gnu/store/722yyaa1qvkaakn6p7ywwr4dnm5wmddz-flightgear-2018.3.5.drv|-1610199336

The registration date is:

Toggle snippet (5 lines)
scheme@(guile-user)> ,use(srfi srfi-19)
scheme@(guile-user)> (date->string (time-utc->date (make-time time-utc 0 1623360391)) "~1" )
$164 = "2021-06-10"

Toggle quote (3 lines)
> I think there were safeguards put in place to avoid bad data making it's
> way in to the store database?

I thought so too (commit 13a7d2a538b00aa0a8cf9b999f1a4ff3e5959af9)!
So we must be using another code path.

Mathieu, these store items get registered via an RPC, or via (guix store
database)?

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 25 Nov 2021 00:24
Re: bug#51983: guix build: error: integer expected from stream
(name . Tobias Geerinckx-Rice)(address . me@tobias.gr)
877dcxgms7.fsf@gnu.org
Hi,

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

Toggle quote (8 lines)
> Tobias Geerinckx-Rice via Bug reports for GNU Guix 写道:
>> substitute: updating substitutes from
>> 'https://ci.guix.gnu.org'... 100.0%
>> guix build: error: integer expected from stream:
>> 18446744071924943736
>
> And it is indeed specific to querying ci.guix.

This reminds me of:


and indeed:

Toggle snippet (12 lines)
$ guix build flightgear -n --no-grafts
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bayfront.guix.gnu.org'... 100.0%
guix build: error: integer expected from stream
$ guix describe
Generacio 195 Nov 22 2021 00:37:58 (nuna)
guix 24aa7b3
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 24aa7b3c21309b63cc6e8e18d6417d2cddccf6c6

The ‘NarSize’ field is still broken on ci.guix:

Toggle snippet (10 lines)
$ wget -qO - https://ci.guix.gnu.org/ashmi6vr9d0k03z9q6ncbb2wf36p9nyk.narinfo |grep Size
FileSize: 1876560180
FileSize: 1715771086
FileSize: 1753645085
NarSize: 18446744072093825552
$ wget -qO - https://bordeaux.guix.gnu.org/ashmi6vr9d0k03z9q6ncbb2wf36p9nyk.narinfo |grep Size
FileSize: 1715771086
NarSize: 2679241232

and again the database on berlin has a negative size:

Toggle snippet (7 lines)
$ sudo sqlite3 /var/guix/db/db.sqlite
SQLite version 3.32.3 2020-06-18 14:00:33
Enter ".help" for usage hints.
sqlite> select * from validpaths where path = '/gnu/store/ashmi6vr9d0k03z9q6ncbb2wf36p9nyk-flightgear-2020.3.11';
56551239|/gnu/store/ashmi6vr9d0k03z9q6ncbb2wf36p9nyk-flightgear-2020.3.11|sha256:1c68105efd8ee4677085a2cecaa5859f1f5f213d52e213ef4949384717dc493a|1636968959|/gnu/store/xvyar43mcfw0r7jfm1x0a2i9rhnsdh87-flightgear-2020.3.11.drv|-1615726064

[Time passes…]

OK, this one is now fixed by commit
1eb40a6dc4917f5a2e915f8b31b750dba3d378c6m, which means you now get:

Toggle snippet (6 lines)
$ guix build flightgear --no-grafts -n
1,715.8 MB would be downloaded:
/gnu/store/ashmi6vr9d0k03z9q6ncbb2wf36p9nyk-flightgear-2020.3.11
guix build: warning: at least 18,446,744,072,093.8 MB needed but only 4,753.4 MB available in /gnu/store

Fun, no? :-)

Also, after substitution, the size in the store database would be the
wrong one, so ‘guix size’ would keep telling you it’s this big. Not
great but harmless as the nar size is used for UI purposes only.

Ludo’.
L
L
Ludovic Courtès wrote on 25 Nov 2021 00:28
Re: bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
(name . Christopher Baines)(address . mail@cbaines.net)
87zgptf806.fsf@gnu.org
Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (18 lines)
> This is what I get for the last one:
>
> sqlite> select * from validpaths where path = '/gnu/store/q6qa5s2z2l20q25qpxyfv1wni5wwhk24-flightgear-2018.3.5';
> 48732920|/gnu/store/q6qa5s2z2l20q25qpxyfv1wni5wwhk24-flightgear-2018.3.5|sha256:86426115e1f49835a24612ae89a180e87ac835332b4e83d19ecbd43211336cd2|1623360391|/gnu/store/722yyaa1qvkaakn6p7ywwr4dnm5wmddz-flightgear-2018.3.5.drv|-1610199336
>
>
> The registration date is:
>
> scheme@(guile-user)> ,use(srfi srfi-19)
> scheme@(guile-user)> (date->string (time-utc->date (make-time time-utc 0 1623360391)) "~1" )
> $164 = "2021-06-10"
>
>> I think there were safeguards put in place to avoid bad data making it's
>> way in to the store database?
>
> I thought so too (commit 13a7d2a538b00aa0a8cf9b999f1a4ff3e5959af9)!
> So we must be using another code path.

At last I found the culprit (me! :-)). This is fixed by commit
f9b1bb916c284bea00dd5549a43e0894b219d650.

The reason ci.guix would experience it and not bayfront is because the
Cuirass worker mechanism relies on substitutes to retrieve files from a
worker, and this is how it would end up storing a negative size.

We’ll have to upgrade the ‘guix’ package and deploy it on berlin. I
don’t think it’s easily feasible to fix existing entries in the store
database and (more importantly) narinfos so I’d just leave them around;
the bogus nar size propagates but it’s harmless and doesn’t prevent
substitution.

Thoughts?

Ludo’.
L
L
Ludovic Courtès wrote on 25 Nov 2021 00:29
control message for bug #46212
(address . control@debbugs.gnu.org)
87y25df7zt.fsf@gnu.org
close 46212
quit
C
C
Christopher Baines wrote on 25 Nov 2021 09:42
Re: bug#46212: ci.guix.gnu.org narinfos with excessive NarSize
(name . Ludovic Courtès)(address . ludo@gnu.org)
875ysg39s2.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (25 lines)
> Hi,
>
> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> This is what I get for the last one:
>>
>> sqlite> select * from validpaths where path = '/gnu/store/q6qa5s2z2l20q25qpxyfv1wni5wwhk24-flightgear-2018.3.5';
>> 48732920|/gnu/store/q6qa5s2z2l20q25qpxyfv1wni5wwhk24-flightgear-2018.3.5|sha256:86426115e1f49835a24612ae89a180e87ac835332b4e83d19ecbd43211336cd2|1623360391|/gnu/store/722yyaa1qvkaakn6p7ywwr4dnm5wmddz-flightgear-2018.3.5.drv|-1610199336
>>
>>
>> The registration date is:
>>
>> scheme@(guile-user)> ,use(srfi srfi-19)
>> scheme@(guile-user)> (date->string (time-utc->date (make-time time-utc 0 1623360391)) "~1" )
>> $164 = "2021-06-10"
>>
>>> I think there were safeguards put in place to avoid bad data making it's
>>> way in to the store database?
>>
>> I thought so too (commit 13a7d2a538b00aa0a8cf9b999f1a4ff3e5959af9)!
>> So we must be using another code path.
>
> At last I found the culprit (me! :-)). This is fixed by commit
> f9b1bb916c284bea00dd5549a43e0894b219d650.

Awesome :)

Toggle quote (12 lines)
> The reason ci.guix would experience it and not bayfront is because the
> Cuirass worker mechanism relies on substitutes to retrieve files from a
> worker, and this is how it would end up storing a negative size.
>
> We’ll have to upgrade the ‘guix’ package and deploy it on berlin. I
> don’t think it’s easily feasible to fix existing entries in the store
> database and (more importantly) narinfos so I’d just leave them around;
> the bogus nar size propagates but it’s harmless and doesn’t prevent
> substitution.
>
> Thoughts?

I thought this did prevent substitution, at least I have problems:

→ guix build --substitute-urls=https://ci.guix.gnu.org/gnu/store/q6qa5s2z2l20q25qpxyfv1wni5wwhk24-flightgear-2018.3.5
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
guix build: error: integer expected from stream

Maybe the database could be queried to find the affected items, then
they could be GC'd?
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmGfTL1fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9Xc0tw/+MYQlRA8NY0RSRbiOPWZwpeqh88OjwDkd
DxGPISERNI/JE+aIYwf4YUHr1cIlY6IUZV4os4F+yq/C03VbGU24vmFklyTjOztA
h1cmW65srs4xMCEpqbXZMG0dvolQKtIM+k7oVX27OJfMFB6wbfmJp+rJ9oT/jBrC
GtcKKr55wBkuJ4YgYGf7vE56eL48fXf+g8wq35HcLgyKjnH3pSndXRYpeIVsUUIa
PTVVvbO9Qtlh8NvaMxgjgnFlTCHPnvkWpehqt0TflhRHxAThn9ErqP8fjWcSmXDV
P/Y+plo4fg2mmaUOe6pr/HE+5AxKU681RQcq17NDwHcck9GHBgwNXeq6z5CHCD+w
PLBPPBgNKgR+xBkIuWSIiEjMxmiIRvlF4vT85fkqUe/IOlkGJG8Xa0rMVpuQ8xsg
ORpH0DxnZj8dOL9qurmoLnk0vs6VekyHt5SDAAuulyey/3sLvCfLXMGM3xTVANZN
d6LD9O/wihJv6OEUn+VnCCNjfg1evRKx6tTSgH9UdH92yVhcR6JM5WVd1HS0NVRC
queIxiMIN/HQUltmZVfZRcmCcBFoG3VPhsopWs2jJ2KmBtNANSu7AQJ7CFEijs3q
TKXMoWiujj+8ZL4dYQiPkaTaEbJ1UNkc/xhiTfkqGkwPOyR1ua8YXptF96jXVrej
1ffHTB9Wq8g=
=llN1
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 26 Nov 2021 11:11
(name . Christopher Baines)(address . mail@cbaines.net)
87y25b9qg3.fsf@gnu.org
Hi!

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (6 lines)
> I thought this did prevent substitution, at least I have problems:
>
> → guix build --substitute-urls=https://ci.guix.gnu.org /gnu/store/q6qa5s2z2l20q25qpxyfv1wni5wwhk24-flightgear-2018.3.5
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> guix build: error: integer expected from stream

This one is fixed here:


We have yet to update the ‘guix’ package though.

Toggle quote (3 lines)
> Maybe the database could be queried to find the affected items, then
> they could be GC'd?

We could do that, but then /var/cache/guix/publish would needed to be
cleaned up as well. It’s a bit tedious doable. (I don’t plan to work
on it right now because I’m trying to focus on other things.)

Thanks,
Ludo’.
?
Your comment

This issue is archived.

To comment on this conversation send email to 46212@debbugs.gnu.org