DrRacket internal error uncompressing

OpenSubmitted by Jack Hill.
Details
5 participants
  • Jack Hill
  • jackhill
  • Malte Gerdes
  • Mark H Weaver
  • Philip McGrath
Owner
unassigned
Severity
normal
J
J
Jack Hill wrote on 11 Mar 2021 07:21
(address . bug-guix@gnu.org)
alpine.DEB.2.21.2103110058320.8138@marsh.hcoop.net
Hi Guix,

Using guix 80739ea480a7db667b83b45e3a08be740449f689 on x86_64, running:

`guix environment --ad-hoc racket -- drracket` produces:

```
$bytevector-uncompress: internal error uncompressing #"\0\0\0\0chez\310\224\206:\r()#\201\256R-d\205\233\24\363\5\20\201P\6A\v\300\0\16\f\6\31\2\f\6\f&H\275\0\1\0\362\bA\377e\0\1\0C\6A\21\3\v\300\0\201\265!\f\6\n\0\a\1\35\0\1+\0\360\27\201\375\300\0\0\0\17\205\210Z\0\0M\215\245\b\4\0\0M9fH\17\206fZ\0\0I\2...
context...:
body of "/gnu/store/mmrax3f1vx3c8ih9hhgffpvka6chk96w-racket-8.0/share/racket/pkgs/gui-lib/mred/private/wx/gtk/utils.rkt"
body of "/gnu/store/mmrax3f1vx3c8ih9hhgffpvka6chk96w-racket-8.0/share/racket/pkgs/gui-lib/mred/private/wx/platform.rkt"
```

However, `guix time-machine --commit=b1248016e0492e1897f4d1127ccb07736c9bb6a5 -- environment --ad-hoc racket -- drracket`

successfully opens drracket. b1248016e0492e1897f4d1127ccb07736c9bb6a5 is
the commit before the Racket 8.0 update, so I think the update introduced
the problem. Other Racket GUI programs like slideshow and gracket also
also exhibit the problem. The command line REPL works however.

I have not investigated to determine if the problem is specific to the GUI
packages or if loading other extra Racket packages would also exhibit the
problem.

Best,
Jack
J
J
Jack Hill wrote on 15 Mar 2021 05:53
(address . 47064@debbugs.gnu.org)
alpine.DEB.2.21.2103150050440.8138@marsh.hcoop.net
I wrote to the racket-users [0] list to see if they have advice on what
could be causing this problem.


Best,
Jack
J
J
Jack Hill wrote on 15 Mar 2021 22:05
FYI: DrRacket problem
(name . Philip McGrath)(address . philip@philipmcgrath.com)(address . 47064@debbugs.gnu.org)
alpine.DEB.2.21.2103151703110.8138@marsh.hcoop.net
Philip,

Thanks for the work you've been doing on improving our Racket package. I
wanted to make sure you were aware of the failure of DrRacket and other
graphical Racket components to load in the latest Racket package.


Best,
Jack
P
P
Philip McGrath wrote on 15 Mar 2021 22:38
Re: [racket-users] bytevector-uncompress: internal error uncompressing
07f46e79-4fdb-49ab-63bd-2c106db077d1@philipmcgrath.com
Jack, thanks for the pointer: I hadn't noticed this earlier.

On 3/15/21 12:49 AM, Jack Hill wrote:
Toggle quote (10 lines)
> ```
> $bytevector-uncompress: internal error uncompressing
> #"\0\0\0\0chez\310\224\206:\r()#\201\256R-d\205\233\24\363\5\20\201P\6A\v\300\0\16\f\6\31\2\f\6\f&H\275\0\1\0\362\bA\377e\0\1\0C\6A\21\3\v\300\0\201\265!\f\6\n\0\a\1\35\0\1+\0\360\27\201\375\300\0\0\0\17\205\210Z\0\0M\215\245\b\4\0\0M9fH\17\206fZ\0\0I\2...
>   context...:
>    body of
> "/gnu/store/bbnhjamch9125c412bl7ybf28a0jxrkd-racket-8.0/share/racket/pkgs/gui-lib/mred/private/wx/gtk/utils.rkt"
>    body of
> "/gnu/store/bbnhjamch9125c412bl7ybf28a0jxrkd-racket-8.0/share/racket/pkgs/gui-lib/mred/private/wx/platform.rkt"
> ```

This appears to be an error uncompressing Racket's compiled code—whether
generated by the Racket layer or the Chez Scheme layer, I'm not sure.
The stack trace is pointing to the module `mred/private/wx/gtk/utils`,
which is here:

At first glance, the most unusual-looking thing about that module is the
use of `(only-in '#%foreign ctype-c->scheme)` … but it could also be
something else altogether. I'll try to investigate further, but thoughts
from others would be very welcome!

-Philip
Attachment: file
P
P
Philip McGrath wrote on 15 Mar 2021 23:26
79c97b4c-defb-3966-dc50-a79fa8f5c053@philipmcgrath.com
I can reproduce the error with just:
```
$ racket -l mred/private/wx/gtk/utils
$bytevector-uncompress: internal error uncompressing
#"\0\0\0\0chez\310\224\206:\r()#\201\256R-d\205\233\24\363\5\20\201P\6A\v\300\0\16\f\6\31\2\f\6\f&H\275\0\1\0\362\bA\377e\0\1\0C\6A\21\3\v\300\0\201\265!\f\6\n\0\a\1\35\0\1+\0\360\27\201\375\300\0\0\0\17\205\210Z\0\0M\215\245\b\4\0\0M9fH\17\206fZ\0\0I\2...
context...:
body of
"/gnu/store/bbnhjamch9125c412bl7ybf28a0jxrkd-racket-8.0/share/racket/pkgs/gui-lib/mred/private/wx/gtk/utils.rkt"
```
P
P
Philip McGrath wrote on 15 Mar 2021 23:38
53940c03-4f40-3954-0cb4-3d4ae01c39a2@philipmcgrath.com
Aha! Running:

guix environment --ad-hoc --no-grafts racket -- drracket

launches DrRacket just fine.

My guess is that Racket CS is compressing string literals in compiled
code. Currently, Guix patches Racket source files to include the
absolute paths to foreign libraries in the store as string literals.
There are a bunch of grafts for GTK and such: if I'm right, Guix somehow
mangles the compiled code while attempting to apply the grafts.

I already thought this strategy was a bad idea. If it is really the
problem, I should be able to patch it fairly quickly: I've already been
experimenting along these lines.

-Philip
M
M
Malte Gerdes wrote on 16 Mar 2021 00:01
Re: bug#47064: FYI: DrRacket problem
(name . Jack Hill)(address . jackhill@jackhill.us)
CAMVpQiXKH010oe4_2_nsJDO4qjpqBBVC3fAq+p9uomP74014kg@mail.gmail.com
Hi,

I do not have problems with racket on a foreign distro. On guix system I
experience the same error.

Malte

On Mon, 15 Mar 2021, 22:06 Jack Hill, <jackhill@jackhill.us> wrote:

Toggle quote (14 lines)
> Philip,
>
> Thanks for the work you've been doing on improving our Racket package. I
> wanted to make sure you were aware of the failure of DrRacket and other
> graphical Racket components to load in the latest Racket package.
>
> https://issues.guix.gnu.org/47064
>
> Best,
> Jack
>
>
>
>
Attachment: file
J
J
jackhill wrote on 16 Mar 2021 01:28
Re: [racket-users] bytevector-uncompress: internal error uncompressing
(name . Philip McGrath)(address . philip@philipmcgrath.com)
alpine.DEB.2.21.2103152026420.8138@marsh.hcoop.net
On Mon, 15 Mar 2021, Philip McGrath wrote:

Toggle quote (16 lines)
> Aha! Running:
>
> guix environment --ad-hoc --no-grafts racket -- drracket
>
> launches DrRacket just fine.
>
> My guess is that Racket CS is compressing string literals in compiled code.
> Currently, Guix patches Racket source files to include the absolute paths to
> foreign libraries in the store as string literals. There are a bunch of
> grafts for GTK and such: if I'm right, Guix somehow mangles the compiled code
> while attempting to apply the grafts.
>
> I already thought this strategy was a bad idea. If it is really the problem,
> I should be able to patch it fairly quickly: I've already been experimenting
> along these lines.

Aha, that does sound promising. This certinially wouldn't be the only
grafts corner case:


Thanks for taking a look,
Jack
P
P
Philip McGrath wrote on 16 Mar 2021 04:01
(name . jackhill)(address . jackhill@trilug.org)
f7b8dfd6-ca6b-71b7-db6b-0cb39d2147ef@philipmcgrath.com
I've submitted a patch at https://issues.guix.gnu.org/47180that I hope
will resolve this issue.

-Philip
M
M
Mark H Weaver wrote on 14 Apr 2021 07:54
Re: bug#47064: [racket-users] bytevector-uncompress: internal error uncompressing
87o8ehquk6.fsf@netris.org
Hi Philip, [removed 'racket-users' from the recipient list]

Philip McGrath <philip@philipmcgrath.com> writes:

Toggle quote (6 lines)
> My guess is that Racket CS is compressing string literals in compiled
> code. Currently, Guix patches Racket source files to include the
> absolute paths to foreign libraries in the store as string literals.
> There are a bunch of grafts for GTK and such: if I'm right, Guix somehow
> mangles the compiled code while attempting to apply the grafts.

I think I know what happened here.

Recall that the grafting code performs a set of substitutions, replacing
store item names (i.e. file names in /gnu/store) with replacement store
items of the same length, with rules like:
"fx3979c88s9yxdbchyf36qryawgzpwb5-libx11-1.6.10" =>
"rwkqxykm91a75w9afhb41saj0dmf30hw-libx11-1.6.12".

The grafting code currently only checks the first 33 bytes, consisting
of the nix-base32 hash and the "-". It *assumes* that the remainder of
the associated store item name immediately follows, and blindly writes
the replacement string over whatever is there.

In this case, I suspect that within a *.zo file, a Guix store item name
was split into pieces, with the hash and "-" together in one piece but
split somewhere between the "-" and the last byte of the store item.
This results in corruption of the bytes following that piece.

I've recently observed the splitting of store item names in *.zo files
(see https://bugs.gnu.org/47614), but in that case the "-" was
separated from the hash, and as a result the reference was _invisible_
to the grafter.

For the record, when I originally wrote this fast(er) grafting code
(commit 5a1add373ab427a3b336981d857252e703a9f8d1), by design it only
rewrote the hashes, and so naturally it had the following desirable
property: it never overwrote any byte without first checking it against
an expected value. Later, starting in commit
57bdd79e485801ccf405ca7389bd099809fe5d67, the grafting code was modified
to allow rewriting the entire store item name (notably including the
version number). Unfortunately, although the set of overwritten bytes
was extended past the "-", the set of bytes *checked* was left
unchanged, and thus the aforementioned desirable property was lost.

I think we ought to restore that property. I'm already working on some
other changed to the grafting code (supporting UTF-16 and UTF-32 encoded
references), so I'll try to find the time to fix this problem as well.

Regards,
Mark
P
P
Philip McGrath wrote on 15 Apr 2021 00:03
b669b5ff-2435-a956-5c1d-3cc28a345857@philipmcgrath.com
Hi Mark and everyone,

On 4/14/21 1:54 AM, Mark H Weaver wrote:
Toggle quote (21 lines)
> Recall that the grafting code performs a set of substitutions, replacing
> store item names (i.e. file names in /gnu/store) with replacement store
> items of the same length, with rules like:
> "fx3979c88s9yxdbchyf36qryawgzpwb5-libx11-1.6.10" =>
> "rwkqxykm91a75w9afhb41saj0dmf30hw-libx11-1.6.12".
>
> The grafting code currently only checks the first 33 bytes, consisting
> of the nix-base32 hash and the "-". It *assumes* that the remainder of
> the associated store item name immediately follows, and blindly writes
> the replacement string over whatever is there.
>
> In this case, I suspect that within a *.zo file, a Guix store item name
> was split into pieces, with the hash and "-" together in one piece but
> split somewhere between the "-" and the last byte of the store item.
> This results in corruption of the bytes following that piece.
>
> I've recently observed the splitting of store item names in *.zo files
> (see <https://bugs.gnu.org/47614>), but in that case the "-" was
> separated from the hash, and as a result the reference was _invisible_
> to the grafter.

Yes, I agree with this diagnosis.

It seems the discussion has become a bit fragmented, since Jack first
reported one set of symptoms in https://issues.guix.gnu.org/47064 and
you then reported another in https://issues.guix.gnu.org/47614 (with
much better forensics than I'd found on my own—thanks!).

Both issues should have been fixed (at least with respect to Racket) by
my patch in https://issues.guix.gnu.org/47180, which was applied on
Monday.

-Philip
M
M
Mark H Weaver wrote on 15 Apr 2021 10:44
87y2dkhr7t.fsf@netris.org
Hi Philip,

Philip McGrath <philip@philipmcgrath.com> writes:

Toggle quote (6 lines)
> On 4/14/21 1:54 AM, Mark H Weaver wrote:
>> In this case, I suspect that within a *.zo file, a Guix store item name
>> was split into pieces, with the hash and "-" together in one piece but
>> split somewhere between the "-" and the last byte of the store item.
>> This results in corruption of the bytes following that piece.

FYI, I've now verified this suspicion. Here's precisely the corruption
that occurred:

Toggle snippet (23 lines)
mhw@jojen ~$ diff -u <(hexdump -C $(guix build racket --no-grafts)/share/racket/pkgs/gui-lib/mred/private/wx/gtk/compiled/utils_rkt.zo) \
<(hexdump -C $(guix build racket )/share/racket/pkgs/gui-lib/mred/private/wx/gtk/compiled/utils_rkt.zo)
--- /dev/fd/63 2021-04-15 04:36:01.240427788 -0400
+++ /dev/fd/62 2021-04-15 04:36:01.240427788 -0400
@@ -2047,11 +2047,11 @@
00007fe0 49 8b 6f 0b 08 00 09 d0 02 2f d7 fe d0 02 07 f2 |I.o....../......|
00007ff0 02 0b 00 62 12 04 00 12 12 05 00 3e 12 06 00 17 |...b.......>....|
00008000 12 07 3d 02 f0 28 32 02 04 75 6e 69 78 00 1e 26 |..=..(2..unix..&|
-00008010 5a 2f 67 6e 75 2f 73 74 6f 72 65 2f 69 72 6a 61 |Z/gnu/store/irja|
-00008020 6e 35 77 71 37 6a 32 35 66 61 32 6d 36 6e 32 78 |n5wq7j25fa2m6n2x|
-00008030 68 6c 38 6d 67 6c 73 61 71 78 6e 34 2d 49 02 02 |hl8mglsaqxn4-I..|
-00008040 a5 02 fd 01 2b 73 76 67 2d 32 2e 34 30 2e 30 2f |....+svg-2.40.0/|
-00008050 6c 69 62 2f c2 02 39 2e 73 6f 9b 02 1d 43 9b 02 |lib/..9.so...C..|
+00008010 5a 2f 67 6e 75 2f 73 74 6f 72 65 2f 36 66 32 30 |Z/gnu/store/6f20|
+00008020 38 64 61 6b 32 77 64 62 30 61 72 31 68 6e 38 79 |8dak2wdb0ar1hn8y|
+00008030 6b 31 39 30 79 77 67 67 69 77 33 63 2d 67 64 6b |k190ywggiw3c-gdk|
+00008040 2d 70 69 78 62 75 66 2b 73 76 67 2d 32 2e 34 30 |-pixbuf+svg-2.40|
+00008050 2e 30 62 2f c2 02 39 2e 73 6f 9b 02 1d 43 9b 02 |.0b/..9.so...C..|
00008060 57 30 07 01 02 0e 35 00 05 a2 02 c3 12 08 0c 26 |W0....5........&|
00008070 00 18 12 09 02 2c 84 00 55 02 0f 54 02 09 13 7f |.....,..U..T....|
00008080 54 02 1b 49 54 02 15 32 54 02 15 1c 54 02 1f 01 |T..IT..2T...T...|

Toggle quote (11 lines)
> Yes, I agree with this diagnosis.
>
> It seems the discussion has become a bit fragmented, since Jack first
> reported one set of symptoms in <https://issues.guix.gnu.org/47064> and
> you then reported another in <https://issues.guix.gnu.org/47614> (with
> much better forensics than I'd found on my own—thanks!).
>
> Both issues should have been fixed (at least with respect to Racket) by
> my patch in <https://issues.guix.gnu.org/47180>, which was applied on
> Monday.

Sounds good, thank you!

Mark
?