recursive import between (gnu packages chez) and (gnu packages racket)

  • Done
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Philip McGrath
  • raingloom
Owner
unassigned
Submitted by
raingloom
Severity
normal
R
R
raingloom wrote on 26 May 2021 20:54
(name . Guix Bugs)(address . bug-guix@gnu.org)
20210526205455.3431430c@riseup.net
bisected it to this commit:
ae88e30a0f8403e781f8b01262766cdc46b1018a

* bisect.sh: the script used for git bisect run
* BISECT_LOG: obvious
* test.scm: more detailed test, with backtrace. guile dies while
printing the backtrace.

Issue doesn't occur when i comment out the use-module line in
racket.scm that imports (@ (gnu packages chez) chez-scheme).

IRC discussion:
roptat suggests it might be triggered by something inheriting from
chez-scheme, cbaines says recursive modules usually don't cause issues,
only recursive inputs do. i haven't found any relevant inherits with a
quick grep, but maybe it's transitive or something.
I also haven't found any other mention of chez-scheme in racket.scm
other than the line that imports it, which is weird, given that Racket
is now built on Chez, so I'd expect it to use it as an input. I guess
it's a bundled version? In any case I don't think I should just remove
the import, because it will be needed eventually, so this issue needs
to be fixed by then.

That's as far as I've gotten today, will hopefully have time to look
into it tomorrow.

ps.: should I forward this to the guile list?
(set! %load-verbosely #t) (load "gnu/packages/chez.scm") ,bt
Attachment: bisect.sh
Attachment: BISECT_LOG
P
P
Philip McGrath wrote on 27 May 2021 06:26
7bb14837-cd53-45f6-f572-7493c2b1bb50@philipmcgrath.com
Hi, I've been working on Racket packaging. I haven't looked into this
much yet---hopefully I can tomorrow---but here are a few quick thoughts.

On 5/26/21 2:54 PM, raingloom wrote:
> I also haven't found any other mention of chez-scheme in racket.scm
> other than the line that imports it, which is weird, given that Racket

This was my doing. When I added `(gnu packages racket)` in [1], I
adapted the patch from a racket.scm I'd been using to experiment ([2],
also some branches of [3]): the changes that actually use (gnu packages
chez) aren't ready to go yet, but apparently I left the import line in.


On 5/26/21 2:54 PM, raingloom wrote:
> other than the line that imports it, which is weird, given that Racket
> is now built on Chez, so I'd expect it to use it as an input. I guess
> it's a bundled version? In any case I don't think I should just remove
> the import, because it will be needed eventually, so this issue needs
> to be fixed by then.

Racket uses a fork of Chez Scheme: I described the situation in [4].
Racket won't have upstream Chez as an input in the foreseeable, but,
from a packaging perspective, building Racket's Chez fork uses many of
the same pieces as building upstream Chez. I think it may make sense to
refactor out some of the common infrastructure, so both `(gnu packages
racket)` and `(gnu packages chez)` could depend on some low-level
module, but I'm not sure what the Guix-preferred way to organize things
would be. (I'm mostly a Racketeer using Racket packaging to learn more
about Guix.)

Here are some ways `(gnu packages chez)` and `(gnu packages racket)` are
intertwined, in no particular order:

- `racket-minimal` should use the `nanopass` and `stex` origins
as `chez-scheme` does. (But it would be easiest to do this
once I've changed `racket-minimal` to build from the Git source,
rather than the bootstrapped tarball: I hope that will be soon.)
Some more of my not-quite-current thoughts on that at [5].

- The `chez-scheme` phases `unpack-nanopass+stex`, `configure`,
`prepare-stex`, and `install-doc` should be shared with Racket.
I think it would be better to put them in a build-side module and
actually share them, rather than to do tricky things to extract
their s-expression representation from
`(package-arguments chez-scheme)`. On the other hand, I think a
build system would be overkill: it would only build vanilla Chez
and Racket-flavored Chez.

- Racket-flavored Chez has added some backends that vanilla Chez
doesn't support (and can't be readily ported, unless upstream Chez
eventually adopts Racket's changes to generating backends).
In particular, Racket-flavored Chez adds support for threading on
ARMv6 and support for aarch64 (which vanilla Chez doesn't support at
all). I haven't thought at all deeply about this, but it might make
sense for the default `chez-scheme` package on those architectures
to be Racket-flavored Chez.

- We may in fact want to use Racket to bootstrap vanilla Chez.

Chez has the usual problem with self-hosting compilers that you
need the old version to compile the new version. Specifically, you
need "bootfiles", which encapsulate the Scheme-implemented portion
of Chez, compiled to machine code. Thus, (a) they are platform-
specific and (b) they need to agree precisely with the C-implemented
part of Chez on the layout of datatypes and such.

The vanilla Chez Git repository [6] has bootfiles for i686, x86_64,
and (non-threaded) ARMv6. Once you've done a native build for one of
those platforms, you can use it to cross-compile for any platform
Chez supports (ppc32, various BSDs, etc.). But these are binary
blobs. From a "trusting trust" perspective, it's especially striking
if you consider that Chez Scheme was non-free software from 1984 to
2016, and the first libre release likewise needed the bootfiles of
its ancestors. (However, building bootfiles is reproducible:
Chez in fact builds them twice and errors if they aren't
byte-for-byte identical.)

But Racket is able to simulate enough of Chez to (slowly)
compile the Chez compiler and generate bootfiles, providing a path
to Chez from just a C compiler. Racket does its whole bootstrapping
process regularly in CI, and I'm working on getting the Guix package
to do likewise.

Unfortunately, the Racket fork has diverged enough from vanilla Chez
that the current Racket "cs-bootstrap" package [7] can't build
vanilla Chez, but I hope that the solution is just a matter of
walking through the Git history to find the right older version of
the bootstrapping package, before the Racket fork's `#!base-rtd`
gained a vector of ancestors rather than a parent and a few such
things. But I'd like to be able to build Racket packages with Guix
before I try that :)

Ok, that got a bit long.

I don't know where the cycle came from with `emacs-geiser-racket`, but I
think it would be reasonable either to do some refactoring to `(gnu
packages chez)` and `(gnu packages racket)`, or to just remove the
import for now to un-break things and figure out the rest later.

-Philip

[7]:
R
R
raingloom wrote on 28 May 2021 04:02
(name . Philip McGrath)(address . philip@philipmcgrath.com)
20210528040258.41d66653@riseup.net
Attachment: file
From cef344639ccbf45b5397368d643a36fa48dac098 Mon Sep 17 00:00:00 2001
From: raingloom <raingloom@riseup.net>
Date: Fri, 28 May 2021 03:59:16 +0200
Subject: [PATCH] gnu: Break up import loop between (gnu packages racket) and
(gnu packages chez).

* gnu/packages/racket.scm: Remove (gnu packages chez) import.
---
gnu/packages/racket.scm | 2 --
1 file changed, 2 deletions(-)

Toggle diff (15 lines)
diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
index be33270b30..2d606071fe 100644
--- a/gnu/packages/racket.scm
+++ b/gnu/packages/racket.scm
@@ -32,8 +32,6 @@
#:use-module (ice-9 match)
#:use-module (gnu packages)
#:use-module (gnu packages bash)
- #:use-module ((gnu packages chez)
- #:select (chez-scheme))
#:use-module (gnu packages compression)
#:use-module (gnu packages databases)
#:use-module (gnu packages fontutils)
--
2.31.1
P
P
Philip McGrath wrote on 28 May 2021 20:19
(name . raingloom)(address . raingloom@riseup.net)
89d20138-a0b9-f556-8a37-9c2e3684e94e@philipmcgrath.com
On 5/27/21 10:02 PM, raingloom wrote:
Toggle quote (2 lines)
> Well, here is a quick and obvious fix for now.

This looks good to me.

-Philip
L
L
Ludovic Courtès wrote on 29 May 2021 22:15
Re: bug#48682: recursive import between (gnu packages chez) and (gnu packages racket)
(name . raingloom)(address . raingloom@riseup.net)
87o8ct5mal.fsf@gnu.org
Hi,

raingloom <raingloom@riseup.net> skribis:

Toggle quote (11 lines)
>>From cef344639ccbf45b5397368d643a36fa48dac098 Mon Sep 17 00:00:00 2001
> From: raingloom <raingloom@riseup.net>
> Date: Fri, 28 May 2021 03:59:16 +0200
> Subject: [PATCH] gnu: Break up import loop between (gnu packages racket) and
> (gnu packages chez).
>
> * gnu/packages/racket.scm: Remove (gnu packages chez) import.
> ---
> gnu/packages/racket.scm | 2 --
> 1 file changed, 2 deletions(-)

Applied, thanks!

Toggle quote (12 lines)
> diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
> index be33270b30..2d606071fe 100644
> --- a/gnu/packages/racket.scm
> +++ b/gnu/packages/racket.scm
> @@ -32,8 +32,6 @@
> #:use-module (ice-9 match)
> #:use-module (gnu packages)
> #:use-module (gnu packages bash)
> - #:use-module ((gnu packages chez)
> - #:select (chez-scheme))
> #:use-module (gnu packages compression)

In general we cannot use #:select for (gnu packages …) modules because
that doesn’t play well with circular module dependencies.

That said… what problem is this fixing raingloom?

Usually, circular top-level references lead things like ‘guix show
chez-scheme’ (this amounts to loading (gnu packages chez)), but it
worked fine for me.

Thanks,
Ludo’.
Closed
P
P
Philip McGrath wrote on 29 May 2021 23:42
(address . 48682-done@debbugs.gnu.org)
d5da3932-bb10-ffe0-644a-6b1853709bb3@philipmcgrath.com
Hi, thanks!

Toggle quote (6 lines)
> That said… what problem is this fixing raingloom?
>
> Usually, circular top-level references lead things like ‘guix show
> chez-scheme’ (this amounts to loading (gnu packages chez)), but it
> worked fine for me.

It seems like some other commit may have broken the cycle in the mean
time. Here's what I had been seeing:

```
philip@avalon:~$ guix time-machine
--commit=ae88e30a0f8403e781f8b01262766cdc46b1018a -- show chez-scheme
Backtrace:
19 (primitive-load-path "gnu/packages/racket" #<procedure …>)
In gnu/packages/racket.scm:
22:0 18 (_)
In ice-9/boot-9.scm:
3409:4 17 (define-module* _ #:filename _ #:pure _ #:version _ # _ …)
3422:24 16 (_)
222:29 15 (map1 (((guix licenses) #:select (asl2.0 expat #)) (#) …))
222:29 14 (map1 (((guix packages)) ((guix download)) ((guix #)) …))
222:29 13 (map1 (((guix download)) ((guix git-download)) ((# …)) …))
222:29 12 (map1 (((guix git-download)) ((guix utils)) ((guix …)) …))
222:29 11 (map1 (((guix utils)) ((guix gexp)) ((guix # gnu)) (#) …))
222:29 10 (map1 (((guix gexp)) ((guix build-system gnu)) ((# …)) …))
222:29 9 (map1 (((guix build-system gnu)) ((srfi srfi-1)) ((…)) …))
222:29 8 (map1 (((srfi srfi-1)) ((ice-9 match)) ((gnu #)) ((…)) …))
222:29 7 (map1 (((ice-9 match)) ((gnu packages)) ((gnu # #)) # …))
222:29 6 (map1 (((gnu packages)) ((gnu packages bash)) ((…) …) …))
222:29 5 (map1 (((gnu packages bash)) ((gnu packages chez) # …) …))
222:17 4 (map1 (((gnu packages chez) #:select (chez-scheme)) # …))
3353:12 3 (resolve-interface (gnu packages chez) #:select _ #:hide …)
260:13 2 (for-each #<procedure 7fe3d526c500 at ice-9/boot-9.scm…> …)
3360:19 1 (_ _)
1685:16 0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
Throw to key `unbound-variable' with args `("resolve-interface" "no
binding `~A' in module ~A" (chez-scheme (gnu packages chez)) #f)'.
```

I don't think anyone ever figured out where the cycle was. The tl;dr of
my long comment is that there wasn't a logical cycle yet, and I was
intending to have (gnu packages racket) import (gnu packages chez), but
not vice versa, for now.

raingloom had discussed more on IRC---I only happened to see a little of
it, but I could reproduce.

-Philip
Closed
P
P
Philip McGrath wrote on 30 May 2021 00:02
(address . 48682-done@debbugs.gnu.org)
fbd68cec-5ff6-0fac-28a7-03b532147065@philipmcgrath.com
On 5/29/21 4:15 PM, Ludovic Courtès wrote:
> In general we cannot use #:select for (gnu packages …) modules because
> that doesn’t play well with circular module dependencies.

Ah, interesting, I'll keep that in mind. I'm used to Racket, where all
cyclic module dependencies cause errors at compile time.

Do you have any advice on what would be good practice?

In the near future, I'll want to get the `nanopass` and `stex` origins
for Racket, potentially via `(package-inputs chez-scheme)`---especially
because those are not exported variables. And also this part:

> - The `chez-scheme` phases `unpack-nanopass+stex`, `configure`,
> `prepare-stex`, and `install-doc` should be shared with Racket.
> I think it would be better to put them in a build-side module and
> actually share them, rather than to do tricky things to extract
> their s-expression representation from
> `(package-arguments chez-scheme)`. On the other hand, I think a
> build system would be overkill: it would only build vanilla Chez
> and Racket-flavored Chez.

I'm getting very close to being able to make `racket` accept
`racket-minimal` as an input, rather than duplicate most of it. This is
exercising some features Racket has theoretically had for a while, but
which apparently didn't quite work until ... well, this afternoon it
worked for me outside of Guix, thanks to a bunch of fixes in the last
few weeks by Matthew Flatt.

Would it make sense for me, when some useful amount of this works, to
send a patch series adding a `racket-next` package? I think the changes
are too much to backport to Racket 8.1.

-Philip
Closed
L
L
Ludovic Courtès wrote on 31 May 2021 18:23
(name . Philip McGrath)(address . philip@philipmcgrath.com)
87lf7u3m9s.fsf@gnu.org
Hi,

Philip McGrath <philip@philipmcgrath.com> skribis:

Toggle quote (7 lines)
> On 5/29/21 4:15 PM, Ludovic Courtès wrote:
>> In general we cannot use #:select for (gnu packages …) modules because
>> that doesn’t play well with circular module dependencies.
>
> Ah, interesting, I'll keep that in mind. I'm used to Racket, where all
> cyclic module dependencies cause errors at compile time.

Yeah, in hindsight, that’s probably safer…

Toggle quote (2 lines)
> Do you have any advice on what would be good practice?

For package modules, the main things are:

1. Don’t use #:select or #:autoload for (gnu packages …) modules in a
(gnu packages …) module.

2. At the top level of a module, only refer to variables within that
module. For instance, the following would be wrong:

(define-module (gnu packages racket)
#:use-module (gnu packages chez)
…)

(define whatever
;; Wrong because ‘chez-scheme’ is defined in another module,
;; which might be part of a cycle with this one.
(package (inherit chez-scheme) …))

(define something
(package
;; …
(license (package-license chez-scheme)))) ;likewise

Note that references from ‘inputs’ and ‘arguments’ fields are perfectly
fine (fortunately!) because those fields are “thunked” (their value is
wrapped in a thunk).

Toggle quote (21 lines)
> In the near future, I'll want to get the `nanopass` and `stex` origins
> for Racket, potentially via `(package-inputs
> chez-scheme)`---especially because those are not exported
> variables. And also this part:
>
>> - The `chez-scheme` phases `unpack-nanopass+stex`, `configure`,
>> `prepare-stex`, and `install-doc` should be shared with Racket.
>> I think it would be better to put them in a build-side module and
>> actually share them, rather than to do tricky things to extract
>> their s-expression representation from
>> `(package-arguments chez-scheme)`. On the other hand, I think a
>> build system would be overkill: it would only build vanilla Chez
>> and Racket-flavored Chez.
>
> I'm getting very close to being able to make `racket` accept
> `racket-minimal` as an input, rather than duplicate most of it. This
> is exercising some features Racket has theoretically had for a while,
> but which apparently didn't quite work until ... well, this afternoon
> it worked for me outside of Guix, thanks to a bunch of fixes in the
> last few weeks by Matthew Flatt.

Neat.

Toggle quote (4 lines)
> Would it make sense for me, when some useful amount of this works, to
> send a patch series adding a `racket-next` package? I think the
> changes are too much to backport to Racket 8.1.

Possibly, your call! :-)

Thanks,
Ludo’.
Closed
?