recursive import between (gnu packages chez) and (gnu packages racket)

DoneSubmitted by raingloom.
Details
3 participants
  • Ludovic Courtès
  • Philip McGrath
  • raingloom
Owner
unassigned
Severity
normal
R
R
raingloom wrote on 26 May 20:54 +0200
(name . Guix Bugs)(address . bug-guix@gnu.org)
20210526205455.3431430c@riseup.net
bisected it to this commit:ae88e30a0f8403e781f8b01262766cdc46b1018a
* bisect.sh: the script used for git bisect run* BISECT_LOG: obvious* test.scm: more detailed test, with backtrace. guile dies while printing the backtrace.
Issue doesn't occur when i comment out the use-module line inracket.scm that imports (@ (gnu packages chez) chez-scheme).
IRC discussion:roptat suggests it might be triggered by something inheriting fromchez-scheme, cbaines says recursive modules usually don't cause issues,only recursive inputs do. i haven't found any relevant inherits with aquick grep, but maybe it's transitive or something.I also haven't found any other mention of chez-scheme in racket.scmother than the line that imports it, which is weird, given that Racketis now built on Chez, so I'd expect it to use it as an input. I guessit's a bundled version? In any case I don't think I should just removethe import, because it will be needed eventually, so this issue needsto be fixed by then.
That's as far as I've gotten today, will hopefully have time to lookinto it tomorrow.
ps.: should I forward this to the guile list?
(set! %load-verbosely #t) (load "gnu/packages/chez.scm") ,bt
Attachment: bisect.sh
Attachment: BISECT_LOG
P
P
Philip McGrath wrote on 27 May 06:26 +0200
7bb14837-cd53-45f6-f572-7493c2b1bb50@philipmcgrath.com
Hi, I've been working on Racket packaging. I haven't looked into this much yet---hopefully I can tomorrow---but here are a few quick thoughts.
On 5/26/21 2:54 PM, raingloom wrote: > I also haven't found any other mention of chez-scheme in racket.scm > other than the line that imports it, which is weird, given that Racket
This was my doing. When I added `(gnu packages racket)` in [1], I adapted the patch from a racket.scm I'd been using to experiment ([2], also some branches of [3]): the changes that actually use (gnu packages chez) aren't ready to go yet, but apparently I left the import line in.

On 5/26/21 2:54 PM, raingloom wrote: > other than the line that imports it, which is weird, given that Racket > is now built on Chez, so I'd expect it to use it as an input. I guess > it's a bundled version? In any case I don't think I should just remove > the import, because it will be needed eventually, so this issue needs > to be fixed by then.
Racket uses a fork of Chez Scheme: I described the situation in [4]. Racket won't have upstream Chez as an input in the foreseeable, but, from a packaging perspective, building Racket's Chez fork uses many of the same pieces as building upstream Chez. I think it may make sense to refactor out some of the common infrastructure, so both `(gnu packages racket)` and `(gnu packages chez)` could depend on some low-level module, but I'm not sure what the Guix-preferred way to organize things would be. (I'm mostly a Racketeer using Racket packaging to learn more about Guix.)
Here are some ways `(gnu packages chez)` and `(gnu packages racket)` are intertwined, in no particular order:
- `racket-minimal` should use the `nanopass` and `stex` origins as `chez-scheme` does. (But it would be easiest to do this once I've changed `racket-minimal` to build from the Git source, rather than the bootstrapped tarball: I hope that will be soon.) Some more of my not-quite-current thoughts on that at [5].
- The `chez-scheme` phases `unpack-nanopass+stex`, `configure`, `prepare-stex`, and `install-doc` should be shared with Racket. I think it would be better to put them in a build-side module and actually share them, rather than to do tricky things to extract their s-expression representation from `(package-arguments chez-scheme)`. On the other hand, I think a build system would be overkill: it would only build vanilla Chez and Racket-flavored Chez.
- Racket-flavored Chez has added some backends that vanilla Chez doesn't support (and can't be readily ported, unless upstream Chez eventually adopts Racket's changes to generating backends). In particular, Racket-flavored Chez adds support for threading on ARMv6 and support for aarch64 (which vanilla Chez doesn't support at all). I haven't thought at all deeply about this, but it might make sense for the default `chez-scheme` package on those architectures to be Racket-flavored Chez.
- We may in fact want to use Racket to bootstrap vanilla Chez.
Chez has the usual problem with self-hosting compilers that you need the old version to compile the new version. Specifically, you need "bootfiles", which encapsulate the Scheme-implemented portion of Chez, compiled to machine code. Thus, (a) they are platform- specific and (b) they need to agree precisely with the C-implemented part of Chez on the layout of datatypes and such.
The vanilla Chez Git repository [6] has bootfiles for i686, x86_64, and (non-threaded) ARMv6. Once you've done a native build for one of those platforms, you can use it to cross-compile for any platform Chez supports (ppc32, various BSDs, etc.). But these are binary blobs. From a "trusting trust" perspective, it's especially striking if you consider that Chez Scheme was non-free software from 1984 to 2016, and the first libre release likewise needed the bootfiles of its ancestors. (However, building bootfiles is reproducible: Chez in fact builds them twice and errors if they aren't byte-for-byte identical.)
But Racket is able to simulate enough of Chez to (slowly) compile the Chez compiler and generate bootfiles, providing a path to Chez from just a C compiler. Racket does its whole bootstrapping process regularly in CI, and I'm working on getting the Guix package to do likewise.
Unfortunately, the Racket fork has diverged enough from vanilla Chez that the current Racket "cs-bootstrap" package [7] can't build vanilla Chez, but I hope that the solution is just a matter of walking through the Git history to find the right older version of the bootstrapping package, before the Racket fork's `#!base-rtd` gained a vector of ancestors rather than a parent and a few such things. But I'd like to be able to build Racket packages with Guix before I try that :)
Ok, that got a bit long.
I don't know where the cycle came from with `emacs-geiser-racket`, but I think it would be reasonable either to do some refactoring to `(gnu packages chez)` and `(gnu packages racket)`, or to just remove the import for now to un-break things and figure out the rest later.
-Philip
[1]: https://issues.guix.gnu.org/47829[2]: https://gitlab.com/philip1/guix-racket-experiment[3]: https://gitlab.com/philip1/guix-patches/[4]: https://issues.guix.gnu.org/46865#3[5]: https://issues.guix.gnu.org/47153#2[6]: https://github.com/cisco/ChezScheme[7]: https://github.com/racket/racket/tree/master/racket/src/ChezScheme/rktboot
R
R
raingloom wrote on 28 May 04:02 +0200
(name . Philip McGrath)(address . philip@philipmcgrath.com)
20210528040258.41d66653@riseup.net
On Thu, 27 May 2021 00:26:56 -0400Philip McGrath <philip@philipmcgrath.com> wrote:
Toggle quote (113 lines)> Hi, I've been working on Racket packaging. I haven't looked into this > much yet---hopefully I can tomorrow---but here are a few quick> thoughts.> > On 5/26/21 2:54 PM, raingloom wrote:> > I also haven't found any other mention of chez-scheme in racket.scm> > other than the line that imports it, which is weird, given that> > Racket > > This was my doing. When I added `(gnu packages racket)` in [1], I > adapted the patch from a racket.scm I'd been using to experiment> ([2], also some branches of [3]): the changes that actually use (gnu> packages chez) aren't ready to go yet, but apparently I left the> import line in.> > > On 5/26/21 2:54 PM, raingloom wrote:> > other than the line that imports it, which is weird, given that> > Racket is now built on Chez, so I'd expect it to use it as an> > input. I guess it's a bundled version? In any case I don't think I> > should just remove the import, because it will be needed> > eventually, so this issue needs to be fixed by then. > > Racket uses a fork of Chez Scheme: I described the situation in [4]. > Racket won't have upstream Chez as an input in the foreseeable, but, > from a packaging perspective, building Racket's Chez fork uses many> of the same pieces as building upstream Chez. I think it may make> sense to refactor out some of the common infrastructure, so both> `(gnu packages racket)` and `(gnu packages chez)` could depend on> some low-level module, but I'm not sure what the Guix-preferred way> to organize things would be. (I'm mostly a Racketeer using Racket> packaging to learn more about Guix.)> > Here are some ways `(gnu packages chez)` and `(gnu packages racket)`> are intertwined, in no particular order:> > - `racket-minimal` should use the `nanopass` and `stex` origins> as `chez-scheme` does. (But it would be easiest to do this> once I've changed `racket-minimal` to build from the Git source,> rather than the bootstrapped tarball: I hope that will be soon.)> Some more of my not-quite-current thoughts on that at [5].> > - The `chez-scheme` phases `unpack-nanopass+stex`, `configure`,> `prepare-stex`, and `install-doc` should be shared with Racket.> I think it would be better to put them in a build-side module and> actually share them, rather than to do tricky things to extract> their s-expression representation from> `(package-arguments chez-scheme)`. On the other hand, I think a> build system would be overkill: it would only build vanilla Chez> and Racket-flavored Chez.> > - Racket-flavored Chez has added some backends that vanilla Chez> doesn't support (and can't be readily ported, unless upstream> Chez eventually adopts Racket's changes to generating backends).> In particular, Racket-flavored Chez adds support for threading on> ARMv6 and support for aarch64 (which vanilla Chez doesn't> support at all). I haven't thought at all deeply about this, but it> might make sense for the default `chez-scheme` package on those> architectures to be Racket-flavored Chez.> > - We may in fact want to use Racket to bootstrap vanilla Chez.> > Chez has the usual problem with self-hosting compilers that you> need the old version to compile the new version. Specifically,> you need "bootfiles", which encapsulate the Scheme-implemented portion> of Chez, compiled to machine code. Thus, (a) they are platform-> specific and (b) they need to agree precisely with the> C-implemented part of Chez on the layout of datatypes and such.> > The vanilla Chez Git repository [6] has bootfiles for i686,> x86_64, and (non-threaded) ARMv6. Once you've done a native build for> one of those platforms, you can use it to cross-compile for any> platform Chez supports (ppc32, various BSDs, etc.). But these are> binary blobs. From a "trusting trust" perspective, it's especially> striking if you consider that Chez Scheme was non-free software from> 1984 to 2016, and the first libre release likewise needed the> bootfiles of its ancestors. (However, building bootfiles is> reproducible: Chez in fact builds them twice and errors if they aren't> byte-for-byte identical.)> > But Racket is able to simulate enough of Chez to (slowly)> compile the Chez compiler and generate bootfiles, providing a> path to Chez from just a C compiler. Racket does its whole> bootstrapping process regularly in CI, and I'm working on getting the> Guix package to do likewise.> > Unfortunately, the Racket fork has diverged enough from vanilla> Chez that the current Racket "cs-bootstrap" package [7] can't build> vanilla Chez, but I hope that the solution is just a matter of> walking through the Git history to find the right older version> of the bootstrapping package, before the Racket fork's `#!base-rtd`> gained a vector of ancestors rather than a parent and a few such> things. But I'd like to be able to build Racket packages with> Guix before I try that :)> > Ok, that got a bit long.> > I don't know where the cycle came from with `emacs-geiser-racket`,> but I think it would be reasonable either to do some refactoring to> `(gnu packages chez)` and `(gnu packages racket)`, or to just remove> the import for now to un-break things and figure out the rest later.> > -Philip> > [1]: https://issues.guix.gnu.org/47829> [2]: https://gitlab.com/philip1/guix-racket-experiment> [3]: https://gitlab.com/philip1/guix-patches/> [4]: https://issues.guix.gnu.org/46865#3> [5]: https://issues.guix.gnu.org/47153#2> [6]: https://github.com/cisco/ChezScheme> [7]: > https://github.com/racket/racket/tree/master/racket/src/ChezScheme/rktboot
Well, here is a quick and obvious fix for now.
From cef344639ccbf45b5397368d643a36fa48dac098 Mon Sep 17 00:00:00 2001From: raingloom <raingloom@riseup.net>Date: Fri, 28 May 2021 03:59:16 +0200Subject: [PATCH] gnu: Break up import loop between (gnu packages racket) and (gnu packages chez).
* gnu/packages/racket.scm: Remove (gnu packages chez) import.--- gnu/packages/racket.scm | 2 -- 1 file changed, 2 deletions(-)
Toggle diff (15 lines)diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scmindex be33270b30..2d606071fe 100644--- a/gnu/packages/racket.scm+++ b/gnu/packages/racket.scm@@ -32,8 +32,6 @@ #:use-module (ice-9 match) #:use-module (gnu packages) #:use-module (gnu packages bash)- #:use-module ((gnu packages chez)- #:select (chez-scheme)) #:use-module (gnu packages compression) #:use-module (gnu packages databases) #:use-module (gnu packages fontutils)-- 2.31.1
P
P
Philip McGrath wrote on 28 May 20:19 +0200
(name . raingloom)(address . raingloom@riseup.net)
89d20138-a0b9-f556-8a37-9c2e3684e94e@philipmcgrath.com
On 5/27/21 10:02 PM, raingloom wrote:
Toggle quote (2 lines)> Well, here is a quick and obvious fix for now.
This looks good to me.
-Philip
L
L
Ludovic Courtès wrote on 29 May 22:15 +0200
Re: bug#48682: recursive import between (gnu packages chez) and (gnu packages racket)
(name . raingloom)(address . raingloom@riseup.net)
87o8ct5mal.fsf@gnu.org
Hi,
raingloom <raingloom@riseup.net> skribis:
Toggle quote (11 lines)>>From cef344639ccbf45b5397368d643a36fa48dac098 Mon Sep 17 00:00:00 2001> From: raingloom <raingloom@riseup.net>> Date: Fri, 28 May 2021 03:59:16 +0200> Subject: [PATCH] gnu: Break up import loop between (gnu packages racket) and> (gnu packages chez).>> * gnu/packages/racket.scm: Remove (gnu packages chez) import.> ---> gnu/packages/racket.scm | 2 --> 1 file changed, 2 deletions(-)
Applied, thanks!
Toggle quote (12 lines)> diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm> index be33270b30..2d606071fe 100644> --- a/gnu/packages/racket.scm> +++ b/gnu/packages/racket.scm> @@ -32,8 +32,6 @@> #:use-module (ice-9 match)> #:use-module (gnu packages)> #:use-module (gnu packages bash)> - #:use-module ((gnu packages chez)> - #:select (chez-scheme))> #:use-module (gnu packages compression)
In general we cannot use #:select for (gnu packages …) modules becausethat doesn’t play well with circular module dependencies.
That said… what problem is this fixing raingloom?
Usually, circular top-level references lead things like ‘guix showchez-scheme’ (this amounts to loading (gnu packages chez)), but itworked fine for me.
Thanks,Ludo’.
Closed
P
P
Philip McGrath wrote on 29 May 23:42 +0200
(address . 48682-done@debbugs.gnu.org)
d5da3932-bb10-ffe0-644a-6b1853709bb3@philipmcgrath.com
Hi, thanks!
Toggle quote (6 lines)> That said… what problem is this fixing raingloom?> > Usually, circular top-level references lead things like ‘guix show> chez-scheme’ (this amounts to loading (gnu packages chez)), but it> worked fine for me.
It seems like some other commit may have broken the cycle in the mean time. Here's what I had been seeing:
```philip@avalon:~$ guix time-machine --commit=ae88e30a0f8403e781f8b01262766cdc46b1018a -- show chez-schemeBacktrace: 19 (primitive-load-path "gnu/packages/racket" #<procedure …>)In gnu/packages/racket.scm: 22:0 18 (_)In ice-9/boot-9.scm: 3409:4 17 (define-module* _ #:filename _ #:pure _ #:version _ # _ …) 3422:24 16 (_) 222:29 15 (map1 (((guix licenses) #:select (asl2.0 expat #)) (#) …)) 222:29 14 (map1 (((guix packages)) ((guix download)) ((guix #)) …)) 222:29 13 (map1 (((guix download)) ((guix git-download)) ((# …)) …)) 222:29 12 (map1 (((guix git-download)) ((guix utils)) ((guix …)) …)) 222:29 11 (map1 (((guix utils)) ((guix gexp)) ((guix # gnu)) (#) …)) 222:29 10 (map1 (((guix gexp)) ((guix build-system gnu)) ((# …)) …)) 222:29 9 (map1 (((guix build-system gnu)) ((srfi srfi-1)) ((…)) …)) 222:29 8 (map1 (((srfi srfi-1)) ((ice-9 match)) ((gnu #)) ((…)) …)) 222:29 7 (map1 (((ice-9 match)) ((gnu packages)) ((gnu # #)) # …)) 222:29 6 (map1 (((gnu packages)) ((gnu packages bash)) ((…) …) …)) 222:29 5 (map1 (((gnu packages bash)) ((gnu packages chez) # …) …)) 222:17 4 (map1 (((gnu packages chez) #:select (chez-scheme)) # …)) 3353:12 3 (resolve-interface (gnu packages chez) #:select _ #:hide …) 260:13 2 (for-each #<procedure 7fe3d526c500 at ice-9/boot-9.scm…> …) 3360:19 1 (_ _) 1685:16 0 (raise-exception _ #:continuable? _)
ice-9/boot-9.scm:1685:16: In procedure raise-exception:Throw to key `unbound-variable' with args `("resolve-interface" "no binding `~A' in module ~A" (chez-scheme (gnu packages chez)) #f)'.```
I don't think anyone ever figured out where the cycle was. The tl;dr of my long comment is that there wasn't a logical cycle yet, and I was intending to have (gnu packages racket) import (gnu packages chez), but not vice versa, for now.
raingloom had discussed more on IRC---I only happened to see a little of it, but I could reproduce.
-Philip
Closed
P
P
Philip McGrath wrote on 30 May 00:02 +0200
(address . 48682-done@debbugs.gnu.org)
fbd68cec-5ff6-0fac-28a7-03b532147065@philipmcgrath.com
On 5/29/21 4:15 PM, Ludovic Courtès wrote: > In general we cannot use #:select for (gnu packages …) modules because > that doesn’t play well with circular module dependencies.
Ah, interesting, I'll keep that in mind. I'm used to Racket, where all cyclic module dependencies cause errors at compile time.
Do you have any advice on what would be good practice?
In the near future, I'll want to get the `nanopass` and `stex` origins for Racket, potentially via `(package-inputs chez-scheme)`---especially because those are not exported variables. And also this part:
> - The `chez-scheme` phases `unpack-nanopass+stex`, `configure`, > `prepare-stex`, and `install-doc` should be shared with Racket. > I think it would be better to put them in a build-side module and > actually share them, rather than to do tricky things to extract > their s-expression representation from > `(package-arguments chez-scheme)`. On the other hand, I think a > build system would be overkill: it would only build vanilla Chez > and Racket-flavored Chez.
I'm getting very close to being able to make `racket` accept `racket-minimal` as an input, rather than duplicate most of it. This is exercising some features Racket has theoretically had for a while, but which apparently didn't quite work until ... well, this afternoon it worked for me outside of Guix, thanks to a bunch of fixes in the last few weeks by Matthew Flatt.
Would it make sense for me, when some useful amount of this works, to send a patch series adding a `racket-next` package? I think the changes are too much to backport to Racket 8.1.
-Philip
Closed
L
L
Ludovic Courtès wrote on 31 May 18:23 +0200
(name . Philip McGrath)(address . philip@philipmcgrath.com)
87lf7u3m9s.fsf@gnu.org
Hi,
Philip McGrath <philip@philipmcgrath.com> skribis:
Toggle quote (7 lines)> On 5/29/21 4:15 PM, Ludovic Courtès wrote:>> In general we cannot use #:select for (gnu packages …) modules because>> that doesn’t play well with circular module dependencies.>> Ah, interesting, I'll keep that in mind. I'm used to Racket, where all> cyclic module dependencies cause errors at compile time.
Yeah, in hindsight, that’s probably safer…
Toggle quote (2 lines)> Do you have any advice on what would be good practice?
For package modules, the main things are:
1. Don’t use #:select or #:autoload for (gnu packages …) modules in a (gnu packages …) module.
2. At the top level of a module, only refer to variables within that module. For instance, the following would be wrong:
(define-module (gnu packages racket) #:use-module (gnu packages chez) …)
(define whatever ;; Wrong because ‘chez-scheme’ is defined in another module, ;; which might be part of a cycle with this one. (package (inherit chez-scheme) …))
(define something (package ;; … (license (package-license chez-scheme)))) ;likewise
Note that references from ‘inputs’ and ‘arguments’ fields are perfectlyfine (fortunately!) because those fields are “thunked” (their value iswrapped in a thunk).
Toggle quote (21 lines)> In the near future, I'll want to get the `nanopass` and `stex` origins> for Racket, potentially via `(package-inputs> chez-scheme)`---especially because those are not exported> variables. And also this part:>>> - The `chez-scheme` phases `unpack-nanopass+stex`, `configure`,>> `prepare-stex`, and `install-doc` should be shared with Racket.>> I think it would be better to put them in a build-side module and>> actually share them, rather than to do tricky things to extract>> their s-expression representation from>> `(package-arguments chez-scheme)`. On the other hand, I think a>> build system would be overkill: it would only build vanilla Chez>> and Racket-flavored Chez.>> I'm getting very close to being able to make `racket` accept> `racket-minimal` as an input, rather than duplicate most of it. This> is exercising some features Racket has theoretically had for a while,> but which apparently didn't quite work until ... well, this afternoon> it worked for me outside of Guix, thanks to a bunch of fixes in the> last few weeks by Matthew Flatt.
Neat.
Toggle quote (4 lines)> Would it make sense for me, when some useful amount of this works, to> send a patch series adding a `racket-next` package? I think the> changes are too much to backport to Racket 8.1.
Possibly, your call! :-)
Thanks,Ludo’.
Closed
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 48682@debbugs.gnu.org