G-expressions don't consistently preserve #nil

  • Open
  • quality assurance status badge
Details
2 participants
  • Maxime Devos
  • Philip McGrath
Owner
unassigned
Submitted by
Philip McGrath
Severity
normal
P
P
Philip McGrath wrote on 23 Dec 2021 05:25
(address . bug-guix@gnu.org)
23d2ac1d-737d-787c-5535-c816566461dd@philipmcgrath.com
G-expressions currently do not consistently preserve the distinction
between #nil and '(), which causes trouble for programs that rely on
that distinction. In particular, the issue affects programs that use
(guix build json), because that library uses #nil to represent the JSON
value `null', whereas it uses '() to represent an empty JSON array.

The following program exposes the error:

Toggle snippet (67 lines)
(use-modules (guix build json)
(guix gexp)
(guix monads)
(guix store)
(guix derivations)
(rnrs conditions)
(ice-9 textual-ports))

(define (check-equal? actual expected message)
(define who 'check-equal?)
(unless (equal? actual expected)
(raise-exception
(condition
(make-assertion-violation)
(make-who-condition 'check-equal?)
(make-irritants-condition (list actual))
(make-message-condition
(format #f "~a: ~a\n ~a: ~s\n ~a: ~s\n ~a: ~s"
who "test failed"
"message" message
"expected" expected
"actual" actual))))))

(define (sexp->json-string sx)
(call-with-output-string
(lambda (out)
(write-json sx out))))

(define (gexp->json-string gx)
(run-with-store (open-connection)
(mlet* %store-monad ((drv (gexp->derivation "example.json"
(with-imported-modules `((guix build json))
#~(begin
(use-modules (guix build json))
(call-with-output-file #$output
(lambda (out)
(write-json #$gx out)))))))
(_built (built-derivations (list drv))))
(return (call-with-input-file (derivation->output-path drv)
get-string-all)))))

(check-equal? (sexp->json-string '())
"[]"
"sexp: empty array")

(check-equal? (gexp->json-string #~'())
"[]"
"gexp: empty array")

(check-equal? (sexp->json-string #nil)
"null"
"sexp: null")

(check-equal? (gexp->json-string #~#nil)
"null"
"gexp: null")

(check-equal? (sexp->json-string '(@ ("k" . #nil)))
"{\"k\":null}"
"sexp: null in object")

;; This one fails!
(check-equal? (gexp->json-string #~'(@ ("k" . #nil)))
"{\"k\":null}"
"gexp: null in object")

-Philip
M
M
Maxime Devos wrote on 23 Dec 2021 07:59
6211bc6e48fa8f5dcf8711bba186812f3a5e52c4.camel@telenet.be
Hi,

Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
Toggle quote (6 lines)
> G-expressions currently do not consistently preserve the distinction
> between #nil and '(), which causes trouble for programs that rely on
> that distinction. In particular, the issue affects programs that use
> (guix build json), because that library uses #nil to represent the JSON
> value `null', whereas it uses '() to represent an empty JSON array.

The constant #nil is only for elisp compatibility and not something
supposed to be used in Scheme code that isn't for Scheme/elisp
compatibility, so this seems more a bug in (guix build json) to me.

Greetings,
Maxime.
M
M
Maxime Devos wrote on 23 Dec 2021 18:58
5624d97b0044f4f7e0dc761a590c857e40fc4ed9.camel@telenet.be
Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
Toggle quote (15 lines)
> G-expressions currently do not consistently preserve the distinction
> between #nil and '(), which causes trouble for programs that rely on
> that distinction. In particular, the issue affects programs that use
> (guix build json), because that library uses #nil to represent the JSON
> value `null', whereas it uses '() to represent an empty JSON array.
>
> The following program exposes the error:
> [
> ;...]
>
> ; This one fails!
> (check-equal? (gexp->json-string #~'(@ ("k" . #nil)))
>                "{\"k\":null}"
>                "gexp: null in object")

A simpler test:

Compare this:
(cdr (gexp->approximate-sexp #~("stuff" . #nil)))
; output: #nil --- seems like everything is ok?

with:
(gexp->approximate-sexp #~("stuff" . #nil))
; output: ("stuff") --- where did the #nil go?

I think the idea is that, if you construct a list (a b c . #nil)
in elisp, and pass it to Scheme, then Scheme should treat it as a
Scheme list, so it should be printed as (a b c) when using Scheme's
'write' or 'display'.

Greetings,
Maxime.
M
M
Maxime Devos wrote on 25 Dec 2021 12:13
637bb8909bd524ce239d66cc73d1e5ad43ce2ea9.camel@telenet.be
Maxime Devos schreef op do 23-12-2021 om 06:59 [+0000]:
Toggle quote (11 lines)
> Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
> > G-expressions currently do not consistently preserve the distinction
> > between #nil and '(), which causes trouble for programs that rely on
> > that distinction. In particular, the issue affects programs that use
> > (guix build json), because that library uses #nil to represent the JSON
> > value `null', whereas it uses '() to represent an empty JSON array.
>
> The constant #nil is only for elisp compatibility and not something
> supposed to be used in Scheme code that isn't for Scheme/elisp
> compatibility, so this seems more a bug in (guix build json) to me.

That said, it would be less surprising if the #nil/() distinction is
preserved by gexp->derivation and friends. This can be done by writing
our own 'write' procedure. Downside: it might be less efficient than
Guile's write which is implemented in C. Can be resolved by writing our
own 'write' procedure in C.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYcb8wxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7n4eAQDc7rGRQG2dwYgOgzrwkVv0iUnv
Wo1bt2wxpXEhr0FxrAD/V0TKSYgYTyCcFFMFNn7Mhan/QCa19bozyWYaunweOwo=
=/Wmv
-----END PGP SIGNATURE-----


P
P
Philip McGrath wrote on 27 Dec 2021 19:38
d7d030d9-c7aa-0297-d343-daafc6ec0691@philipmcgrath.com
Hi!

Just as a general disclaimer, I'm a Racketeer and only incidentally a
Schemer, so I'm not very familiar with the universe of Guile libraries.

On 12/23/21 01:59, Maxime Devos wrote:
> Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
>> G-expressions currently do not consistently preserve the distinction
>> between #nil and '(), which causes trouble for programs that rely on
>> that distinction. In particular, the issue affects programs that use
>> (guix build json), because that library uses #nil to represent the JSON
>> value `null', whereas it uses '() to represent an empty JSON array.
>
> The constant #nil is only for elisp compatibility and not something
> supposed to be used in Scheme code that isn't for Scheme/elisp
> compatibility, so this seems more a bug in (guix build json) to me.

That was not the impression I had gotten from `info "(guile)Nil"`. For
example, I think someone who wanted to finish the implementation
described in `info "(guile)ECMAScript"` might want to use #nil for one
of the false-y ECMAScript values to take advantages of the documented
efficiencies in its bit-level representation. More concretely,
guile-json@1 and guile-json@3 use #nil in the same way as (guix build json).

On 12/23/21 12:58, Maxime Devos wrote:
> Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
>> G-expressions currently do not consistently preserve the distinction
>> between #nil and '(), which causes trouble for programs that rely on
>> that distinction. In particular, the issue affects programs that use
>> (guix build json), because that library uses #nil to represent the JSON
>> value `null', whereas it uses '() to represent an empty JSON array.
>>
>> The following program exposes the error:
>> [
>> ;...]
>>
>> ; This one fails!
>> (check-equal? (gexp->json-string #~'(@ ("k" . #nil)))
>> "{\"k\":null}"
>> "gexp: null in object")
>
> A simpler test:
>
> Compare this:
> (cdr (gexp->approximate-sexp #~("stuff" . #nil)))
> ; output: #nil --- seems like everything is ok?
>
> with:
> (gexp->approximate-sexp #~("stuff" . #nil))
> ; output: ("stuff") --- where did the #nil go?
>
> I think the idea is that, if you construct a list (a b c . #nil)
> in elisp, and pass it to Scheme, then Scheme should treat it as a
> Scheme list, so it should be printed as (a b c) when using Scheme's
> 'write' or 'display'.

Since `write` and `list?` are specified by various Scheme standards, I
think it is the correct choice for `write` to use a Scheme-compatible
external representation for values recognized by `list?`, at least by
default. (Perhaps a parameter could control this behavior?)

I think the behavior of `gexp->approximate-sexp` is at least defensible,
since its documentation (`info guix "gexp->approximate-sexp"`) warns
that "some information can be lost".

But I think the implementation of G-expressions faces more stringent
obligations. I see it as analogous to the implementation of syntax
objects, a macro expander, or a compiler, in that it should have a
semantics-preserving representation of arbitrary Guile code, including
Guile's extensions to Scheme.

(I haven't yet understood at a theoretical level how "strata" and
"staging" relate to the more familiar concept of "phases", but my
intuition is that, while the R6RS model of phases wouldn't be enough, it
seems like would probably to express staging/strata in terms of phases
with Racket enhancements like the label phase level and arbitrary
submodule-implemented phases.)

So, I agree that:

On 12/25/21 06:13, Maxime Devos wrote:
Toggle quote (6 lines)
> That said, it would be less surprising if the #nil/() distinction is
> preserved by gexp->derivation and friends. This can be done by writing
> our own 'write' procedure. Downside: it might be less efficient than
> Guile's write which is implemented in C. Can be resolved by writing our
> own 'write' procedure in C.

I haven't looked at the implementation at all, but extending `write`
certainly would be a reasonable option, and, longer-term, it might be
possible to upstream a patch adding the needed behavior.

A more radical option could be to use a format other than plain-text
s-expressions for compiled G-expressions. For example, Racket has a
forward-compatible "fast-load serialization" binary format for the kinds
of values that can be embedded in compiled code.[0] There are obvious
disadvantages to a binary format, but advantages include the ability to
preserve source-location information and to avoid some the quirks that
come with functions like `write` and `read`, for historical reasons or
for the convenience of humans writing code directly. The implementation
is in Racket, so it should be fairly easy to port to Guile, if that were
wanted.[1] Or maybe there's something related to Guile bytecode that
would work, or maybe just making a `#nil`-preserving version of `write`
would be easier.

-Philip

[1]:
M
M
Maxime Devos wrote on 27 Dec 2021 21:24
20d06a3f6857a8d30039bc720a9b8c4ecf09702d.camel@telenet.be
Philip McGrath schreef op ma 27-12-2021 om 13:38 [-0500]:
Toggle quote (24 lines)
> Hi!
>
> Just as a general disclaimer, I'm a Racketeer and only incidentally a
> Schemer, so I'm not very familiar with the universe of Guile libraries.
>
> On 12/23/21 01:59, Maxime Devos wrote:
>  > Philip McGrath schreef op wo 22-12-2021 om 23:25 [-0500]:
>  >> G-expressions currently do not consistently preserve the distinction
>  >> between #nil and '(), which causes trouble for programs that rely on
>  >> that distinction. In particular, the issue affects programs that use
>  >> (guix build json), because that library uses #nil to represent the JSON
>  >> value `null', whereas it uses '() to represent an empty JSON array.
>  >
>  > The constant #nil is only for elisp compatibility and not something
>  > supposed to be used in Scheme code that isn't for Scheme/elisp
>  > compatibility, so this seems more a bug in (guix build json) to me.
>
> That was not the impression I had gotten from `info "(guile)Nil"`. For
> example, I think someone who wanted to finish the implementation
> described in `info "(guile)ECMAScript"` might want to use #nil for one
> of the false-y ECMAScript values to take advantages of the documented
> efficiencies in its bit-level representation. More concretely,
> guile-json@1 and guile-json@3 use #nil in the same way as (guix build json).

There is

‘Guile has chosen to support ‘nil’ as a separate value, distinct from
‘#f’ and ‘'()’. This allows existing Scheme and Elisp code to maintain
their current semantics. ‘nil’, which in Elisp would just be written
and read as ‘nil’, in Scheme has the external representation ‘#nil’.’

and

‘This decision to have ‘nil’ as a low-level distinct value facilitates
interoperability between the two languages. Guile has chosen to have
Scheme deal with ‘nil’ as follows: [...]’

and this is only documented under ‘Emacs Lisp’, though this doesn't
explicitely say it's not supposed to be used elsewhere.

Also, see e.g.

Anyway, this doesn't really matter here, because:

Toggle quote (10 lines)
> So, I agree that:
>
> On 12/25/21 06:13, Maxime Devos wrote:
> > That said, it would be less surprising if the #nil/() distinction is
> > preserved by gexp->derivation and friends. This can be done by writing
> > our own 'write' procedure. Downside: it might be less efficient than
> > Guile's write which is implemented in C. Can be resolved by writing our
> > own 'write' procedure in C.
> [...]

I'll try to look into other parts of your response later.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYcog8hccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7gn5AP9t2YqS7I+2HUBkqSQN1jTC42RV
8wQP0aZeD3BHgHIN2wD/VzrsOL+oHJ+dA3SY8ZSLzajD0HJvpQxs4enH/LUopwE=
=ZOjA
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 3 Jan 2022 11:28
d7e066710582b2abbb84b5387fbe60bfcec360e4.camel@telenet.be
Philip McGrath schreef op ma 27-12-2021 om 13:38 [-0500]:
Toggle quote (4 lines)
> I think the behavior of `gexp->approximate-sexp` is at least defensible,
> since its documentation (`info guix "gexp->approximate-sexp"`) warns
> that "some information can be lost".

But no information is lost in this case?

$ guix repl
Toggle quote (2 lines)
> (use-modules (guix gexp))
> (gexp->approximate-sexp #~(a . #nil))
$1 = (a) ; sure, the #nil isn't printed ...
Toggle quote (1 lines)
> (cdr $1)
$2 = () ; ... also not printed ...
Toggle quote (1 lines)
> (values (eq? (cdr $1) #nil) (eq? (cdr $1) '()))
$3 = #t ; but it's still there!
$4 = #f

Greetings,
Maxime
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYdLP0hccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7no9AP9qmkinnkmbOY2aj5/hH3V1uG6K
cXbOpPbisV4iGr/xmAD8DIxuo4v6ZNaNOspeqtVxUH6M8wmPxu+fiEcAOO6t3gI=
=B0iV
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 3 Jan 2022 11:49
7ec439e7e4115ce927f7d1a87b2663d5468304cc.camel@telenet.be
Philip McGrath schreef op ma 27-12-2021 om 13:38 [-0500]:
Toggle quote (4 lines)
> I haven't looked at the implementation at all, but extending `write`
> certainly would be a reasonable option, and, longer-term, it might be
> possible to upstream a patch adding the needed behavior.

Not sure what the API should be (an optional argument #:preserve-nil?
#true? A parameter nil-writing-style? A procedure 'write-preserving-
nil'?), but sure.

Toggle quote (11 lines)
> A more radical option could be to use a format other than plain-text
> s-expressions for compiled G-expressions. For example, Racket has a
> forward-compatible "fast-load serialization" binary format for the kinds
> of values that can be embedded in compiled code.[0] There are obvious
> disadvantages to a binary format, but advantages include the ability to
> preserve source-location information and to avoid some the quirks that
> come with functions like `write` and `read`, for historical reasons or
> for the convenience of humans writing code directly. The implementation
> is in Racket, so it should be fairly easy to port to Guile, if that were
> wanted.[1]

The primary purpose of that data format appears to be able to load
S-expressions _fast_, which seems less useful in Guix because every
derivations are only built once (ignoring GC and build failures).
More important in Guix, is being able to _write_ G-exps fast.
Though perhaps fasl can be written fast?

Anyway, this fasl format might be one way to represent preserve
information. jpoiret on #guix was interested in preserving position

Toggle quote (4 lines)
> Or maybe there's something related to Guile bytecode that
> would work, or maybe just making a `#nil`-preserving version of `write`
> would be easier.

The G-exp machinery doesn't compile things to bytecode, because the
bytecode format changes between Guile versions and for bootstrapping,
old Guiles are used (from the 2.0 series IIRC). Also, this can lead
to world-rebuilds whenever an optimisation in guile is added or
tweaked.

Anyway, I think that in the short term, it would be easiest to
modify (guix build json) to not use #nil (though that would lead
to rebuilding all rust and node stuff because it is used in cargo-
build-system). Longer term, maybe '(guix build json)' could be
eliminated and we could use (json) from 'guile-json' instead,
like documented in the manual ((guix)G-Expressions):

‘In the same vein, sometimes you want to import not just pure-Scheme
modules, but also “extensions” such as Guile bindings to C libraries or
other “full-blown” packages. Say you need the ‘guile-json’ package
available on the build side, here’s how you would do it:

(use-modules (gnu packages guile)) ;for 'guile-json'

(with-extensions (list guile-json)
(gexp->derivation "something-with-json"
#~(begin
(use-modules (json))
...)))’

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYdLU0RccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7pG4AP0feD6y+Er4BeBoDjOhKfBybZ5W
U6KBOKMlRxxtI0UOkgEA3GXdGgjWjhI/m+QL30aZfhMHBNl2ihSvCKy3XUA2JAE=
=kTxC
-----END PGP SIGNATURE-----


?