asdf-build-system packages have priority over user ones

Done

Details

2 participants

Guillaume Le Vaillant
Pierre Neidhardt

Owner: unassigned

Submitted by: Pierre Neidhardt

Severity: normal

Pierre Neidhardt wrote on 3 Dec 2020 10:42

Recipients:(address . bug-guix@gnu.org)(name . Guillaume Le Vaillant)(address . glv@posteo.net)

Message-ID:87wnxzmdiy.fsf@ambrevar.xyz

Since staging was merged a few days ago, I've noticed an undesirable

side-effect of our revamped asdf-build-system: the systems packages have

priority over the user-local ones.

Here is the default list of registries:

Toggle snippet (11 lines)(defparameter* *default-source-registries*
    '(environment-source-registry
      user-source-registry
      user-source-registry-directory
      default-user-source-registry
      system-source-registry
      system-source-registry-directory
      default-system-source-registry)
    "List of default source registries" "3.1.0.102")

`user-source-registry' points to what is read in

~/.config/common-lisp/source-registry.conf(.d/)?, so nothing by default.

user-source-registry-directory reads what's in XDG_CONFIG_DIRS, and it

finds the Guix packages there.

The default-user-source-registry expands to

Toggle snippet (8 lines)

(:SOURCE-REGISTRY (:TREE (:HOME "common-lisp/"))

(:DIRECTORY (:HOME ".sbcl/systems/"))

(:DIRECTORY

#P"/home/ambrevar/.local/share/common-lisp/systems/")

(:TREE #P"/home/ambrevar/.local/share/common-lisp/source/")

:INHERIT-CONFIGURATION)

which is where I would drop my user-local packages.

Finally, default-system-source-registry reads what's in XDG_DATA_DIRS,

and it finds the packages there again, but it does not read the

Guix-generated conf which collects the dependencies of each package.

I believe the issue is that `user-source-registry-directory' should read

from XDG_CONFIG_HOME, and not XDG_CONFIG_DIRS (which would make it

redundant with `user-source-registry'). This would make it an upstream

issue, but I wanted to double-check I properly understood the problem

with the Guix community first.

Guillaume, thoughts on this?

Pierre Neidhardt

https://ambrevar.xyz/

Guillaume Le Vaillant wrote on 3 Dec 2020 13:04

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . bug-guix@gnu.org)

Message-ID:87im9j2j06.fsf@yamatai

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

Toggle quote (51 lines)> Since staging was merged a few days ago, I've noticed an undesirable
> side-effect of our revamped asdf-build-system: the systems packages have
> priority over the user-local ones.
>
> Here is the default list of registries:
>
> --8<---------------cut here---------------start------------->8---
> (defparameter* *default-source-registries*
>     '(environment-source-registry
>       user-source-registry
>       user-source-registry-directory
>       default-user-source-registry
>       system-source-registry
>       system-source-registry-directory
>       default-system-source-registry)
>     "List of default source registries" "3.1.0.102")
> --8<---------------cut here---------------end--------------->8---
>
> `user-source-registry' points to what is read in
> ~/.config/common-lisp/source-registry.conf(.d/)?, so nothing by default.
>
> user-source-registry-directory reads what's in XDG_CONFIG_DIRS, and it
> finds the Guix packages there.
>
> The default-user-source-registry expands to
>
> --8<---------------cut here---------------start------------->8---
> (:SOURCE-REGISTRY (:TREE (:HOME "common-lisp/"))
>                 (:DIRECTORY (:HOME ".sbcl/systems/"))
>                 (:DIRECTORY
>                  #P"/home/ambrevar/.local/share/common-lisp/systems/")
>                 (:TREE #P"/home/ambrevar/.local/share/common-lisp/source/")
>                 :INHERIT-CONFIGURATION)
> --8<---------------cut here---------------end--------------->8---
>
> which is where I would drop my user-local packages.
>
> Finally, default-system-source-registry reads what's in XDG_DATA_DIRS,
> and it finds the packages there again, but it does not read the
> Guix-generated conf which collects the dependencies of each package.
>
>
>
> I believe the issue is that `user-source-registry-directory' should read
> from XDG_CONFIG_HOME, and not XDG_CONFIG_DIRS (which would make it
> redundant with `user-source-registry').  This would make it an upstream
> issue, but I wanted to double-check I properly understood the problem
> with the Guix community first.
>
> Guillaume, thoughts on this?

Our cl-asdf package definition has little patches for
'user-source-registry-directory' and
'user-output-translations-directory-pathname' to make ASDF read
configuration files in all the directories listed in 'XDG_CONFIG_DIRS'
(IIRC regular ASDF was only reading from the first valid one in
'XDG_CONFIG_HOME' or 'XDG_CONFIG_DIRS'):

Toggle snippet (17 lines)(substitute* dst-asdf
  (("\\(xdg-config-pathname \\*source-registry-directory\\* direction\\)")
   "`(:source-registry
      ,@(loop
           for dir in (xdg-config-dirs
                       \"common-lisp/source-registry.conf.d/\")
           collect `(:include ,dir))
      :inherit-configuration)")
  (("\\(xdg-config-pathname \\*output-translations-directory\\* direction\\)")
   "`(:output-translations
      ,@(loop
           for dir in (xdg-config-dirs
                       \"common-lisp/asdf-output-translations.conf.d/\")
           collect `(:include ,dir))
      :inherit-configuration)")))

I guess 'XDG_CONFIG_HOME' is missing here and we could add it in the

list before the 'XDG_CONFIG_DIRS' directories. Something like:

Toggle snippet (19 lines)(substitute* dst-asdf
  (("\\(xdg-config-pathname \\*source-registry-directory\\* direction\\)")
   "`(:source-registry
      (:include (xdg-config-home \"common-lisp/source-registry.conf.d/\"))
      ,@(loop
           for dir in (xdg-config-dirs
                       \"common-lisp/source-registry.conf.d/\")
           collect `(:include ,dir))
      :inherit-configuration)")
  (("\\(xdg-config-pathname \\*output-translations-directory\\* direction\\)")
   "`(:output-translations
      (:include (xdg-config-home \"common-lisp/asdf-output-translations.conf.d/\"))
      ,@(loop
           for dir in (xdg-config-dirs
                       \"common-lisp/asdf-output-translations.conf.d/\")
           collect `(:include ,dir))
      :inherit-configuration)")))

Or maybe we could make a better patch not changing
'user-source-registry-directory', but adding a new
'guix-source-registry' between 'default-user-source-registry' and
'system-source-registry' in the list of registries (and likewise for
output-translations).
I think this way the systems in '$HOME/.config/...', '$HOME/.local/...'
or '$HOME/common-lisp/...' will have priority on the ones in Guix
profiles.

What do you think?

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX8jUSQ8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j80MwD+IUbf1WqpRRobAUhPWcFZM2FolI7tBPEztCd6

b/ac7twA/R9l9WVYDHkaS+nb28pPTo5Gr1v7K2eKjT/p9nE7Qd/T

=iXrd

-----END PGP SIGNATURE-----

Pierre Neidhardt wrote on 3 Dec 2020 13:40

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)(address . bug-guix@gnu.org)

Message-ID:87360nm5b1.fsf@ambrevar.xyz

Oh, I had missed that! Good thing I asked you first then ;)

Toggle quote (20 lines)> --8<---------------cut here---------------start------------->8---
> (substitute* dst-asdf
>   (("\\(xdg-config-pathname \\*source-registry-directory\\* direction\\)")
>    "`(:source-registry
>       (:include (xdg-config-home \"common-lisp/source-registry.conf.d/\"))
>       ,@(loop
>            for dir in (xdg-config-dirs
>                        \"common-lisp/source-registry.conf.d/\")
>            collect `(:include ,dir))
>       :inherit-configuration)")
>   (("\\(xdg-config-pathname \\*output-translations-directory\\* direction\\)")
>    "`(:output-translations
>       (:include (xdg-config-home \"common-lisp/asdf-output-translations.conf.d/\"))
>       ,@(loop
>            for dir in (xdg-config-dirs
>                        \"common-lisp/asdf-output-translations.conf.d/\")
>            collect `(:include ,dir))
>       :inherit-configuration)")))
> --8<---------------cut here---------------end--------------->8---

Hmm, I don't think this would work when the user has no

~/.config/common-lisp/source-registry.conf.

Would it?

Toggle quote (9 lines)

> Or maybe we could make a better patch not changing

> 'user-source-registry-directory', but adding a new

> 'guix-source-registry' between 'default-user-source-registry' and

> 'system-source-registry' in the list of registries (and likewise for

> output-translations).

> I think this way the systems in '$HOME/.config/...', '$HOME/.local/...'

> or '$HOME/common-lisp/...' will have priority on the ones in Guix

> profiles.

I like it!

Possible pitfall: if the user set *default-source-registries* in their

Lisp RC file (e.g. ~/.sbclrc) this will prevent ASDF from seeing this

new Guix source registry.

Alternately, we could simply move the patch from

user-source-registry-directory to system-source-registry-directory; same

for the output translation.

Thoughts?

Pierre Neidhardt

https://ambrevar.xyz/

Guillaume Le Vaillant wrote on 3 Dec 2020 15:29

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 45017@debbugs.gnu.org)

Message-ID:87ft4n2cax.fsf@yamatai

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

Toggle quote (4 lines)

> Alternately, we could simply move the patch from

> user-source-registry-directory to system-source-registry-directory; same

> for the output translation.

What do you think of the attached patch? Does it work for you?

Attachment: 0001-gnu-cl-asdf-Improve-priorities-of-configuration-file.patch

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX8j2Ng8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j8+7AD/THNUo+OfK4z7fkW4cWucZ47kjoqjxPjLCKbY

oVy9nlIA/il1BZr2Y1VQ7q42nbM9ktVjFNdXxiTzb+A4g7puqSj4

=rxih

-----END PGP SIGNATURE-----

Pierre Neidhardt wrote on 4 Dec 2020 09:51

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)(address . 45017@debbugs.gnu.org)

Message-ID:87k0tyuf6y.fsf@ambrevar.xyz

Thanks for taking a shot at this, looks great!

I'll test later, but for now one question: we patch sbcl to use our

cl-asdf, but what about the other compilers? Seems to me that the other

compilers are going to have the same problem, aren't they?

Pierre Neidhardt

https://ambrevar.xyz/

Guillaume Le Vaillant wrote on 4 Dec 2020 10:12

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 45017@debbugs.gnu.org)

Message-ID:871rg6szo9.fsf@yamatai

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

Toggle quote (6 lines)

> Thanks for taking a shot at this, looks great!

> I'll test later, but for now one question: we patch sbcl to use our

> cl-asdf, but what about the other compilers? Seems to me that the other

> compilers are going to have the same problem, aren't they?

SBCL and ECL are patched to use our cl-asdf because it is necessary to
build the sbcl-* and ecl-* packages. Also patching ABCL, CCL, and Clisp
sounds like a good idea. At least all the compilers would use the same
version of ASDF, and hopefully behave in the same way when looking for
configuration files.

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX8n9Zg8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j+FUQD6A+dO9VKhH+ANkQrVe21mscoJbEgVpVqs3mh7

UITf7TgA+wUwfYlb2hIPIlWhybM6l/LH5aJekK2rMYPXuaztzfrA

=h/7l

-----END PGP SIGNATURE-----

Pierre Neidhardt wrote on 4 Dec 2020 13:46

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)(address . 45017@debbugs.gnu.org)

Message-ID:87blf9u4bf.fsf@ambrevar.xyz

Guillaume Le Vaillant <glv@posteo.net> writes:

Toggle quote (6 lines)

> SBCL and ECL are patched to use our cl-asdf because it is necessary to

> build the sbcl-* and ecl-* packages. Also patching ABCL, CCL, and Clisp

> sounds like a good idea. At least all the compilers would use the same

> version of ASDF, and hopefully behave in the same way when looking for

> configuration files.

Yup, that'd be neat :)

Related to this, I've noticed another issue with SBCL packages: when an

SBCL package has a dependency that's updated in ~/common-lisp,

(asdf:load-system ...) tries to recompile it in its folder, which fails

since /gnu/store is read-only.

Example:

- Apply the above patch or edit

~/.config/common-lisp/source-registry.conf

to make sure ~/common-lisp is loaded before system packages.

- Install sbcl-cl-cookie.

- Check out https://github.com/fukamachi/quri:

git clone https://github.com/fukamachi/quri~/common-lisp/quri

- Now run

Toggle snippet (11 lines)

$ sbcl

* (asdf:load-system :cl-cookie)

WARNING: System definition file #P"/gnu/store/81cwmspx3366vdjs6v20rnd8a0xyr6in-sbcl-cl-fad-0.7.6/share/common-lisp/sbcl/cl-fad/cl-fad.asd" contains definition for system "cl-fad-test". Please only define "cl-fad" and secondary systems with a name starting with "cl-fad/" (e.g. "cl-fad/test") in that file.

debugger invoked on a SB-INT:SIMPLE-FILE-ERROR in thread

#<THREAD "main thread" RUNNING {100B790203}>:

Error opening #P"/gnu/store/22q4ydm0pagi4irz0clssgkhkyh115j8-sbcl-cl-cookie-0.9.10-1.cea55ae/lib/common-lisp/sbcl/cl-cookie/src/cl-cookie-tmpGHU3ALSV.fasl":

Read-only file system

Not sure what to do about this.

I'm guessing that Common Lisp development is not practical with Guix'

SBCL packages and I should just stick to the CL packages.

Thoughts?

Pierre Neidhardt

https://ambrevar.xyz/

Guillaume Le Vaillant wrote on 4 Dec 2020 14:53

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 45017@debbugs.gnu.org)

Message-ID:87y2idsmnv.fsf@yamatai

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

Toggle quote (10 lines)> Guillaume Le Vaillant <glv@posteo.net> writes:
>
>> SBCL and ECL are patched to use our cl-asdf because it is necessary to
>> build the sbcl-* and ecl-* packages. Also patching ABCL, CCL, and Clisp
>> sounds like a good idea. At least all the compilers would use the same
>> version of ASDF, and hopefully behave in the same way when looking for
>> configuration files.
>
> Yup, that'd be neat :)

Patches for CCL and Clisp attached. ABCL fails to build with ASDF 3.3.4

and I have not seen an obvious error message in the logs, so this one

will need more investigation...

Toggle quote (37 lines)> Related to this, I've noticed another issue with SBCL packages: when an
> SBCL package has a dependency that's updated in ~/common-lisp,
> (asdf:load-system ...) tries to recompile it in its folder, which fails
> since /gnu/store is read-only.
>
> Example:
>
> - Apply the above patch or edit
>   ~/.config/common-lisp/source-registry.conf
>   to make sure ~/common-lisp is loaded before system packages.
>
> - Install sbcl-cl-cookie.
>
> - Check out https://github.com/fukamachi/quri:
>   git clone https://github.com/fukamachi/quri ~/common-lisp/quri
>
> - Now run
>
> --8<---------------cut here---------------start------------->8---
> $ sbcl
> * (asdf:load-system :cl-cookie)
> WARNING: System definition file #P"/gnu/store/81cwmspx3366vdjs6v20rnd8a0xyr6in-sbcl-cl-fad-0.7.6/share/common-lisp/sbcl/cl-fad/cl-fad.asd" contains definition for system "cl-fad-test". Please only define "cl-fad" and secondary systems with a name starting with "cl-fad/" (e.g. "cl-fad/test") in that file.
>
> debugger invoked on a SB-INT:SIMPLE-FILE-ERROR in thread
> #<THREAD "main thread" RUNNING {100B790203}>:
>   Error opening #P"/gnu/store/22q4ydm0pagi4irz0clssgkhkyh115j8-sbcl-cl-cookie-0.9.10-1.cea55ae/lib/common-lisp/sbcl/cl-cookie/src/cl-cookie-tmpGHU3ALSV.fasl":
>
>     Read-only file system
> --8<---------------cut here---------------end--------------->8---
>
> Not sure what to do about this.
>
> I'm guessing that Common Lisp development is not practical with Guix'
> SBCL packages and I should just stick to the CL packages.
>
> Thoughts?

The Lisp systems of sbcl-* and ecl-* packages installed in a profile are
supposed to be already compiled and immutable, so recompiling them
anyway would require tweaking the ASDF configuration from inside SBCL to
remove the output translations for these systems.
When doing development, it might be more convenient to just use the
sources or source packages of every system that has to be recompiled.

From 5ce14232fafef61fdd5f9412c50dbfbb90f0d03e Mon Sep 17 00:00:00 2001
From: Guillaume Le Vaillant <glv@posteo.net>
Date: Fri, 4 Dec 2020 10:29:34 +0100
Subject: [PATCH 2/4] gnu: ccl: Use system ASDF instead of bundled one.

* gnu/packages/lisp.scm (ccl)[native-inputs]: Add cl-asdf.
  [arguments]: Add 'replace-asdf' phase.
  [native-search-paths]: Add XDG_CONFIG_DIRS and XDG_DATA_DIRS.
---
 gnu/packages/lisp.scm | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

Toggle diff (48 lines)diff --git a/gnu/packages/lisp.scm b/gnu/packages/lisp.scm
index 0038cb0dca..2a8c59546e 100644
--- a/gnu/packages/lisp.scm
+++ b/gnu/packages/lisp.scm
@@ -603,7 +603,8 @@ statistical profiler, a code coverage tool, and many other extensions.")
                 "0x4bjx6cxsjvxyagijhlvmc7jkyxifdvz5q5zvz37028va65243c")
                (_ "15l7cfa4a7jkfwdzsfm4q3n22jnb57imxahpql3h77xin57v1gbz"))))))))
     (native-inputs
-     `(("m4" ,m4)))
+     `(("cl-asdf" ,cl-asdf)
+       ("m4" ,m4)))
     (arguments
      `(#:tests? #f                      ;no 'check' target
        #:modules ((ice-9 match)
@@ -615,6 +616,17 @@ statistical profiler, a code coverage tool, and many other extensions.")
          (add-after 'unpack 'unpack-image
            (lambda* (#:key inputs #:allow-other-keys)
              (invoke "tar" "xzvf" (assoc-ref inputs "ccl-bootstrap"))))
+         (add-after 'unpack 'replace-asdf
+           ;; Use system ASDF instead of bundled one.
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let* ((cl-asdf (assoc-ref inputs "cl-asdf"))
+                    (guix-asdf (string-append
+                                cl-asdf
+                                "/share/common-lisp/source/asdf/asdf.lisp"))
+                    (out (string-append (assoc-ref outputs "out")))
+                    (contrib-asdf "tools/asdf.lisp"))
+               (delete-file contrib-asdf)
+               (copy-file guix-asdf contrib-asdf))))
          (delete 'configure)
          (add-before 'build 'pre-build
            ;; Enter the source directory for the current platform's lisp
@@ -678,6 +690,13 @@ statistical profiler, a code coverage tool, and many other extensions.")
                      "exec -a \"$0\" " libdir kernel " \"$@\"\n"))))
                (chmod wrapper #o755))
              #t)))))
+    (native-search-paths
+     (list (search-path-specification
+            (variable "XDG_DATA_DIRS")
+            (files '("share")))
+           (search-path-specification
+            (variable "XDG_CONFIG_DIRS")
+            (files '("etc")))))
     (supported-systems '("i686-linux" "x86_64-linux" "armhf-linux"))
     (home-page "https://ccl.clozure.com/")
     (synopsis "Common Lisp implementation")
-- 
2.29.2

From 15c7a20030a7e7643af7948982791074e4c358be Mon Sep 17 00:00:00 2001
From: Guillaume Le Vaillant <glv@posteo.net>
Date: Fri, 4 Dec 2020 10:48:41 +0100
Subject: [PATCH 3/4] gnu: clisp: Use system ASDF instead of bundled one.

* gnu/packages/lisp.scm (clisp)[native-inputs]: Add cl-asdf.
  [arguments]: Add 'replace-asdf' phase.
  [native-search-paths]: Add XDG_CONFIG_DIRS and XDG_DATA_DIRS.
---
 gnu/packages/lisp.scm | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

Toggle diff (42 lines)diff --git a/gnu/packages/lisp.scm b/gnu/packages/lisp.scm
index 2a8c59546e..07d54747cb 100644
--- a/gnu/packages/lisp.scm
+++ b/gnu/packages/lisp.scm
@@ -322,6 +322,8 @@ supporting ASDF, Sockets, Gray streams, MOP, and other useful components.")
        (sha256
         (base32 "0k2dmgl0miz3767iks4p0mvp6xw0ysyxhjpklyh11j010rmh6hqb"))))
     (build-system gnu-build-system)
+    (native-inputs
+     `(("cl-asdf" ,cl-asdf)))
     (inputs `(("libffcall" ,libffcall)
               ("ncurses" ,ncurses)
               ("readline" ,readline)
@@ -356,7 +358,25 @@ supporting ASDF, Sockets, Gray streams, MOP, and other useful components.")
                (("/bin/sh") "sh"))
              (substitute* '("src/clisp-link.in")
                (("/bin/pwd") "pwd"))
-             #t)))))
+             #t))
+         (add-after 'unpack 'replace-asdf
+           ;; Use system ASDF instead of bundled one.
+           (lambda* (#:key inputs outputs #:allow-other-keys)
+             (let* ((cl-asdf (assoc-ref inputs "cl-asdf"))
+                    (guix-asdf (string-append
+                                cl-asdf
+                                "/share/common-lisp/source/asdf/asdf.lisp"))
+                    (out (string-append (assoc-ref outputs "out")))
+                    (contrib-asdf "modules/asdf/asdf.lisp"))
+               (delete-file contrib-asdf)
+               (copy-file guix-asdf contrib-asdf)))))))
+    (native-search-paths
+     (list (search-path-specification
+            (variable "XDG_DATA_DIRS")
+            (files '("share")))
+           (search-path-specification
+            (variable "XDG_CONFIG_DIRS")
+            (files '("etc")))))
     (home-page "https://clisp.sourceforge.io/")
     (synopsis "A Common Lisp implementation")
     (description
-- 
2.29.2

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX8o/RA8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j+4jQD/eU9QBTkS74UZ/tBs4a7jjBcFP7e7Wq2lZJkE

LerqxAIA/iex2+Wa83D30T9woxkcDwpjJqheZIPYRK75rrG8ZpYm

=bgpt

-----END PGP SIGNATURE-----

Guillaume Le Vaillant wrote on 4 Dec 2020 18:30

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 45017@debbugs.gnu.org)

Message-ID:87v9dhscm1.fsf@yamatai

Guillaume Le Vaillant <glv@posteo.net> skribis:

Toggle quote (7 lines)

> The Lisp systems of sbcl-* and ecl-* packages installed in a profile are

> supposed to be already compiled and immutable, so recompiling them

> anyway would require tweaking the ASDF configuration from inside SBCL to

> remove the output translations for these systems.

> When doing development, it might be more convenient to just use the

> sources or source packages of every system that has to be recompiled.

I think entering the following expressions just after starting SBCL will

allow development mixing libraries from user directories and from Guix

profiles:

Toggle snippet (8 lines)

(require :asdf)

(asdf:clear-output-translations)

(asdf:initialize-output-translations

'(:output-translations

:enable-user-cache

:ignore-inherited-configuration))

However with this configuration, the pre-compiled files in
"/gnu/store/..." will not be used at all; everything will be compiled
again if necessary and put in the user cache in ".cache/common-lisp/...".

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX8pyJg8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j8gPAD/Y2NUlxIEiBAfCofgCpJztKd08iunmtuxmnQ0

0RDDH5UBAJqhtF08EhoX+EvwaWahrmZqd65GLaqwMc0O1akM0MrW

=tjXu

-----END PGP SIGNATURE-----

Pierre Neidhardt wrote on 5 Dec 2020 09:18

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)(address . 45017@debbugs.gnu.org)

Message-ID:877dpwsm1h.fsf@ambrevar.xyz

A few comments:

Toggle quote (23 lines)> @@ -603,7 +603,8 @@ statistical profiler, a code coverage tool, and many other extensions.")
>                  "0x4bjx6cxsjvxyagijhlvmc7jkyxifdvz5q5zvz37028va65243c")
>                 (_ "15l7cfa4a7jkfwdzsfm4q3n22jnb57imxahpql3h77xin57v1gbz"))))))))
>      (native-inputs
> -     `(("m4" ,m4)))
> +     `(("cl-asdf" ,cl-asdf)
> +       ("m4" ,m4)))
>      (arguments
>       `(#:tests? #f                      ;no 'check' target
>         #:modules ((ice-9 match)
> @@ -615,6 +616,17 @@ statistical profiler, a code coverage tool, and many other extensions.")
>           (add-after 'unpack 'unpack-image
>             (lambda* (#:key inputs #:allow-other-keys)
>               (invoke "tar" "xzvf" (assoc-ref inputs "ccl-bootstrap"))))
> +         (add-after 'unpack 'replace-asdf
> +           ;; Use system ASDF instead of bundled one.
> +           (lambda* (#:key inputs outputs #:allow-other-keys)
> +             (let* ((cl-asdf (assoc-ref inputs "cl-asdf"))
> +                    (guix-asdf (string-append
> +                                cl-asdf
> +                                "/share/common-lisp/source/asdf/asdf.lisp"))
> +                    (out (string-append (assoc-ref outputs "out")))

"out" is unused. By the way, the same "out" is unused in SBCL.

Toggle quote (3 lines)

> + (contrib-asdf "tools/asdf.lisp"))

> + (delete-file contrib-asdf)

I think you don't need to delete the file, copy-file should overwrite.

That's what SBCL does.

Same with CLISP.

Pierre Neidhardt

https://ambrevar.xyz/

Pierre Neidhardt wrote on 5 Dec 2020 10:17

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)(address . 45017@debbugs.gnu.org)

Message-ID:874kl0sjbe.fsf@ambrevar.xyz

Tested!

I've installed sbcl to my "common-lisp" profile along quri.

I've also cloned quri to ~/common-lisp.

Then:

Toggle snippet (14 lines)

$ sbcl

* (asdf:locate-system :quri)

NIL

#P"/gnu/store/5gj1inwiqpn2fm9w384zd0grpyadx96m-sbcl-quri-0.1.0-2.b53231c/share/common-lisp/sbcl/quri/quri.asd"

NIL

;; Indeed:

* (asdf:user-source-registry-directory)

#P"/gnu/store/75qppl3a062b138fkrn324qq8f912zqh-profile/etc/common-lisp/source-registry.conf.d/"

It does not work because the sbcl package definition sets the

XDG_CONFIG_DIRS native search path which is picked by

user-source-registry-directory, which has higher priority than

default-user-source-registry.

I think the flaw is ASDF's this time:

Toggle snippet (15 lines)

(defun user-source-registry-directory (&key (direction :input))

(xdg-config-pathname *source-registry-directory* direction))

;...

(defun xdg-config-pathnames (&rest more)

"Return a list of pathnames for application configuration.

MORE may contain specifications for a subpath relative to these directories: a

subpathname specification and keyword arguments as per RESOLVE-LOCATION \(see

also \"Configuration DSL\"\) in the ASDF manual."

(filter-pathname-set

`(,(xdg-config-home more)

,@(xdg-config-dirs more))))

So the user registry looks into XDG_CONFIG_DIRS, while really it should

just look at XDG_CONFIG_HOME.

So this patch should do (untested):

Toggle snippet (6 lines)

- (defun user-source-registry-directory (&key (direction :input))

- (xdg-config-pathname *source-registry-directory* direction))

+ (defun user-source-registry-directory (&key (direction :input))

+ (find-preferred-file (list (xdg-config-home *source-registry-directory*)) :direction direction))

If you confirm, then we should also report upstream I think.

Cheers!

Pierre Neidhardt

https://ambrevar.xyz/

Guillaume Le Vaillant wrote on 5 Dec 2020 10:42

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 45017@debbugs.gnu.org)

Message-ID:871rg4lhb0.fsf@yamatai

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

Toggle quote (28 lines)> A few comments:
>
>> @@ -603,7 +603,8 @@ statistical profiler, a code coverage tool, and many other extensions.")
>>                  "0x4bjx6cxsjvxyagijhlvmc7jkyxifdvz5q5zvz37028va65243c")
>>                 (_ "15l7cfa4a7jkfwdzsfm4q3n22jnb57imxahpql3h77xin57v1gbz"))))))))
>>      (native-inputs
>> -     `(("m4" ,m4)))
>> +     `(("cl-asdf" ,cl-asdf)
>> +       ("m4" ,m4)))
>>      (arguments
>>       `(#:tests? #f                      ;no 'check' target
>>         #:modules ((ice-9 match)
>> @@ -615,6 +616,17 @@ statistical profiler, a code coverage tool, and many other extensions.")
>>           (add-after 'unpack 'unpack-image
>>             (lambda* (#:key inputs #:allow-other-keys)
>>               (invoke "tar" "xzvf" (assoc-ref inputs "ccl-bootstrap"))))
>> +         (add-after 'unpack 'replace-asdf
>> +           ;; Use system ASDF instead of bundled one.
>> +           (lambda* (#:key inputs outputs #:allow-other-keys)
>> +             (let* ((cl-asdf (assoc-ref inputs "cl-asdf"))
>> +                    (guix-asdf (string-append
>> +                                cl-asdf
>> +                                "/share/common-lisp/source/asdf/asdf.lisp"))
>> +                    (out (string-append (assoc-ref outputs "out")))
>
> "out" is unused.  By the way, the same "out" is unused in SBCL.
>

Indeed, I'll fix that.

Toggle quote (8 lines)

>> + (contrib-asdf "tools/asdf.lisp"))

>> + (delete-file contrib-asdf)

> I think you don't need to delete the file, copy-file should overwrite.

> That's what SBCL does.

> Same with CLISP.

Actually, for CCL and Clisp, I get a "In procedure copy-file: Permission
denied" error if I don't delete the current file before copying the new
one.

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX8tWIw8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j8SygD9HSufWhgk2xmdoB5tFwVIP6+LmCkFayKi44ju

4rLfb5oA/3wUGLwn7B4+q+zzVWt5m+AC9tOG4SmEqB1yWi2I6M6k

=/vO5

-----END PGP SIGNATURE-----

Guillaume Le Vaillant wrote on 5 Dec 2020 11:03

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 45017@debbugs.gnu.org)

Message-ID:87y2ick1sm.fsf@yamatai

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

Toggle quote (49 lines)> Tested!
>
> I've installed sbcl to my "common-lisp" profile along quri.
> I've also cloned quri to ~/common-lisp.
>
> Then:
>
> --8<---------------cut here---------------start------------->8---
> $ sbcl
> * (asdf:locate-system :quri)
> T
> NIL
> #P"/gnu/store/5gj1inwiqpn2fm9w384zd0grpyadx96m-sbcl-quri-0.1.0-2.b53231c/share/common-lisp/sbcl/quri/quri.asd"
> NIL
> NIL
> NIL
>
> ;; Indeed:
> * (asdf:user-source-registry-directory)
> #P"/gnu/store/75qppl3a062b138fkrn324qq8f912zqh-profile/etc/common-lisp/source-registry.conf.d/"
> --8<---------------cut here---------------end--------------->8---
>
> It does not work because the sbcl package definition sets the
> XDG_CONFIG_DIRS native search path which is picked by
> user-source-registry-directory, which has higher priority than
> default-user-source-registry.
>
> I think the flaw is ASDF's this time:
>
> --8<---------------cut here---------------start------------->8---
>   (defun user-source-registry-directory (&key (direction :input))
>     (xdg-config-pathname *source-registry-directory* direction))
>
> ;...
>
>   (defun xdg-config-pathnames (&rest more)
>     "Return a list of pathnames for application configuration.
> MORE may contain specifications for a subpath relative to these directories: a
> subpathname specification and keyword arguments as per RESOLVE-LOCATION \(see
> also \"Configuration DSL\"\) in the ASDF manual."
>     (filter-pathname-set
>      `(,(xdg-config-home more)
>        ,@(xdg-config-dirs more))))
> --8<---------------cut here---------------end--------------->8---
>
> So the user registry looks into XDG_CONFIG_DIRS, while really it should
> just look at XDG_CONFIG_HOME.
>

The same test works for me. I think it's because I have

a ".config/common-lisp/source-registry.conf.d" directory (even if it's

empty), and '(user-source-registry-directory)' returns it instead of the

first directory in XDG_CONFIG_DIRS.

Toggle quote (13 lines)> So this patch should do (untested):
>
> --8<---------------cut here---------------start------------->8---
> - (defun user-source-registry-directory (&key (direction :input))
> -   (xdg-config-pathname *source-registry-directory* direction))
> + (defun user-source-registry-directory (&key (direction :input))
> +   (find-preferred-file (list (xdg-config-home *source-registry-directory*)) :direction direction))
> --8<---------------cut here---------------end--------------->8---
>
> If you confirm, then we should also report upstream I think.
>
> Cheers!

I don't know if the fact that ASDF can give priority to XDG_CONFIG_DIRS
over "~/common-lisp" is a feature or a bug, so you could ask upstream.
Anyway in our case, your patch makes sense since we always want to give
priority to "~/common-lisp" and other usual user directories. I'll add
it to our patch for cl-asdf.

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX8ta6Q8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j84mAD/UGNSdhWLWWSnHbMkCoJqzDavOH1ZBME1mctC

AbdqhDUA/3Pfy+5o+qmpXwuuGkbPIlxmwZVHfmpo12low3Rpdptb

=cVLs

-----END PGP SIGNATURE-----

Guillaume Le Vaillant wrote on 5 Dec 2020 11:52

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 45017@debbugs.gnu.org)

Message-ID:87v9dgjzic.fsf@yamatai

Updated patches in attachment.

Do you see something else to fix or improve?

Attachment: 0001-gnu-cl-asdf-Improve-priorities-of-configuration-file.patch

From 53a63ff01929f4afe0d841ee9f4a09a5514b0ca8 Mon Sep 17 00:00:00 2001
From: Guillaume Le Vaillant <glv@posteo.net>
Date: Fri, 4 Dec 2020 10:29:34 +0100
Subject: [PATCH 2/6] gnu: ccl: Use system ASDF instead of bundled one.

* gnu/packages/lisp.scm (ccl)[native-inputs]: Add cl-asdf.
  [arguments]: Add 'replace-asdf' phase.
  [native-search-paths]: Add XDG_CONFIG_DIRS and XDG_DATA_DIRS.
---
 gnu/packages/lisp.scm | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

Toggle diff (47 lines)diff --git a/gnu/packages/lisp.scm b/gnu/packages/lisp.scm
index 0038cb0dca..a4cbc28ffc 100644
--- a/gnu/packages/lisp.scm
+++ b/gnu/packages/lisp.scm
@@ -603,7 +603,8 @@ statistical profiler, a code coverage tool, and many other extensions.")
                 "0x4bjx6cxsjvxyagijhlvmc7jkyxifdvz5q5zvz37028va65243c")
                (_ "15l7cfa4a7jkfwdzsfm4q3n22jnb57imxahpql3h77xin57v1gbz"))))))))
     (native-inputs
-     `(("m4" ,m4)))
+     `(("cl-asdf" ,cl-asdf)
+       ("m4" ,m4)))
     (arguments
      `(#:tests? #f                      ;no 'check' target
        #:modules ((ice-9 match)
@@ -615,6 +616,16 @@ statistical profiler, a code coverage tool, and many other extensions.")
          (add-after 'unpack 'unpack-image
            (lambda* (#:key inputs #:allow-other-keys)
              (invoke "tar" "xzvf" (assoc-ref inputs "ccl-bootstrap"))))
+         (add-after 'unpack 'replace-asdf
+           ;; Use system ASDF instead of bundled one.
+           (lambda* (#:key inputs #:allow-other-keys)
+             (let* ((cl-asdf (assoc-ref inputs "cl-asdf"))
+                    (guix-asdf (string-append
+                                cl-asdf
+                                "/share/common-lisp/source/asdf/asdf.lisp"))
+                    (contrib-asdf "tools/asdf.lisp"))
+               (delete-file contrib-asdf)
+               (copy-file guix-asdf contrib-asdf))))
          (delete 'configure)
          (add-before 'build 'pre-build
            ;; Enter the source directory for the current platform's lisp
@@ -678,6 +689,13 @@ statistical profiler, a code coverage tool, and many other extensions.")
                      "exec -a \"$0\" " libdir kernel " \"$@\"\n"))))
                (chmod wrapper #o755))
              #t)))))
+    (native-search-paths
+     (list (search-path-specification
+            (variable "XDG_DATA_DIRS")
+            (files '("share")))
+           (search-path-specification
+            (variable "XDG_CONFIG_DIRS")
+            (files '("etc")))))
     (supported-systems '("i686-linux" "x86_64-linux" "armhf-linux"))
     (home-page "https://ccl.clozure.com/")
     (synopsis "Common Lisp implementation")
-- 
2.29.2

From a855535add82bf0ba4cacf84a526be38f8230440 Mon Sep 17 00:00:00 2001
From: Guillaume Le Vaillant <glv@posteo.net>
Date: Fri, 4 Dec 2020 10:48:41 +0100
Subject: [PATCH 3/6] gnu: clisp: Use system ASDF instead of bundled one.

* gnu/packages/lisp.scm (clisp)[native-inputs]: Add cl-asdf.
  [arguments]: Add 'replace-asdf' phase.
  [native-search-paths]: Add XDG_CONFIG_DIRS and XDG_DATA_DIRS.
---
 gnu/packages/lisp.scm | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

Toggle diff (41 lines)diff --git a/gnu/packages/lisp.scm b/gnu/packages/lisp.scm
index a4cbc28ffc..2ec4da8fbb 100644
--- a/gnu/packages/lisp.scm
+++ b/gnu/packages/lisp.scm
@@ -322,6 +322,8 @@ supporting ASDF, Sockets, Gray streams, MOP, and other useful components.")
        (sha256
         (base32 "0k2dmgl0miz3767iks4p0mvp6xw0ysyxhjpklyh11j010rmh6hqb"))))
     (build-system gnu-build-system)
+    (native-inputs
+     `(("cl-asdf" ,cl-asdf)))
     (inputs `(("libffcall" ,libffcall)
               ("ncurses" ,ncurses)
               ("readline" ,readline)
@@ -356,7 +358,24 @@ supporting ASDF, Sockets, Gray streams, MOP, and other useful components.")
                (("/bin/sh") "sh"))
              (substitute* '("src/clisp-link.in")
                (("/bin/pwd") "pwd"))
-             #t)))))
+             #t))
+         (add-after 'unpack 'replace-asdf
+           ;; Use system ASDF instead of bundled one.
+           (lambda* (#:key inputs #:allow-other-keys)
+             (let* ((cl-asdf (assoc-ref inputs "cl-asdf"))
+                    (guix-asdf (string-append
+                                cl-asdf
+                                "/share/common-lisp/source/asdf/asdf.lisp"))
+                    (contrib-asdf "modules/asdf/asdf.lisp"))
+               (delete-file contrib-asdf)
+               (copy-file guix-asdf contrib-asdf)))))))
+    (native-search-paths
+     (list (search-path-specification
+            (variable "XDG_DATA_DIRS")
+            (files '("share")))
+           (search-path-specification
+            (variable "XDG_CONFIG_DIRS")
+            (files '("etc")))))
     (home-page "https://clisp.sourceforge.io/")
     (synopsis "A Common Lisp implementation")
     (description
-- 
2.29.2

From 5b3ae58f494a1728143b85abb5ef5655f04499d7 Mon Sep 17 00:00:00 2001
From: Guillaume Le Vaillant <glv@posteo.net>
Date: Sat, 5 Dec 2020 10:11:38 +0100
Subject: [PATCH 4/6] gnu: ecl: Simplify 'replace-asdf' phase.

* gnu/packages/lisp.scm (ecl)[arguments]: Remove unused 'out' variable in
  'replace-asdf' phase.
---
 gnu/packages/lisp.scm | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Toggle diff (20 lines)diff --git a/gnu/packages/lisp.scm b/gnu/packages/lisp.scm
index 2ec4da8fbb..a540174c88 100644
--- a/gnu/packages/lisp.scm
+++ b/gnu/packages/lisp.scm
@@ -250,12 +250,11 @@ interface to the Tk widget system.")
          (delete 'check)
          (add-after 'unpack 'replace-asdf
            ;; Use system ASDF instead of bundled one.
-           (lambda* (#:key inputs outputs #:allow-other-keys)
+           (lambda* (#:key inputs #:allow-other-keys)
              (let* ((cl-asdf (assoc-ref inputs "cl-asdf"))
                     (guix-asdf (string-append
                                 cl-asdf
                                 "/share/common-lisp/source/asdf/asdf.lisp"))
-                    (out (string-append (assoc-ref outputs "out")))
                     (contrib-asdf "contrib/asdf/asdf.lisp"))
                (copy-file guix-asdf contrib-asdf))
              #t))
-- 
2.29.2

From 466d7cdbee216df3fa298196228e265d9431dcfa Mon Sep 17 00:00:00 2001
From: Guillaume Le Vaillant <glv@posteo.net>
Date: Sat, 5 Dec 2020 10:22:42 +0100
Subject: [PATCH 5/6] gnu: sbcl: Simplify 'replace-asdf' phase.

* gnu/packages/lisp.scm (sbcl)[arguments]: Remove unused 'out' variable in
  'replace-asdf' phase.
---
 gnu/packages/lisp.scm | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

Toggle diff (20 lines)diff --git a/gnu/packages/lisp.scm b/gnu/packages/lisp.scm
index a540174c88..3d446290a7 100644
--- a/gnu/packages/lisp.scm
+++ b/gnu/packages/lisp.scm
@@ -444,12 +444,11 @@ an interpreter, a compiler, a debugger, and much more.")
            ;; of ASDF we use to build software; therefore, replace the contrib
            ;; ASDF with the version packaged into Guix.
            ;; [1] - https://bugs.launchpad.net/sbcl/+bug/1823442
-           (lambda* (#:key inputs outputs #:allow-other-keys)
+           (lambda* (#:key inputs #:allow-other-keys)
              (let* ((cl-asdf (assoc-ref inputs "cl-asdf"))
                     (guix-asdf (string-append
                                 cl-asdf
                                 "/share/common-lisp/source/asdf/asdf.lisp"))
-                    (out (string-append (assoc-ref outputs "out")))
                     (contrib-asdf "contrib/asdf/asdf.lisp"))
                (copy-file guix-asdf contrib-asdf))
              #t))
-- 
2.29.2

-----BEGIN PGP SIGNATURE-----

iIUEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX8tmew8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j8QHgD+KL5dSHMlzdz2UNWf6fJAqmfXkq5//Q5yfXvh

PznRA9wA/i71IMC7cNvar6Hr9mc9sXdQc9jRZwNy+S/Fu8lFoISs

=7zdY

-----END PGP SIGNATURE-----

Pierre Neidhardt wrote on 5 Dec 2020 15:42

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)(address . 45017@debbugs.gnu.org)

Message-ID:87czzofh63.fsf@ambrevar.xyz

Guillaume Le Vaillant <glv@posteo.net> writes:

Toggle quote (3 lines)

> Updated patches in attachment.

> Do you see something else to fix or improve?

Tested and approved!

I suggest we merge on master since this is not very disruptive and it

fixes a regression introduced with the build system overhaul.

Thoughts?

Pierre Neidhardt

https://ambrevar.xyz/

Guillaume Le Vaillant wrote on 5 Dec 2020 16:17

Recipients:(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 45017-done@debbugs.gnu.org)

Message-ID:874kl0qo30.fsf@yamatai

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

Toggle quote (12 lines)> Guillaume Le Vaillant <glv@posteo.net> writes:
>
>> Updated patches in attachment.
>> Do you see something else to fix or improve?
>
> Tested and approved!
>
> I suggest we merge on master since this is not very disruptive and it
> fixes a regression introduced with the build system overhaul.
>
> Thoughts?

Thanks for the review. Pushed as

031fbebafef81244a72e584a46b3fcc219256fcb and following.

-----BEGIN PGP SIGNATURE-----

iIQEAREKAC0WIQTLxZxm7Ce5cXlAaz5r6CCK3yH+PwUCX8ukkw8cZ2x2QHBvc3Rl

by5uZXQACgkQa+ggit8h/j8UkQEAlptiDYowjO3PMtoy8CP5SLsCuVyth5bSeqN4

lXo8l9sA+KzxY6viYHl+BFzTYbyW1S/PSx6Ux+fNBzPg5lBxJQE=

=UDkC

-----END PGP SIGNATURE-----

Closed

Pierre Neidhardt wrote on 5 Dec 2020 16:19

Recipients:(name . Guillaume Le Vaillant)(address . glv@posteo.net)(address . 45017-done@debbugs.gnu.org)

Message-ID:875z5gffgk.fsf@ambrevar.xyz

Thanks a lot!

Pierre Neidhardt

https://ambrevar.xyz/

Closed

Your comment

This issue is archived.

To comment on this conversation send an email to 45017@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date