with-extensions does not add native extensions to the load path

Open

Details

4 participants

Timo Wilken
Ludovic Courtès
Z572
Tomas Volf

Owner: unassigned

Submitted by: Tomas Volf

Severity: normal

Debbugs page

Tomas Volf wrote 4 months ago

Recipients:(address . bug-guix@gnu.org)

Message-ID:871pyzdwzi.fsf@wolfsden.cz

The documentation for `with-extensions' says:

Toggle quote (4 lines)

> In the same vein, sometimes you want to import not just pure-Scheme

> modules, but also “extensions” such as Guile bindings to C libraries

> or other “full-blown” packages.

However it does not actually add those C libraries into

(guile-extensions-path), which means that trying to actually use a

library imported this way will lead to an error.

I guess `load-path-expression' in (guix gexp) needs to also append to

the `guile-extensions-path' based on the search-paths of the imported

packages?

Reproduction is simple. Make sure you *do not* have guile-yamlpp

installed in your profile. Then build the `test' into a store item.

Toggle snippet (6 lines)

(use-modules (gnu packages guile-xyz))

(define test

(with-extensions (list guile-yamlpp)

(program-file "test" #~(use-modules (yamlpp)))))

And try to run it:

Toggle snippet (39 lines)$ /gnu/store/nq0sahcnph5sxms4irq4inv2hq2hfbzr-test
Backtrace:
          19 (primitive-load "/gnu/store/nq0sahcnph5sxms4irq4inv2hq2hfbzr-test")
In ice-9/eval.scm:
   721:20 18 (primitive-eval _)
In ice-9/psyntax.scm:
  1229:36 17 (expand-top-sequence _ _ _ #f _ _ _)
  1221:19 16 (parse _ (("placeholder" placeholder)) ((top) #(ribcage () () ())) _ e (eval) #)
   259:10 15 (parse _ (("placeholder" placeholder)) (()) _ c&e (eval) (hygiene guile-user))
In ice-9/boot-9.scm:
  3935:20 14 (process-use-modules _)
   222:17 13 (map1 (((yamlpp))))
  3936:31 12 (_ ((yamlpp)))
  3327:17 11 (resolve-interface (yamlpp) #:select _ #:hide _ #:prefix _ #:renamer _ # _)
In ice-9/threads.scm:
    390:8 10 (_ _)
In ice-9/boot-9.scm:
  3253:13  9 (_)
In ice-9/threads.scm:
    390:8  8 (_ _)
In ice-9/boot-9.scm:
  3544:20  7 (_)
   2836:4  6 (save-module-excursion _)
  3564:26  5 (_)
In unknown file:
           4 (primitive-load-path "yamlpp" #<procedure 7f405e4fd560 at ice-9/boot-9.scm:35…>)
In yamlpp.scm:
     65:1  3 (_)
In unknown file:
           2 (load-extension "libguile-yamlpp" "init")
In system/foreign-library.scm:
   190:25  1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)
In unknown file:
           0 (dlopen "libguile-yamlpp.so" 1)

ERROR: In procedure dlopen:
In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"

Have a nice day,
Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

Ludovic Courtès wrote 4 months ago

Recipients:(address . 74532@debbugs.gnu.org)

Message-ID:877c8n94kl.fsf@gnu.org

Hi,

Tomas Volf <~@wolfsden.cz> skribis:

Toggle quote (6 lines)

> The documentation for `with-extensions' says:

>> In the same vein, sometimes you want to import not just pure-Scheme

>> modules, but also “extensions” such as Guile bindings to C libraries

>> or other “full-blown” packages.

It’s not just bindings but also pure Scheme libraries like Guile-JSON.

Toggle quote (4 lines)

> However it does not actually add those C libraries into

> (guile-extensions-path), which means that trying to actually use a

> library imported this way will lead to an error.

[...]

Toggle quote (7 lines)

> 190:25 1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)

> In unknown file:

> 0 (dlopen "libguile-yamlpp.so" 1)

> ERROR: In procedure dlopen:

> In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"

Usually, packages like these (guile-gnutls, guile-git, guile-ssh, etc.)

have their .so absolute file name hard-coded, which sidesteps this

problem entirely. I recommend doing that for guile-yamlpp as well.

That said, it would probably make sense to arrange for ‘with-extensions’

to set GUILE_EXTENSIONS_PATH.

Ludo’.

PS: Your MUA sets “Mail-Followup-To: bug-guix@gnu.org”, which is kinda

annoying because that’s the wrong address when replying to a bug.

:-)

Tomas Volf wrote 4 months ago

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 74532@debbugs.gnu.org)

Message-ID:87plm93b7p.fsf@wolfsden.cz

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (12 lines)> Hi,
>
> Tomas Volf <~@wolfsden.cz> skribis:
>
>> The documentation for `with-extensions' says:
>>
>>> In the same vein, sometimes you want to import not just pure-Scheme
>>> modules, but also “extensions” such as Guile bindings to C libraries
>>> or other “full-blown” packages.
>
> It’s not just bindings but also pure Scheme libraries like Guile-JSON.

True, and for those it works well. However the text documentation

explicitly mentions "bindings to C libraries" as one of the use

cases. :)

Toggle quote (18 lines)>
>> However it does not actually add those C libraries into
>> (guile-extensions-path), which means that trying to actually use a
>> library imported this way will lead to an error.
>
> [...]
>
>>    190:25  1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)
>> In unknown file:
>>            0 (dlopen "libguile-yamlpp.so" 1)
>>
>> ERROR: In procedure dlopen:
>> In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"
>
> Usually, packages like these (guile-gnutls, guile-git, guile-ssh, etc.)
> have their .so absolute file name hard-coded, which sidesteps this
> problem entirely.  I recommend doing that for guile-yamlpp as well.

How it that usually done? I cannot do that *before* build, because then

tests would not run (the library is not yet installed into the absolute

path), and I cannot do it after, because I would need to rebuild the .go

files after patching the source code.

Assuming I have a guile library that creates a new .so library during

the build, and the .so library must be loadable by the other modules in

the library during the build (and for the tests), how should I approach

that?

I am pretty sure I cannot just patch the source code, since the library

would not be installed into the absolute path yet during the build.

Honestly, setting GUILE_EXTENSIONS_PATH via pre-inst-env seemed like

fairly elegant solution.

Toggle quote (4 lines)

> That said, it would probably make sense to arrange for ‘with-extensions’

> to set GUILE_EXTENSIONS_PATH.

That would be great.  My current work-around is

  (with-extensions (list guile-wolfsden)
    (program-file
     "audio-cycle-sinks"
     #~(begin
         ;; Bug 74532: Native extensions are not added to the load path.
         (eval-when (expand load eval)
           (let ((ext-path (@ (system foreign-library) guile-extensions-path)))
             ;; Just a temporary hack, we can live with duplicates in the path.
             (ext-path (cons #$(file-append guile-wolfsden
                                            "/lib/guile/3.0/extensions")
                             (ext-path)))))
         ...)))

which is anything but elegant.

Toggle quote (7 lines)

> Ludo’.

> PS: Your MUA sets “Mail-Followup-To: bug-guix@gnu.org”, which is kinda

> annoying because that’s the wrong address when replying to a bug.

> :-)

Thanks for letting me know, I was not aware of that.  After reading
(message)Mailing Lists bit more carefully, adjusting the Posting Styles
and sending a bug fix to the Emacs' bug tracker, I believe it should not
happen anymore.  Sorry for the annoyance.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

Z572 wrote 4 months ago

Recipients:(name . Tomas Volf)(address . ~@wolfsden.cz)(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 74532@debbugs.gnu.org)

Message-ID:87y10x4g2w.fsf@iscas.ac.cn

Tomas Volf <~@wolfsden.cz> writes:

Toggle quote (41 lines)> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi,
>>
>> Tomas Volf <~@wolfsden.cz> skribis:
>>
>>> The documentation for `with-extensions' says:
>>>
>>>> In the same vein, sometimes you want to import not just pure-Scheme
>>>> modules, but also “extensions” such as Guile bindings to C libraries
>>>> or other “full-blown” packages.
>>
>> It’s not just bindings but also pure Scheme libraries like Guile-JSON.
>
> True, and for those it works well.  However the text documentation
> explicitly mentions "bindings to C libraries" as one of the use
> cases. :)
>
>>
>>> However it does not actually add those C libraries into
>>> (guile-extensions-path), which means that trying to actually use a
>>> library imported this way will lead to an error.
>>
>> [...]
>>
>>>    190:25  1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)
>>> In unknown file:
>>>            0 (dlopen "libguile-yamlpp.so" 1)
>>>
>>> ERROR: In procedure dlopen:
>>> In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"
>>
>> Usually, packages like these (guile-gnutls, guile-git, guile-ssh, etc.)
>> have their .so absolute file name hard-coded, which sidesteps this
>> problem entirely.  I recommend doing that for guile-yamlpp as well.
>
> How it that usually done?  I cannot do that *before* build, because then
> tests would not run (the library is not yet installed into the absolute
> path), and I cannot do it after, because I would need to rebuild the .go
> files after patching the source code.

you can use substitute* to adjust source.

e.g.

(or (false-if-exception (load-extension "/path/to/lib-some-object-file.so"))

(load-extension "lib-some-object-file.so"))

Toggle quote (44 lines)>
> Assuming I have a guile library that creates a new .so library during
> the build, and the .so library must be loadable by the other modules in
> the library during the build (and for the tests), how should I approach
> that?
>
> I am pretty sure I cannot just patch the source code, since the library
> would not be installed into the absolute path yet during the build.
>
> Honestly, setting GUILE_EXTENSIONS_PATH via pre-inst-env seemed like
> fairly elegant solution.
>
>>
>> That said, it would probably make sense to arrange for ‘with-extensions’
>> to set GUILE_EXTENSIONS_PATH.
>
> That would be great.  My current work-around is
>
>   (with-extensions (list guile-wolfsden)
>     (program-file
>      "audio-cycle-sinks"
>      #~(begin
>          ;; Bug 74532: Native extensions are not added to the load path.
>          (eval-when (expand load eval)
>            (let ((ext-path (@ (system foreign-library) guile-extensions-path)))
>              ;; Just a temporary hack, we can live with duplicates in the path.
>              (ext-path (cons #$(file-append guile-wolfsden
>                                             "/lib/guile/3.0/extensions")
>                              (ext-path)))))
>          ...)))
>
> which is anything but elegant.
>
>>
>> Ludo’.
>>
>> PS: Your MUA sets “Mail-Followup-To: bug-guix@gnu.org”, which is kinda
>>     annoying because that’s the wrong address when replying to a bug.
>>     :-)
>
> Thanks for letting me know, I was not aware of that.  After reading
> (message)Mailing Lists bit more carefully, adjusting the Posting Styles
> and sending a bug fix to the Emacs' bug tracker, I believe it should not
> happen anymore.  Sorry for the annoyance.

Tomas Volf wrote 3 months ago

Recipients:(name . Z572)(address . zhengjunjie@iscas.ac.cn)(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 74532@debbugs.gnu.org)

Message-ID:87o71flehy.fsf@wolfsden.cz

Hello,

Z572 <zhengjunjie@iscas.ac.cn> writes:

Toggle quote (12 lines)>> How it that usually done?  I cannot do that *before* build, because then
>> tests would not run (the library is not yet installed into the absolute
>> path), and I cannot do it after, because I would need to rebuild the .go
>> files after patching the source code.
>
> you can use substitute* to adjust source.
> e.g.
>
> (or (false-if-exception (load-extension "/path/to/lib-some-object-file.so"))
>     (load-extension "lib-some-object-file.so"))
>

Thank you for the suggestion. I am not sure I like the

false-if-exception part, but temporary adjustment of the (@ (system

foreign-library) guile-extensions-path) should do the trick as well.

One thing I am curious about, I can see you have put the load from

absolute path as the first alternative. But if it fails, it still falls

back to lookup based on the guile-extensions-path. I wonder whether

putting it at the end of the list would not be better, that way you

could still replace the library in the spirit of LD_PRELOAD by setting

the appropriate environment variable. And if reproducibility is the

priority, then even the `or' might not make sense.

Any thoughts on that?

Thanks,

Tomas

There are only two hard things in Computer Science:

cache invalidation, naming things and off-by-one errors.

Ludovic Courtès wrote 3 months ago

Recipients:(name . Tomas Volf)(address . ~@wolfsden.cz)(name . Z572)(address . zhengjunjie@iscas.ac.cn)(address . 74532@debbugs.gnu.org)

Message-ID:87bjxapg79.fsf@gnu.org

Hi,

Tomas Volf <~@wolfsden.cz> skribis:

Toggle quote (18 lines)> Z572 <zhengjunjie@iscas.ac.cn> writes:
>
>>> How it that usually done?  I cannot do that *before* build, because then
>>> tests would not run (the library is not yet installed into the absolute
>>> path), and I cannot do it after, because I would need to rebuild the .go
>>> files after patching the source code.
>>
>> you can use substitute* to adjust source.
>> e.g.
>>
>> (or (false-if-exception (load-extension "/path/to/lib-some-object-file.so"))
>>     (load-extension "lib-some-object-file.so"))
>>
>
> Thank you for the suggestion.  I am not sure I like the
> false-if-exception part, but temporary adjustment of the (@ (system
> foreign-library) guile-extensions-path) should do the trick as well.

I think you can just do:

(load-extension "/path/to/lib-some-object-file.so")

but you need to (1) do that in a post-install phase, and (2) re-run

‘make’ before ‘make install’ if necessary.

See ‘guile-gdbm-ffi’ as one example.

Ludo’.

Timo Wilken wrote 1 weeks ago

Re: with-extensions does not add native extensions to the load path

Recipients:(address . 74532@debbugs.gnu.org)(name . Tomas Volf)(address . ~@wolfsden.cz)(name . Z572)(address . zhengjunjie@iscas.ac.cn)(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:D8H58BCN9SEV.T8XB1KHBJZSR@twilken.net

Hi,

Since I ran into this problem with guile-yamlpp today, I went ahead and

implemented Z572's suggestion.

It still feels wrong to do this on a per-package basis (shouldn't

`with-extension' take care of setting up the path?), but hey ho.

I needed the fallback (as suggested by Z572), else tests would fail (as

predicted by Tomas).

Like this, the whole chain works for me; in my case:

(computed-file "test"

(with-extensions (list guile-yamlpp)

#~(use-modules (yamlpp))))

Patch incoming.

Cheers,

Timo

Timo Wilken wrote 1 weeks ago

[PATCH] gnu: guile-yamlpp: Allow usage with `with-extensions'

Recipients:(address . 74532@debbugs.gnu.org)

Message-ID:20250315211008.8563-3-guix@twilken.net

* gnu/packages/guile-xyz.scm (guile-yamlpp) [arguments] <phases>:

Add 'set-extension-file-name.

Change-Id: I49a7199b4e0fe4053822374bae59c6e1d0b7ead0

---

gnu/packages/guile-xyz.scm | 13 +++++++++++++

1 file changed, 13 insertions(+)

Toggle diff (26 lines)diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index 30c8e847996..f43e06a3a7f 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -1997,6 +1997,19 @@ (define-public guile-yamlpp
        (sha256
         (base32 "0ik69y0vddg0myp0zdbkmklma0qkkrqzwlqwkij1zirklz6hl1ss"))))
     (build-system gnu-build-system)
+    (arguments
+     `(#:phases
+       ,#~(modify-phases %standard-phases
+            ;; In order to let Guile find the compiled library when used with
+            ;; Guix' `with-extension', hardcode the final path to the library.
+            ;; The fallback is needed for tests (i.e. before the 'install phase).
+            ;; See <https://issues.guix.gnu.org/74532>.
+            (add-after 'unpack 'set-extension-file-name
+              (lambda* (#:key outputs #:allow-other-keys)
+                (substitute* "scm/yamlpp.scm"
+                  (("\\(load-extension \"libguile-yamlpp\"(.*)\\)" all init)
+                   (format #f "(or (false-if-exception (load-extension \"~a/lib/guile/3.0/libguile-yamlpp\"~a)) ~a)"
+                           (assoc-ref outputs "out") init all))))))))
     (native-inputs (list autoconf automake libtool pkg-config))
     (inputs (list guile-3.0 yaml-cpp))
     (native-search-paths
-- 
2.48.1

Tomas Volf wrote 1 weeks ago

Recipients:(name . Timo Wilken)(address . guix@twilken.net)(name . Ludovic Courtès)(address . ludo@gnu.org)(name . Z572)(address . zhengjunjie@iscas.ac.cn)(address . 74532@debbugs.gnu.org)

Message-ID:875xk9vyhq.fsf@wolfsden.cz

Hi!

Timo Wilken <guix@twilken.net> writes:

Toggle quote (30 lines)> * gnu/packages/guile-xyz.scm (guile-yamlpp) [arguments] <phases>:
>   Add 'set-extension-file-name.
>
> Change-Id: I49a7199b4e0fe4053822374bae59c6e1d0b7ead0
> ---
>  gnu/packages/guile-xyz.scm | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
> index 30c8e847996..f43e06a3a7f 100644
> --- a/gnu/packages/guile-xyz.scm
> +++ b/gnu/packages/guile-xyz.scm
> @@ -1997,6 +1997,19 @@ (define-public guile-yamlpp
>         (sha256
>          (base32 "0ik69y0vddg0myp0zdbkmklma0qkkrqzwlqwkij1zirklz6hl1ss"))))
>      (build-system gnu-build-system)
> +    (arguments
> +     `(#:phases
> +       ,#~(modify-phases %standard-phases
> +            ;; In order to let Guile find the compiled library when used with
> +            ;; Guix' `with-extension', hardcode the final path to the library.
> +            ;; The fallback is needed for tests (i.e. before the 'install phase).
> +            ;; See <https://issues.guix.gnu.org/74532>.
> +            (add-after 'unpack 'set-extension-file-name
> +              (lambda* (#:key outputs #:allow-other-keys)
> +                (substitute* "scm/yamlpp.scm"
> +                  (("\\(load-extension \"libguile-yamlpp\"(.*)\\)" all init)
> +                   (format #f "(or (false-if-exception (load-extension \"~a/lib/guile/3.0/libguile-yamlpp\"~a)) ~a)"
> +                           (assoc-ref outputs "out") init all))))))))

I am not sure it is optimal to hide *all* errors from load-extension.

Maybe check if file exists and then try to load-extension it without the

false-if-exception?

(Sure there is a TOCTTOU problem, but I think that can be ignored in

this case.)

Toggle quote (4 lines)

> (native-inputs (list autoconf automake libtool pkg-config))

> (inputs (list guile-3.0 yaml-cpp))

> (native-search-paths

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

Tomas Volf wrote 1 weeks ago

Re: with-extensions does not add native extensions to the load path

Recipients:(name . Timo Wilken)(address . guix@twilken.net)(name . Ludovic Courtès)(address . ludo@gnu.org)(name . Z572)(address . zhengjunjie@iscas.ac.cn)(address . 74532@debbugs.gnu.org)

Message-ID:871puxvygl.fsf@wolfsden.cz

"Timo Wilken" <guix@twilken.net> writes:

Toggle quote (3 lines)

> It still feels wrong to do this on a per-package basis (shouldn't

> `with-extension' take care of setting up the path?), but hey ho.

I definitely agree here...

There are only two hard things in Computer Science:

cache invalidation, naming things and off-by-one errors.

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74532@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 74532

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags

mumi command -t +confirmed -t +easy

Or, remove the moreinfo tag and set the help tag

mumi command -t -moreinfo -t +help

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date