with-extensions does not add native extensions to the load path

Open

Details

3 participants

Ludovic Courtès
Z572
Tomas Volf

Owner: unassigned

Submitted by: Tomas Volf

Severity: normal

Tomas Volf wrote on 25 Nov 18:49 +0100

Recipients:(address . bug-guix@gnu.org)

Message-ID:871pyzdwzi.fsf@wolfsden.cz

The documentation for `with-extensions' says:

Toggle quote (4 lines)

> In the same vein, sometimes you want to import not just pure-Scheme

> modules, but also “extensions” such as Guile bindings to C libraries

> or other “full-blown” packages.

However it does not actually add those C libraries into

(guile-extensions-path), which means that trying to actually use a

library imported this way will lead to an error.

I guess `load-path-expression' in (guix gexp) needs to also append to

the `guile-extensions-path' based on the search-paths of the imported

packages?

Reproduction is simple. Make sure you *do not* have guile-yamlpp

installed in your profile. Then build the `test' into a store item.

Toggle snippet (6 lines)

(use-modules (gnu packages guile-xyz))

(define test

(with-extensions (list guile-yamlpp)

(program-file "test" #~(use-modules (yamlpp)))))

And try to run it:

Toggle snippet (39 lines)$ /gnu/store/nq0sahcnph5sxms4irq4inv2hq2hfbzr-test
Backtrace:
          19 (primitive-load "/gnu/store/nq0sahcnph5sxms4irq4inv2hq2hfbzr-test")
In ice-9/eval.scm:
   721:20 18 (primitive-eval _)
In ice-9/psyntax.scm:
  1229:36 17 (expand-top-sequence _ _ _ #f _ _ _)
  1221:19 16 (parse _ (("placeholder" placeholder)) ((top) #(ribcage () () ())) _ e (eval) #)
   259:10 15 (parse _ (("placeholder" placeholder)) (()) _ c&e (eval) (hygiene guile-user))
In ice-9/boot-9.scm:
  3935:20 14 (process-use-modules _)
   222:17 13 (map1 (((yamlpp))))
  3936:31 12 (_ ((yamlpp)))
  3327:17 11 (resolve-interface (yamlpp) #:select _ #:hide _ #:prefix _ #:renamer _ # _)
In ice-9/threads.scm:
    390:8 10 (_ _)
In ice-9/boot-9.scm:
  3253:13  9 (_)
In ice-9/threads.scm:
    390:8  8 (_ _)
In ice-9/boot-9.scm:
  3544:20  7 (_)
   2836:4  6 (save-module-excursion _)
  3564:26  5 (_)
In unknown file:
           4 (primitive-load-path "yamlpp" #<procedure 7f405e4fd560 at ice-9/boot-9.scm:35…>)
In yamlpp.scm:
     65:1  3 (_)
In unknown file:
           2 (load-extension "libguile-yamlpp" "init")
In system/foreign-library.scm:
   190:25  1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)
In unknown file:
           0 (dlopen "libguile-yamlpp.so" 1)

ERROR: In procedure dlopen:
In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"

Have a nice day,
Tomas

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

Ludovic Courtès wrote on 28 Nov 08:54 +0100

Recipients:(address . 74532@debbugs.gnu.org)

Message-ID:877c8n94kl.fsf@gnu.org

Hi,

Tomas Volf <~@wolfsden.cz> skribis:

Toggle quote (6 lines)

> The documentation for `with-extensions' says:

>> In the same vein, sometimes you want to import not just pure-Scheme

>> modules, but also “extensions” such as Guile bindings to C libraries

>> or other “full-blown” packages.

It’s not just bindings but also pure Scheme libraries like Guile-JSON.

Toggle quote (4 lines)

> However it does not actually add those C libraries into

> (guile-extensions-path), which means that trying to actually use a

> library imported this way will lead to an error.

[...]

Toggle quote (7 lines)

> 190:25 1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)

> In unknown file:

> 0 (dlopen "libguile-yamlpp.so" 1)

> ERROR: In procedure dlopen:

> In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"

Usually, packages like these (guile-gnutls, guile-git, guile-ssh, etc.)

have their .so absolute file name hard-coded, which sidesteps this

problem entirely. I recommend doing that for guile-yamlpp as well.

That said, it would probably make sense to arrange for ‘with-extensions’

to set GUILE_EXTENSIONS_PATH.

Ludo’.

PS: Your MUA sets “Mail-Followup-To: bug-guix@gnu.org”, which is kinda

annoying because that’s the wrong address when replying to a bug.

:-)

Tomas Volf wrote on 3 Dec 00:40 +0100

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 74532@debbugs.gnu.org)

Message-ID:87plm93b7p.fsf@wolfsden.cz

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (12 lines)> Hi,
>
> Tomas Volf <~@wolfsden.cz> skribis:
>
>> The documentation for `with-extensions' says:
>>
>>> In the same vein, sometimes you want to import not just pure-Scheme
>>> modules, but also “extensions” such as Guile bindings to C libraries
>>> or other “full-blown” packages.
>
> It’s not just bindings but also pure Scheme libraries like Guile-JSON.

True, and for those it works well. However the text documentation

explicitly mentions "bindings to C libraries" as one of the use

cases. :)

Toggle quote (18 lines)>
>> However it does not actually add those C libraries into
>> (guile-extensions-path), which means that trying to actually use a
>> library imported this way will lead to an error.
>
> [...]
>
>>    190:25  1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)
>> In unknown file:
>>            0 (dlopen "libguile-yamlpp.so" 1)
>>
>> ERROR: In procedure dlopen:
>> In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"
>
> Usually, packages like these (guile-gnutls, guile-git, guile-ssh, etc.)
> have their .so absolute file name hard-coded, which sidesteps this
> problem entirely.  I recommend doing that for guile-yamlpp as well.

How it that usually done? I cannot do that *before* build, because then

tests would not run (the library is not yet installed into the absolute

path), and I cannot do it after, because I would need to rebuild the .go

files after patching the source code.

Assuming I have a guile library that creates a new .so library during

the build, and the .so library must be loadable by the other modules in

the library during the build (and for the tests), how should I approach

that?

I am pretty sure I cannot just patch the source code, since the library

would not be installed into the absolute path yet during the build.

Honestly, setting GUILE_EXTENSIONS_PATH via pre-inst-env seemed like

fairly elegant solution.

Toggle quote (4 lines)

> That said, it would probably make sense to arrange for ‘with-extensions’

> to set GUILE_EXTENSIONS_PATH.

That would be great.  My current work-around is

  (with-extensions (list guile-wolfsden)
    (program-file
     "audio-cycle-sinks"
     #~(begin
         ;; Bug 74532: Native extensions are not added to the load path.
         (eval-when (expand load eval)
           (let ((ext-path (@ (system foreign-library) guile-extensions-path)))
             ;; Just a temporary hack, we can live with duplicates in the path.
             (ext-path (cons #$(file-append guile-wolfsden
                                            "/lib/guile/3.0/extensions")
                             (ext-path)))))
         ...)))

which is anything but elegant.

Toggle quote (7 lines)

> Ludo’.

> PS: Your MUA sets “Mail-Followup-To: bug-guix@gnu.org”, which is kinda

> annoying because that’s the wrong address when replying to a bug.

> :-)

Thanks for letting me know, I was not aware of that.  After reading
(message)Mailing Lists bit more carefully, adjusting the Posting Styles
and sending a bug fix to the Emacs' bug tracker, I believe it should not
happen anymore.  Sorry for the annoyance.

-- 
There are only two hard things in Computer Science:
cache invalidation, naming things and off-by-one errors.

Z572 wrote on 3 Dec 04:10 +0100

Recipients:(name . Tomas Volf)(address . ~@wolfsden.cz)

Message-ID:87y10x4g2w.fsf@iscas.ac.cn

Tomas Volf <~@wolfsden.cz> writes:

Toggle quote (41 lines)> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi,
>>
>> Tomas Volf <~@wolfsden.cz> skribis:
>>
>>> The documentation for `with-extensions' says:
>>>
>>>> In the same vein, sometimes you want to import not just pure-Scheme
>>>> modules, but also “extensions” such as Guile bindings to C libraries
>>>> or other “full-blown” packages.
>>
>> It’s not just bindings but also pure Scheme libraries like Guile-JSON.
>
> True, and for those it works well.  However the text documentation
> explicitly mentions "bindings to C libraries" as one of the use
> cases. :)
>
>>
>>> However it does not actually add those C libraries into
>>> (guile-extensions-path), which means that trying to actually use a
>>> library imported this way will lead to an error.
>>
>> [...]
>>
>>>    190:25  1 (load-foreign-library _ #:extensions _ #:search-ltdl-library-path? _ # _ # _ # …)
>>> In unknown file:
>>>            0 (dlopen "libguile-yamlpp.so" 1)
>>>
>>> ERROR: In procedure dlopen:
>>> In procedure dlopen: file "libguile-yamlpp.so", message "libguile-yamlpp.so: cannot open shared object file: No such file or directory"
>>
>> Usually, packages like these (guile-gnutls, guile-git, guile-ssh, etc.)
>> have their .so absolute file name hard-coded, which sidesteps this
>> problem entirely.  I recommend doing that for guile-yamlpp as well.
>
> How it that usually done?  I cannot do that *before* build, because then
> tests would not run (the library is not yet installed into the absolute
> path), and I cannot do it after, because I would need to rebuild the .go
> files after patching the source code.

you can use substitute* to adjust source.

e.g.

(or (false-if-exception (load-extension "/path/to/lib-some-object-file.so"))

(load-extension "lib-some-object-file.so"))

Toggle quote (44 lines)>
> Assuming I have a guile library that creates a new .so library during
> the build, and the .so library must be loadable by the other modules in
> the library during the build (and for the tests), how should I approach
> that?
>
> I am pretty sure I cannot just patch the source code, since the library
> would not be installed into the absolute path yet during the build.
>
> Honestly, setting GUILE_EXTENSIONS_PATH via pre-inst-env seemed like
> fairly elegant solution.
>
>>
>> That said, it would probably make sense to arrange for ‘with-extensions’
>> to set GUILE_EXTENSIONS_PATH.
>
> That would be great.  My current work-around is
>
>   (with-extensions (list guile-wolfsden)
>     (program-file
>      "audio-cycle-sinks"
>      #~(begin
>          ;; Bug 74532: Native extensions are not added to the load path.
>          (eval-when (expand load eval)
>            (let ((ext-path (@ (system foreign-library) guile-extensions-path)))
>              ;; Just a temporary hack, we can live with duplicates in the path.
>              (ext-path (cons #$(file-append guile-wolfsden
>                                             "/lib/guile/3.0/extensions")
>                              (ext-path)))))
>          ...)))
>
> which is anything but elegant.
>
>>
>> Ludo’.
>>
>> PS: Your MUA sets “Mail-Followup-To: bug-guix@gnu.org”, which is kinda
>>     annoying because that’s the wrong address when replying to a bug.
>>     :-)
>
> Thanks for letting me know, I was not aware of that.  After reading
> (message)Mailing Lists bit more carefully, adjusting the Posting Styles
> and sending a bug fix to the Emacs' bug tracker, I believe it should not
> happen anymore.  Sorry for the annoyance.

Tomas Volf wrote on 13 Dec 17:44 +0100

Recipients:(name . Z572)(address . zhengjunjie@iscas.ac.cn)

Message-ID:87o71flehy.fsf@wolfsden.cz

Hello,

Z572 <zhengjunjie@iscas.ac.cn> writes:

Toggle quote (12 lines)>> How it that usually done?  I cannot do that *before* build, because then
>> tests would not run (the library is not yet installed into the absolute
>> path), and I cannot do it after, because I would need to rebuild the .go
>> files after patching the source code.
>
> you can use substitute* to adjust source.
> e.g.
>
> (or (false-if-exception (load-extension "/path/to/lib-some-object-file.so"))
>     (load-extension "lib-some-object-file.so"))
>

Thank you for the suggestion. I am not sure I like the

false-if-exception part, but temporary adjustment of the (@ (system

foreign-library) guile-extensions-path) should do the trick as well.

One thing I am curious about, I can see you have put the load from

absolute path as the first alternative. But if it fails, it still falls

back to lookup based on the guile-extensions-path. I wonder whether

putting it at the end of the list would not be better, that way you

could still replace the library in the spirit of LD_PRELOAD by setting

the appropriate environment variable. And if reproducibility is the

priority, then even the `or' might not make sense.

Any thoughts on that?

Thanks,

Tomas

There are only two hard things in Computer Science:

cache invalidation, naming things and off-by-one errors.

Ludovic Courtès wrote on 17 Dec 14:55 +0100

Recipients:(name . Tomas Volf)(address . ~@wolfsden.cz)

Message-ID:87bjxapg79.fsf@gnu.org

Hi,

Tomas Volf <~@wolfsden.cz> skribis:

Toggle quote (18 lines)> Z572 <zhengjunjie@iscas.ac.cn> writes:
>
>>> How it that usually done?  I cannot do that *before* build, because then
>>> tests would not run (the library is not yet installed into the absolute
>>> path), and I cannot do it after, because I would need to rebuild the .go
>>> files after patching the source code.
>>
>> you can use substitute* to adjust source.
>> e.g.
>>
>> (or (false-if-exception (load-extension "/path/to/lib-some-object-file.so"))
>>     (load-extension "lib-some-object-file.so"))
>>
>
> Thank you for the suggestion.  I am not sure I like the
> false-if-exception part, but temporary adjustment of the (@ (system
> foreign-library) guile-extensions-path) should do the trick as well.

I think you can just do:

(load-extension "/path/to/lib-some-object-file.so")

but you need to (1) do that in a post-install phase, and (2) re-run

‘make’ before ‘make install’ if necessary.

See ‘guile-gdbm-ffi’ as one example.

Ludo’.

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74532@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 74532

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date