I currently use the flatwhatson channel (cc'ed on this email, hope that is okay Andrew!) and it has worked well for Emacs, though never tried anything else with libgccjit. It would be great to get this upstreamed in Guix, especially since libgccjit is a beast to build (more than emacs if I remember correctly).
Happy to help test and work on this, let me know anything I can do.
On Tuesday, June 28th, 2022 at 12:17 AM, Liliana Marie Prikler wrote:
Toggle quote (8 lines)
> Keyword here is "has worked for emacs". I've tried porting the logic
> from flatwhatson's channel over, but regardless of what I do, it
> already fails in the configure step of Emacs (in a manner that's
> reproducible outside as well). Thus, I think this is a bug in
> libgccjit (or perhaps our packaging of it) that simply happened to be
> ignored during development of Emacs 28, but no longer in the release.
Sorry, I should be extra clear that I mean has in the past and continues to work for Emacs. I've been using emacs-pgtk-native-comp through the flatwhatson channel from well before v28 was released. Currently I'm using emacs-pgtk-native-comp-28.1.50-223.3ddccb5. Everything has built, installed, and run fine for as long as I have been using it. Just in case that was in question, and as a point of reference.
Anyway, I'll try to reproduce when I can (tomorrow likely) what you reported in the first message using this setup, if that is of use.
Appreciate the efforts from everyone working on this!
Found out some useful info and a work around for the original reported issue of the simple "hello world" of gccjit not working.
------- Original Message -------
On Tuesday, June 28th, 2022 at 1:16 AM, John Kehayias wrote:
Toggle quote (18 lines)
> ------- Original Message -------
> On Tuesday, June 28th, 2022 at 12:17 AM, Liliana Marie Prikler wrote:
> > Keyword here is "has worked for emacs". I've tried porting the logic
> > from flatwhatson's channel over, but regardless of what I do, it
> > already fails in the configure step of Emacs (in a manner that's
> > reproducible outside as well). Thus, I think this is a bug in
> > libgccjit (or perhaps our packaging of it) that simply happened to be
> > ignored during development of Emacs 28, but no longer in the release.
> Sorry, I should be extra clear that I mean has in the past and continues to work for Emacs. I've been using emacs-pgtk-native-comp through the flatwhatson channel from well before v28 was released. Currently I'm using emacs-pgtk-native-comp-28.1.50-223.3ddccb5. Everything has built, installed, and run fine for as long as I have been using it. Just in case that was in question, and as a point of reference.
> Anyway, I'll try to reproduce when I can (tomorrow likely) what you reported in the first message using this setup, if that is of use.
I was able to reproduce the original error, though I used the libgccjit package from the flatwhatson channel, at v11.3.0 (along with GCC at that version). For good measure, I also used the tutorial at that version, just in case https://gcc.gnu.org/onlinedocs/gcc-11.3.0/jit/intro/tutorial01.html I chose this version since that is what emacs-native-comp from that channel is built with.
Searching for these error messages of missing libraries/files, I found
I didn't dive into the details and I'm not expert here, but it gave me the clues to work around it. Seems that where gccjit looks for things has some assumptions (bugs?) which we can fix at runtime with:
Looks like the code you sent got line wrapped and I couldn't easily untangle it. The difference that pops out to me is how you are locating the lib/gcc directory, is it maybe picking up from the build system rather than libgccjit? (though I think gcc:lib should be the same here, but not gcc-toolchain)
In any event, I just want to reiterate that the libgccjit and emacs-native-comp from Andrew's channel does indeed work: it compiles, runs, and does native compilations successfully. So hopefully we've narrowed down now that there is some difference in the code you tried and from Andrew's that is leading to different behavior.
> In any event, I just want to reiterate that the libgccjit and emacs-
> native-comp from Andrew's channel does indeed work: it compiles,
> runs, and does native compilations successfully. So hopefully we've
> narrowed down now that there is some difference in the code you tried
> and from Andrew's that is leading to different behavior.
The style changes are not the only thing at play here, though. In
particular, I am trying to build Emacs 28.1, whereas Andrew builds
other versions, though notably the newest one ought to include the
smoke test. You could also try other combinations for libgccjit and
gcc to see if those make a difference – last time I tried it did not.
Attached the definitions as a file this time.
(define-public(make-libgccjitgcc)(package(inheritgcc)(name"libgccjit")(outputs(delete"lib"(package-outputsgcc)))(properties(alist-delete'hidden?(package-propertiesgcc)))(arguments(substitute-keyword-arguments(package-argumentsgcc)((#:modules_'())'((guixbuildgnu-build-system)(guixbuildutils)(ice-9regex)(srfisrfi-1)(srfisrfi-26)))((#:configure-flagsflags)#~(cons*"--disable-bootstrap""--disable-libatomic""--disable-libgomp""--disable-libquadmath""--disable-libssp""--enable-host-shared""--enable-languages=jit"(remove(cutstring-match"--enable-languages.*"<>)#$flags)))((#:phasesphases)#~(modify-phases#$phases(add-after'install'remove-broken-or-conflicting-files(lambda*(#:keyoutputs#:allow-other-keys)(for-eachdelete-file(find-files(string-append(assoc-refoutputs"out")"/bin")".*(c\\+\\+|cpp|g\\+\\+|gcov|gcc|gcc-.*)"))))))))(inputs(modify-inputs(package-inputsgcc)(delete"libstdc++")))(native-inputs(modify-inputs(package-native-inputsgcc)(prependgcc)))(synopsis"GCC library generating machine code on-the-fly at runtime")(description"This package is part of the GNU Compiler Collection and provides an
embeddable library for generating machine code on-the-fly at runtime. This
shared library can then be dynamically-linked into bytecode interpreters and
other such programs that want to generate machine code on-the-fly at run-time.
It can also be used for ahead-of-time code generation for building standalone
compilers. The just-in-time (jit) part of the name is now something of a
(define-publicemacs(package(name"emacs")(version"28.1.90")(source(origin(methodurl-fetch)(uri(string-append"mirror://gnu/emacs/emacs-"version".tar.xz"))(sha256(base32"1qbmmmhnjhn4lvzsnyk7l5ganbi6wzbm38jc1a7hhyh3k78b7c98"))(patches(search-patches"emacs-exec-path.patch""emacs-fix-scheme-indent-function.patch""emacs-source-date-epoch.patch"))(modules'((guixbuildutils)))(snippet'(with-directory-excursion"lisp";; Delete the bundled byte-compiled elisp files and generated
(for-eachdelete-file(append(find-files".""\\.elc$")(find-files".""loaddefs\\.el$")(find-files"eshell""^esh-groups\\.el$")));; Make sure Tramp looks for binaries in the right places on
;; remote Guix System machines, where 'getconf PATH' returns
;; something bogus.
(substitute*"net/tramp.el";; Patch the line after "(defcustom tramp-remote-path".
(("\\(tramp-default-remote-path")(format#f"(tramp-default-remote-path ~s ~s ~s ~s ""~/.guix-profile/bin""~/.guix-profile/sbin""/run/current-system/profile/bin""/run/current-system/profile/sbin")));; Make sure Man looks for C header files in the right
(substitute*"man.el"(("\"/usr/local/include\""line)(string-join(listline"\"~/.guix-profile/include\"""\"/var/guix/profiles/system/profile/include\"")" ")))))))(build-systemglib-or-gtk-build-system)(arguments(list#:tests?#f; no check target
(first-subdirectory/absolute;; host type
(search-input-directoryinputs"lib/gcc")))))(setenv"LIBRARY_PATH"(string-appendlibgccjit-libdir":"(getenv"LIBRARY_PATH"))))))(add-after'unpack'enable-elogind(lambda_(substitute*"configure.ac"(("libsystemd")"libelogind"))(when(file-exists?"configure")(delete-file"configure"))))(add-after'unpack'patch-program-file-names(lambda*(#:keyinputs#:allow-other-keys)(substitute*'("src/callproc.c""lisp/term.el""lisp/htmlfontify.el""lisp/textmodes/artist.el""lisp/progmodes/sh-script.el")(("\"/bin/sh\"")(format#f"~s"(search-input-fileinputs"/bin/sh"))))(substitute*"lisp/doc-view.el"(("\"(gs|dvipdf|ps2pdf|pdftotext)\""allwhat)(let((replacement(false-if-exception(search-input-fileinputs(string-append"/bin/"what)))))(ifreplacement(string-append"\""replacement"\"")all))));; match ".gvfs-fuse-daemon-real" and ".gvfsd-fuse-real"
;; respectively when looking for GVFS processes.
(substitute*"lisp/net/tramp-gvfs.el"(("\\(tramp-compat-process-running-p \"(.*)\"\\)"allprocess)(format#f"(or ~a (tramp-compat-process-running-p ~s))"all(string-append"."process"-real"))))))(add-after'unpack'patch-compilation-driver(lambda_(substitute*"lisp/emacs-lisp/comp.el"(("\\(defcustom native-comp-driver-options nil")(format#f"(defcustom native-comp-driver-options '(~s ~s ~s ~s)"(string-append"-B"#$(this-package-input"binutils")"/bin/")(string-append"-B"#$(this-package-input"glibc")"/lib/")(string-append"-B"#$(this-package-input"libgccjit")"/lib/")(string-append"-B"#$(this-package-input"libgccjit")"/lib/gcc/"))))))(add-before'configure'fix-/bin/pwd(lambda_;; Use `pwd', not `/bin/pwd'.
(substitute*(find-files".""^Makefile\\.in$")(("/bin/pwd")"pwd"))))(add-after'install'install-site-start;; Use 'guix-emacs' in "site-start.el", which is used autoload the
;; Elisp packages found in EMACSLOADPATH.
(lambda*(#:keyinputsoutputs#:allow-other-keys)(let*((out(assoc-refoutputs"out"))(lisp-dir(string-appendout"/share/emacs/site-lisp"))(emacs(string-appendout"/bin/emacs")));; This is duplicated from emacs-utils to prevent coupling.
(define*(emacs-byte-compile-directorydir)(let((expr`(progn(setqbyte-compile-debugt)(byte-recompile-directory(file-name-as-directory,dir)01))))(invokeemacs"--quick""--batch"(format#f"--eval=~s"expr))))(copy-file#$(local-file(search-auxiliary-file"emacs/guix-emacs.el"))(string-appendlisp-dir"/guix-emacs.el"))(with-output-to-file(string-appendlisp-dir"/site-start.el")(lambda()(display(string-append"(when (require 'guix-emacs nil t)\n"" (guix-emacs-autoload-packages)\n"" (advice-add 'package-load-all-descriptors"" :after #'guix-emacs-load-package-descriptors))"))));; Remove the extraneous subdirs.el file, as it causes Emacs to
;; add recursively all the the sub-directories of a profile's
;; share/emacs/site-lisp union when added to EMACSLOADPATH,
;; which leads to conflicts.
(delete-file(string-appendlisp-dir"/subdirs.el"));; Byte compile the site-start files.
(emacs-byte-compile-directorylisp-dir))))(add-after'glib-or-gtk-wrap'restore-emacs-pdmp;; restore the dump file that Emacs installs somewhere in
;; libexec/ to its original state
(lambda*(#:keyoutputstarget#:allow-other-keys)(let*((libexec(string-append(assoc-refoutputs"out")"/libexec"));; each of these ought to only match a single file,
;; but even if not (find-files) sorts by string<,
;; so the Nth element in one maps to the Nth element of
;; the other
(pdmp(find-fileslibexec"\\.pdmp$"))(pdmp-real(find-fileslibexec"\\.pdmp-real$")))(for-eachrename-filepdmp-realpdmp))))(add-after'glib-or-gtk-wrap'strip-double-wrap(lambda*(#:keyoutputs#:allow-other-keys);; Directly copy emacs-X.Y to emacs, so that it is not wrapped
;; twice. This also fixes a minor issue, where WMs would not be
;; able to track emacs back to emacs.desktop.
(with-directory-excursion(assoc-refoutputs"out")(copy-file(car(find-files"bin""^emacs-([0-9]+\\.)+[0-9]+$"))"bin/emacs"))))(add-after'strip-double-wrap'wrap-emacs-paths(lambda*(#:keyinputsoutputs#:allow-other-keys)(let*((out(assoc-refoutputs"out"))(lisp-dirs(find-files(string-appendout"/share/emacs")"^lisp$"#:directories?#t)))(for-each(lambda(prog)(wrap-programprog;; emacs-next and variants rely on uname being in PATH for
;; Tramp. Tramp paths can't be hardcoded, because they
;; need to be portable.
`("PATH"suffix,(mapdirname(list(search-input-fileinputs"/bin/gzip");; for coreutils
(search-input-fileinputs"/bin/yes"))))`("EMACSLOADPATH"suffix,lisp-dirs)))(find-files(string-appendout"/bin");; Matches versioned and unversioned emacs binaries.
;; We don't patch emacsclient, because it takes its
;; environment variables from emacs.
;; Likewise, we don't need to patch helper binaries
;; like etags, ctags or ebrowse.
"^emacs(-[0-9]+(\\.[0-9]+)*)?$"))))))))(inputs(listgnutlsncurses;; For native compilation
binutilsglibclibgccjit;; Required for "core" functionality, such as dired and compression.
coreutilsgzip;; Avoid Emacs's limited movemail substitute that retrieves POP3
;; email only via insecure channels.
;; This is not needed for (modern) IMAP.
mailutilsgpmlibx11gtk+cairopangoharfbuzzlibxftlibtiffgifliblcmslibjpeg-turbolibselinuxacljanssongmpghostscriptpopplerelogind;; When looking for libpng `configure' links with `-lpng -lz', so we
;; must also provide zlib as an input.
libpngzlib(if(target-x86-64?)librsvg-bootstraplibrsvg-2.40)libxpmlibxml2libicelibsmalsa-libdbus;; multilingualization support
libotfm17n-lib))(native-inputs(listautoconfpkg-configtexinfo))(native-search-paths(list(search-path-specification(variable"EMACSLOADPATH")(files'("share/emacs/site-lisp")))(search-path-specification(variable"INFOPATH")(files'("share/info")))))(home-page"https://www.gnu.org/software/emacs/")(synopsis"The extensible, customizable, self-documenting text editor")(description"GNU Emacs is an extensible and highly customizable text editor. It is
based on an Emacs Lisp interpreter with extensions for text editing. Emacs
has been extended in essentially all areas of computing, giving rise to a
vast array of packages supporting, e.g., email, IRC and XMPP messaging,
spreadsheets, remote server editing, and much more. Emacs includes extensive
documentation on all aspects of the system, from basic editing to writing
large Lisp programs. It has full Unicode support for nearly all human
On Friday, August 5th, 2022 at 5:31 PM, Liliana Marie Prikler wrote:
Toggle quote (7 lines)
> The style changes are not the only thing at play here, though. In
> particular, I am trying to build Emacs 28.1, whereas Andrew builds
> other versions, though notably the newest one ought to include the
> smoke test. You could also try other combinations for libgccjit and
> gcc to see if those make a difference – last time I tried it did not.
I'm seeing the same version, 28.1.90, in yours and from Andrew's channel. And just checking that indeed it is 28.1.90 that I've been running locally from that channel, with firstname.lastname@example.org. I have not tried with v29.
Toggle quote (3 lines)
> Attached the definitions as a file this time.
Thanks, will play around with it this weekend to see if I have anything useful to add.
On Saturday, August 6th, 2022 at 1:53 AM, Liliana Marie Prikler <email@example.com> wrote:
Toggle quote (3 lines)
> Pardon that, it's an artifact from trying to fetch 28.1.90 via url-
> fetch and failing. Note that the hash is the one for 28.1.
On Andrew's channel I can change the version/commit/hash to 28.1 and it builds fine. Though this is using a git checkout at the 28.1 tag commit, but I don't think that matters here? (Emacs runs too, but only tried in guix shell and just to see it opens, didn't see if it compiles anything.)
And yet with your code I cannot. I tried using the phase from Andrew instead of your code, tried adding gcc to the native-inputs to match Andrew, used gcc-11 with libgccjit-11, ...still the same configure failure on the libgccjit test program: can't find libgccjit.so
Adding just the libgccjit library path to LD_LIBRARY_PATH makes configure find libgccjit, but then fails to find libx11 in that same libgccjit test program. Instead, making LD_LIBRARY_PATH=LIBRARY_PATH fixes the configure and it finally fails on the runpath validation. Maybe cause of messing with LD_LIBRARY_PATH or could use the patch Andrew has? Anyway, wasn't looking into this.
I thought I had a similar problem with the test program you had started with, but in the end just the LIBRARY_PATH tweak was needed. I wish I could remember the combination of things I tried that also had it failing to find libgccjit. Or perhaps it was something libgccjit was linked to, but I hadn't explored?
So. I tried to see what was different between your and Andrew's code and I'm not seeing what it could be. There must be some subtle difference we're missing? Seems we have all the ingredients and a known working example.
On Sunday, August 7th, 2022 at 9:59 AM, Liliana Marie Prikler wrote:
Toggle quote (6 lines)
> It's a really stupid one. Basically, the tests and really any
> executable you try to build fails to execute without LD_LIBRARY_PATH
> set. Now obviously, that's an issue with ld and you know which package
> has ld? That's right, it's binutils!
Ah! That LD_LIBRARY_PATH was needed was the clue.
Toggle quote (8 lines)
> In Andrew's recipe, he sneakily snarfed out binutils from inputs using
> assoc-ref, but I'm using the new package style with (this-package-
> input) to achieve the same thing, so I had to add binutils. But this
> now shadows ld-wrapper. So we have to add ld-wrapper as well. Now
> this makes me question whether the driver options are actually sane or
> whether we'd have to prepend ld-wrapper to those as well.
Is the assoc-ref for binutils (implicit input?) equivalent to just using #$binutils directly? e.g. (string-append "-B" #$binutils "/bin/") with no added binutils to the inputs. Is that not a good idea? As a test, that does indeed work, everything configures, builds, and runs (only tested opening with no configuration).
As for the original issue here, I guess the LIBRARY_PATH and ld shadowing is the workaround. I don't know if that is something that can/should be incorporated into the libgccjit package definition or should just be handled by any package using it. Currently, that will just be emacs anyway and hopefully the discussion here is useful to anyone trying to use libgccjit.
> As for the original issue here, I guess the LIBRARY_PATH and ld shadowing is the workaround. I don't know if that is something that can/should be incorporated into the libgccjit package definition or should just be handled by any package using it. Currently, that will just be emacs anyway and hopefully the discussion here is useful to anyone trying to use libgccjit.
This might be possible by adding LIBRARY_PATH to native-search-paths
in the libgccjit package definition?
Toggle quote (2 lines)
> Thanks again Lily and Andrew for your work here!
Thanks John & Lily for persisting with getting this submitted, I'm