Maybe partly undo the patch on Elisp comp-el-to-eln-filename

Hi, I suggest to maybe amend one of the things done by this patchset: https://issues.guix.gnu.org/67260

It undoes the hashing effect of the Elisp function `comp-el-to-eln-filename`, and that seems likely to cause issues downstream, for example in my Emacs package: https://github.com/meedstrom/org-node/issues/60.

To summarize: that function is supposed to generate a filename with a hash based not only the filename but the contents of the file. While it makes sense in Guix to ignore the contribution of the filename, I believe it should still output a new filename when the contents change.

Otherwise there seems to be no way for a downstream package to ensure that it is using an up-to-date .eln variant of an .el file.

I may have missed something though. Can someone in the know tell me what happens if you have not updated Emacs (which if I understand correctly, means ELN-DIR does not change), but you do update an Elisp package, whether through Guix or through Emacs' own package managers. Will Emacs then possibly load an old .eln?

I do not believe that user options like `load-prefer-newer` would affect it. It would just rely on running the aforementioned function and counting on it to output an .eln filename that does not exist if the source is newer.

Am Montag, dem 07.10.2024 um 16:56 +0200 schrieb Martin Edström:

There are opposite goals to "make sure that the file hasn't been

tampered with" (upstream) and "to keep files patchable" (Guix). I

don't think we can easily satisfy both. Perhaps we could use the

original store path as some kind of key to match the files (since we

compile them in-store IIRC), but that wouldn't work for the "let's

compile our init.el" use case.

As a matter of fact, we've disabled JIT compilation for the very reason

that stuff can break ;)

What about aggressive-recompilation-on-write?

We write store paths to a subdirs.el – unless specifically prompted to

reload that, Emacs will keep using old libraries. This is by design,

so that updating Emacs does not cause any issues with (byte) compiled

files.

Since all timestamps point to 1970, you are right, `load-prefer-newer'

does nothing.

Cheers

Hi, thanks for the prompt response!

On Mon, 07 Oct 2024 20:02:17 +0200, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

That works in the init.el case, because the user would be the one writing the file.

In my package, the use case is instead that it starts sub-processes, each of which should load a file "org-node-parser.eln". I could ahve made them just load the output of (locate-library "org-node-parser"), but for unrelated reasons, that seems it would only ever locate an .elc and not an .eln.

So I need to use `comp-el-to-eln-filename` to find the up-to-date .eln (or force it to be compiled). Regardless of whether or not the rest of the package has already been native-compiled.

The worst-case scenario is using package version 1.4.1 in the main Emacs process, but 1.4.0 of org-node-parser.eln in the subprocesses. That leads to unintended bugs.

I suppose the other way around could also happen - using package version 1.4.0 but 1.4.1 in the subprocesses - but that'll be my own mess to figure out :)

Let me know if I understand this correctly: updating a package compiles it at the same time, so we can have store paths like

/gnu/store/package-1.4.0/...{.elc,.el,.eln}

/gnu/store/package-1.4.1/...{.elc,.el,.eln}

which sounds like it will be all consistent. An .eln in the second directory would never secretly be 1.4.0, for example.

But since you said we disabled the JIT compilation, and store dirs are read-only, where do the .eln actually end up, after the user starts Emacs and it does its async-native-compile thing?

(or does it not do any JIT compilation at all?)

I don't have a Guix machine at the moment, but is it a deterministic path like ~/.emacs.d/eln-cache/29.4-HASH/package-HASH.eln ? The user in my GitHub issue gets no such path from running `comp-el-to-eln-filename`, but maybe they're on an old Guix Emacs.

Am Montag, dem 07.10.2024 um 22:46 +0200 schrieb Martin Edström:

Could you keep track of modifications to org-node-parser and recompile

that on change? Or is it part of your package already – if the latter,

then we should already have it compiled as a package.

I think it's your own mess either way – can't you downgrade to a model

where you just ask for "org-node-parser" to be loaded, regardless of

compilation? Then, you wouldn't have to native-compile it.

We don't do JIT, but even if we did, we should find the one in the

store (and the correct one, as per Emacs load paths).

It's `~/.emacs.d/eln-cache/29.4-HASH/some/module.eln`, with some/module

being the module that you'd load.

Cheers

On Tue, 08 Oct 2024 06:32:32 +0200, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

It comes as part of the package. I don't want to assume that it has been compiled, since it's fairly performance-sensitive. That's why I'll either use a previously existing compiled object or make a new one.

What I'm getting from this is that it might be safer to just compile the object into /tmp and use that, regardless of what may already exist. At the moment, that's necessary for keeping the option open of using an .eln on Guix.

So let's ignore my package, it is just an example of a downstream use of `comp-el-to-eln-filename` that relied on its hashing functionality.

Let's just discuss that function.

I have to point out that the emacs `load-path` does not include any native paths. When I inspect the value on my non-Guix emacs, I see no references to .../eln-cache/..., just references to directories where there are .elc and .el files.

I infer that Emacs starts with finding a library in load-path, then converts the path with `comp-el-to-eln-filename`, and checks if that file exists, then loads it.

And crucially, it is not just about the filepath, the function hashes the file contents as well. That ensures that the output path is always different if the source file changes.

Since Guix has a patch that removes this effect, it seems like a package could be upgraded many, many times, without the .eln path ever changing, and so the user would stay on that very outdated file.

Is that not a regression/bug?

Am Dienstag, dem 08.10.2024 um 12:41 +0200 schrieb Martin Edström:

Could you leave that decision to the user?

There is a separate load path for natively compiled files, called

`native-comp-eln-load-path'.

I think relying on such implementation details is perhaps permitted if

it's inside of Emacs itself, but even then it clashes with our

expectation that Emacs be graftable.

The way our load paths are set up, it is actually the opposite (which

still is a bug, just not the one reported). While `guix upgrade` or a

command to the similar effect will swap out the .eln under the hood,

the `.el` and `.elc` files stay stable – remember what I wrote in the

previous message about that having caused issues with byte compilation?

We also get a similar-looking bug if our packages aren't actually

native-compiled, but Emacs itself vendors them. That is resolved by

dropping those .eln-files from the Emacs package.

Cheers

Am Mittwoch, dem 09.10.2024 um 17:15 +0200 schrieb Martin Edström:

The user can just use the output of the help function, it will tell

them whether a function is natively-compiled, byte-compiled, or just

interpreted. Emacs 30 even gives you `native-comp-function-p'. See

`gnu/packages/aux-files/emacs/comp-integrity[-next].el' for how we

assert that our Emacs loads natively-compiled libraries.

I'm not sure these things work the way you think. IIUC, `load-prefer-

newer' should guard against the case where the .el is newer than the

.el[cn], not the other way round, so it should still load the

compiled/native-compiled variant if they have the same stamp.

I mean, the C code is there to read (along with the patch we've made),

but I can't fault you for not going that deep.

The grafting process doesn't look that deep into file names. On paper,

that could work, since the length of the name is preserved, but it'd

also make the grafting process take longer and not really add all that

much to its reliability.

There is only one `comp-el-to-eln-filename' to call, you can't (without

a much more ugly hack) have two strategies, and even if you did have

them, it'd be even more error-prone.

I haven't linked at the shared objects themselves, but I have strong

hunch that they use dynamic linking between the libraries. Since we

only update the /gnu/store/emacs-whatever-HERE-WE-HAVE-A-HASH portion

and nothing post that, we get file names that don't exist :)

If you dig into the patch series, it references the bug it fixes.

For example, GTK is a package deep in the world, that gets grafted a

lot and causes Emacs to be grafted as well.

I haven't looked in the ABI stability of Emacs Lisp packages, but most

of them would actually not need a graft. It's Emacs itself that needs

the grafting :)

We had issues, where a major upgrade between Emacsen would cause

invalid bytecode to be loaded, yes. The issue in Guix is not so much

that a package is not upgrade, but Emacs being upgraded while Emacs is

running. You'd get a functioning Emacs once you exit the process and

start a new one, but that feels very dirty. (Imagine exiting Emacs,

because your package manager tells you so, ugh!)

Yes, the .el, .elc, and .eln files are built as *one* package. Any

package manager you use with Emacs should provide you with that

invariant.

Cheers

In case anyone reads this, I believe I have the right

algorithm now.

First, the trick to locate an .eln regardless of OS:

(comp-lookup-eln (locate-library "my-library.el"))

Turns out I do not need the trick, because if an OS like Guix

has indeed shipped an .eln and set it up to be loaded,

then `symbol-file` should return that one.

TL;DR: it took a while to figure out, but the following

snippet is how my program chooses an .el, .elc or .eln that

corresponds to currently loaded Lisp definitions, so that we

know the file is safe to be executed by a subprocess.

If the current definitions come from an .el, then it

opportunistically builds an .eln and returns that instead.

(let ((loaded-file

(or (ignore-errors

(symbol-file 'a-func-from-my-library nil t))

(symbol-file 'a-func-from-my-library))))

(if (string-suffix-p ".el" loaded-file)

(let ((out (comp-el-to-eln-filename loaded-file)))

(native-compile loaded-file out)

out)

loaded-file))