[PATCH] gnu: emacs-org: Make build reproducible

Done

Details

4 participants

Morgan.J.Smith
Ludovic Courtès
Nicolas Goaziou
Michael Rohleder

Owner: unassigned

Submitted by: Morgan.J.Smith

Severity: normal

Debbugs page

Morgan.J.Smith wrote 4 years ago

Recipients:(address . guix-patches@gnu.org)(name . Morgan Smith)(address . Morgan.J.Smith@outlook.com)

Message-ID:BYAPR05MB40239B904A0A20F4BFB8B2A6C55B9@BYAPR05MB4023.namprd05.prod.outlook.com

* gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use

autoloads instead

---

gnu/packages/emacs-xyz.scm | 6 ++++++

1 file changed, 6 insertions(+)

Toggle diff (19 lines)diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index c950f116f9..a73b7959ef 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -11243,6 +11243,12 @@ (define-public emacs-org
                (("org-release \"\"")
                 (string-append "org-release \"" ,version "\"")))
              #t))
+         (add-after 'fix-org-version 'use-autoloads
+           (lambda _
+             (delete-file "org-loaddefs.el")
+             (substitute* (find-files "." ".*.el")
+                 (("org-loaddefs.el") "org-autoloads.el"))
+             #t))
          (add-after 'install 'install-documentation
            (lambda* (#:key outputs #:allow-other-keys)
              (let* ((share (string-append (assoc-ref outputs "out") "/share"))
-- 
2.31.1

Ludovic Courtès wrote 4 years ago

Recipients:(address . Morgan.J.Smith@outlook.com)(address . 48205@debbugs.gnu.org)

Message-ID:87h7j936ox.fsf@gnu.org

Hi,

Morgan.J.Smith@outlook.com skribis:

Toggle quote (5 lines)

> From: Morgan Smith <Morgan.J.Smith@outlook.com>

> * gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use

> autoloads instead

[...]

Toggle quote (7 lines)

> + (add-after 'fix-org-version 'use-autoloads

> + (lambda _

> + (delete-file "org-loaddefs.el")

> + (substitute* (find-files "." ".*.el")

> + (("org-loaddefs.el") "org-autoloads.el"))

> + #t))

You can drop the trailing #t.

Could you explain in a comment in the phase what it does? It’s not

clear to me how this relates to making the build bit-reproducible.

Remember that ‘find-files’ takes a regexp as its second argument; thus,

to match .el files, you need to write "\\.el$".

Could you send an updated patch?

Thanks,

Ludo’.

Morgan.J.Smith wrote 4 years ago

[PATCH v2] gnu: emacs-org: Make build reproducible

Recipients:(address . ludo@gnu.org)

Message-ID:BYAPR05MB4023DFEC30DC46812C6AE7D1C5369@BYAPR05MB4023.namprd05.prod.outlook.com

* gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use

autoloads instead. Copy autoloads into build directory.

---

Thanks for the suggestions! Sorry about the delay. Also sorry about the

comment. It's probably a little too long :/. I need to work on my english and

my ability to be concise.

gnu/packages/emacs-xyz.scm | 21 +++++++++++++++++++++

1 file changed, 21 insertions(+)

Toggle diff (34 lines)diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 1d954ec5bd..180b2709fe 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -11761,6 +11761,27 @@ (define-public emacs-org
     (arguments
      `(#:phases
        (modify-phases %standard-phases
+         ;; Org ships with the file org-loaddefs.el that functions like our
+         ;; autoloads. Something about the interaction between the loaddefs
+         ;; and autoloads makes the build non-deterministic. We should opt for
+         ;; our autoloads system over the loaddefs since the loaddefs are
+         ;; autogenerated for the release tarball. org-loaddefs.el is
+         ;; mentioned by name in the source files so we have to do a blanket
+         ;; regex search and replace.
+         (add-after 'unpack 'use-autoloads
+           (lambda _
+             (delete-file "org-loaddefs.el")
+             (substitute* (find-files "." "\\.el$")
+               (("org-loaddefs.el") "org-autoloads.el"))))
+         ;; We create the autoloads directly in the install folder. During
+         ;; compilation org checks for the autoloads file so we need to copy
+         ;; it to the build directory.
+         (add-after 'make-autoloads 'copy-autoload-to-build-dir
+           (lambda* (#:key outputs #:allow-other-keys)
+             (copy-file (string-append
+                         (elpa-directory (assoc-ref outputs "out"))
+                         "/org-autoloads.el")
+                        "org-autoloads.el")))
          (add-after 'install 'install-documentation
            (lambda* (#:key outputs #:allow-other-keys)
              (let* ((share (string-append (assoc-ref outputs "out") "/share"))
-- 
2.32.0

Ludovic Courtès wrote 4 years ago

Recipients:(address . Morgan.J.Smith@outlook.com)(address . 48205@debbugs.gnu.org)

Message-ID:87a6nw3tcj.fsf@gnu.org

Hi,

Morgan.J.Smith@outlook.com skribis:

Toggle quote (5 lines)

> From: Morgan Smith <Morgan.J.Smith@outlook.com>

> * gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use

> autoloads instead. Copy autoloads into build directory.

[...]

Toggle quote (9 lines)

> (modify-phases %standard-phases

> + ;; Org ships with the file org-loaddefs.el that functions like our

> + ;; autoloads. Something about the interaction between the loaddefs

> + ;; and autoloads makes the build non-deterministic. We should opt for

> + ;; our autoloads system over the loaddefs since the loaddefs are

> + ;; autogenerated for the release tarball. org-loaddefs.el is

> + ;; mentioned by name in the source files so we have to do a blanket

> + ;; regex search and replace.

I wonder about the interaction.

Are you sure using org-autoloads.el everywhere is enough to fix the

problem? With current master, I see:

Toggle snippet (11 lines)$ guix challenge emacs-org --substitute-urls='https://ci.guix.gnu.org https://bayfront.guix.gnu.org'
/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6 contents differ:
  no local build for '/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6'
  https://ci.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 098vc0wpc0rp9qrmh8l75n77j0bd4xjya3q8vd317rxdsfs9dd4n
  https://bayfront.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 0zfzlf28rs7rjvvlq1d3b7cngcvz5grvrkjvsc5j9vp0ikn0zg9p
  differing files:
    /share/emacs/site-lisp/org-9.4.6/org-loaddefs.el
    /share/emacs/site-lisp/org-9.4.6/org-autoloads.elc
    /share/emacs/site-lisp/org-9.4.6/org-autoloads.el

And with diffoscope:

Toggle snippet (57 lines)$ guix challenge emacs-org --substitute-urls='https://ci.guix.gnu.org https://bayfront.guix.gnu.org' --diff=diffoscope
/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6 contents differ:
  no local build for '/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6'
  https://ci.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 098vc0wpc0rp9qrmh8l75n77j0bd4xjya3q8vd317rxdsfs9dd4n
  https://bayfront.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 0zfzlf28rs7rjvvlq1d3b7cngcvz5grvrkjvsc5j9vp0ikn0zg9p
 bayfront.guix.gnu.org  1.7MiB                          7.9MiB/s 00:00 [##################] 100.0%--- /tmp/guix-directory.vqBGkT
+++ /tmp/guix-directory.H4HEMX
│   --- /tmp/guix-directory.vqBGkT/share
├── +++ /tmp/guix-directory.H4HEMX/share
│ │   --- /tmp/guix-directory.vqBGkT/share/emacs
│ ├── +++ /tmp/guix-directory.H4HEMX/share/emacs
│ │ │   --- /tmp/guix-directory.vqBGkT/share/emacs/site-lisp
│ │ ├── +++ /tmp/guix-directory.H4HEMX/share/emacs/site-lisp
│ │ │ │   --- /tmp/guix-directory.vqBGkT/share/emacs/site-lisp/org-9.4.6
│ │ │ ├── +++ /tmp/guix-directory.H4HEMX/share/emacs/site-lisp/org-9.4.6
│ │ │ │ │   --- /tmp/guix-directory.vqBGkT/share/emacs/site-lisp/org-9.4.6/org-autoloads.el
│ │ │ │ ├── +++ /tmp/guix-directory.H4HEMX/share/emacs/site-lisp/org-9.4.6/org-autoloads.el
│ │ │ │ │ @@ -1071,28 +1071,52 @@
│ │ │ │ │  ;;;### (autoloads nil "org-mouse" "org-mouse.el" (0 1 0 0))
│ │ │ │ │  ;;; Generated autoloads from org-mouse.el
│ │ │ │ │  
│ │ │ │ │  (if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-mouse" '("org-mouse-")))
│ │ │ │ │  
│ │ │ │ │  ;;;***
│ │ │ │ │  
│ │ │ │ │ +;;;### (autoloads "actual autoloads are elsewhere" "org-num" "org-num.el"
│ │ │ │ │ +;;;;;;  (0 1 0 0))
│ │ │ │ │ +;;; Generated autoloads from org-num.el
│ │ │ │ │ +
│ │ │ │ │ +(if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-num" '("org-num-")))
│ │ │ │ │ +
│ │ │ │ │ +;;;***
│ │ │ │ │ +
│ │ │ │ │  ;;;### (autoloads nil "org-pcomplete" "org-pcomplete.el" (0 1 0 0))
│ │ │ │ │  ;;; Generated autoloads from org-pcomplete.el
│ │ │ │ │  
│ │ │ │ │  (if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-pcomplete" '("org-" "pcomplete/org-mode/")))
│ │ │ │ │  
│ │ │ │ │  ;;;***
│ │ │ │ │  
│ │ │ │ │ +;;;### (autoloads "actual autoloads are elsewhere" "org-plot" "org-plot.el"
│ │ │ │ │ +;;;;;;  (0 1 0 0))
│ │ │ │ │ +;;; Generated autoloads from org-plot.el
│ │ │ │ │ +
│ │ │ │ │ +(if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-plot" '("org-plot")))
│ │ │ │ │ +
│ │ │ │ │ +;;;***
│ │ │ │ │ +

│ │ │ │ │  ;;;### (autoloads nil "org-protocol" "org-protocol.el" (0 1 0 0))
│ │ │ │ │  ;;; Generated autoloads from org-protocol.el
│ │ │ │ │  
│ │ │ │ │  (if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-protocol" '("org-protocol-")))
│ │ │ │ │  
│ │ │ │ │  ;;;***

Thanks,

Ludo’.

Morgan.J.Smith wrote 3 years ago

[PATCH] gnu: emacs-org: Don't use release tar

Recipients:(name . Morgan Smith)(address . Morgan.J.Smith@outlook.com)

Message-ID:BYAPR05MB402332594AFD396371C61822C5639@BYAPR05MB4023.namprd05.prod.outlook.com

* gnu/packages/emacs-xyz.scm (emacs-org):

[source]: Use git repository

[arguments]: Added phases to help with building

[native-inputs]: Added texinfo

---

So I still have no clue why the current emacs-org package is not reproducible,

but I made a package that is reproducible. It also is building from the

development source so people can use package transformations easier and it

actually runs the tests.

gnu/packages/emacs-xyz.scm | 46 +++++++++++++++++++++++++++-----------

1 file changed, 33 insertions(+), 13 deletions(-)

Toggle diff (64 lines)diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 0cc66b6787..ab7b6fcf6a 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -12439,24 +12439,44 @@ (define-public emacs-org
     (version "9.5")
     (source
      (origin
-       (method url-fetch)
-       (uri (string-append "https://elpa.gnu.org/packages/org-" version ".tar"))
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://git.savannah.gnu.org/git/emacs/org-mode.git")
+             (commit (string-append "release_" version))))
+       (file-name (git-file-name name version))
        (sha256
-        (base32 "16cflg5nms5nb8w86nvwkg49zkl0rvdhigkf4xpvbs0v7zb5y3ky"))))
+        (base32
+         "1gnzx53gkg2c6ljam31bvbibkra66lfx0w04jqmsv2gk8613527r"))))
     (build-system emacs-build-system)
     (arguments
-     `(#:phases
+     `(#:tests? #t
+       #:test-command '("make" "test-dirty")
+       #:phases
        (modify-phases %standard-phases
-         (add-after 'install 'install-documentation
+         (delete 'build)
+         (add-before 'check 'make
+           (lambda _
+             (invoke "make" (string-append "ORGVERSION=" ,version))))
+         (replace 'install
            (lambda* (#:key outputs #:allow-other-keys)
-             (let* ((share (string-append (assoc-ref outputs "out") "/share"))
-                    (info-dir (string-append share "/info"))
-                    (doc-dir (string-append share "/doc/" ,name "-" ,version)))
-               (install-file "org.info" info-dir)
-               (install-file "orgguide.info" info-dir)
-               ;; XXX: "orgcard.pdf" is not built in Org 9.5.
-               ;; (install-file "orgcard.pdf" doc-dir)
-               ))))))
+             (substitute* "local.mk"
+               (("^prefix.*") (string-append "prefix = " (assoc-ref outputs "out")))
+               (("^lispdir.*")
+                (string-append "lispdir = " (elpa-directory (assoc-ref outputs "out")))))
+             (invoke "make" "install")))
+         (add-after 'unpack 'fix-tests
+           (lambda _
+             ;; These files are modified during testing
+             (make-file-writable "testing/examples/babel.org")
+             (make-file-writable "testing/examples/ob-awk-test.org")
+             (make-file-writable "testing/examples/ob-sed-test.org")
+             (make-file-writable "testing/examples/ob-shell-test.org")
+
+             ;; Specify where sh is
+             (substitute* "testing/examples/babel.org"
+               (("/bin/sh") (which "sh"))))))))
+    (native-inputs
+     `(("texinfo" ,texinfo)))
     (home-page "https://orgmode.org/")
     (synopsis "Outline-based notes management and organizer")
     (description "Org is an Emacs mode for keeping notes, maintaining TODO
-- 
2.34.0

Nicolas Goaziou wrote 3 years ago

Recipients:(address . Morgan.J.Smith@outlook.com)

Message-ID:87pmqknge1.fsf@nicolasgoaziou.fr

Hello,

Morgan.J.Smith@outlook.com writes:

Toggle quote (6 lines)

> * gnu/packages/emacs-xyz.scm (emacs-org):

> [source]: Use git repository

> [arguments]: Added phases to help with building

> [native-inputs]: Added texinfo

> ---

I tweaked the definition a bit and applied your patch. Thank you.

There's an issue with the M-x org-version output, tho.

Regards,

Nicolas Goaziou

Closed

Michael Rohleder wrote 3 years ago

Re: bug#48205: [PATCH] gnu: emacs-org: Don't use release tar

Recipients:(address . 48205@debbugs.gnu.org)

Message-ID:8735nb4vl5.fsf@rohleder.de

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

Toggle quote (2 lines)

> There's an issue with the M-x org-version output, tho.

Looks like, this issue breaks emacs-org-re-reveal:

...

starting phase `build'

Checking /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/...

Compiling /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/org-re-reveal-autoloads.el...

Compiling /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/org-re-reveal.el...

Compiling /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/ox-re-reveal.el...

Invalid version syntax: ‘N/A’ (must start with a number)

command "/gnu/store/4w5rkbxb4l0q06bn07qjm5wps316s3sf-emacs-minimal-27.2/bin/emacs" "--quick" "--batch" "--eval=(eval '(progn (setq byte-compile-debug t) (byte-recompile-directory (file-name-as-directory \"/gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4\") 0 1)) t)" failed with status 255

There are only three things certain in programmers life: taxes, death and parsing text.

Morgan.J.Smith wrote 3 years ago

[PATCH] gnu: emacs-org: Preserve version information

Recipients:(name . Morgan Smith)(address . Morgan.J.Smith@outlook.com)

Message-ID:BYAPR05MB40233F38A7720A3D6984D98DC5699@BYAPR05MB4023.namprd05.prod.outlook.com

* gnu/packages/emacs-xyz.scm (emacs-org)[#:phases]: Prevent install phase from

deleting the version information

---

Fixed it!

gnu/packages/emacs-xyz.scm | 4 ++++

1 file changed, 4 insertions(+)

Toggle diff (17 lines)diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 1d4c682482..9ced8198c7 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -12463,6 +12463,10 @@ (define-public emacs-org
                (("^lispdir.*")
                 (string-append "lispdir = "
                                (elpa-directory (assoc-ref outputs "out")))))
+             ;; The dependent targets for install perform cleanup that
+             ;; partially undoes our make phase
+             (substitute* "lisp/Makefile"
+               (("^install:.*") "install:\n"))
              (invoke "make" "install")))
          (add-after 'unpack 'fix-tests
            (lambda* (#:key inputs #:allow-other-keys)
-- 
2.34.0

Nicolas Goaziou wrote 3 years ago

Recipients:(address . Morgan.J.Smith@outlook.com)

Message-ID:87sfv9qwbn.fsf@nicolasgoaziou.fr

Hello,

Morgan.J.Smith@outlook.com writes:

Toggle quote (3 lines)

> * gnu/packages/emacs-xyz.scm (emacs-org)[#:phases]: Prevent install phase from

> deleting the version information

Applied. Thank you.

Regards,

Nicolas Goaziou

Closed

Your comment

This issue is archived.

To comment on this conversation send an email to 48205@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 48205

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

You may also tag this issue. See list of standard tags. For example, to set the confirmed and easy tags

mumi command -t +confirmed -t +easy

Or, remove the moreinfo tag and set the help tag

mumi command -t -moreinfo -t +help

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date