[PATCH] gnu: emacs-org: Make build reproducible

Done

Details

4 participants

Morgan.J.Smith
Ludovic Courtès
Nicolas Goaziou
Michael Rohleder

Owner: unassigned

Submitted by: Morgan.J.Smith

Severity: normal

Morgan.J.Smith wrote on 3 May 2021 22:54

Recipients:(address . guix-patches@gnu.org)(name . Morgan Smith)(address . Morgan.J.Smith@outlook.com)

Message-ID:BYAPR05MB40239B904A0A20F4BFB8B2A6C55B9@BYAPR05MB4023.namprd05.prod.outlook.com

* gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use

autoloads instead

---

gnu/packages/emacs-xyz.scm | 6 ++++++

1 file changed, 6 insertions(+)

Toggle diff (19 lines)diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index c950f116f9..a73b7959ef 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -11243,6 +11243,12 @@ (define-public emacs-org
                (("org-release \"\"")
                 (string-append "org-release \"" ,version "\"")))
              #t))
+         (add-after 'fix-org-version 'use-autoloads
+           (lambda _
+             (delete-file "org-loaddefs.el")
+             (substitute* (find-files "." ".*.el")
+                 (("org-loaddefs.el") "org-autoloads.el"))
+             #t))
          (add-after 'install 'install-documentation
            (lambda* (#:key outputs #:allow-other-keys)
              (let* ((share (string-append (assoc-ref outputs "out") "/share"))
-- 
2.31.1

Ludovic Courtès wrote on 11 May 2021 12:28

Recipients:(address . Morgan.J.Smith@outlook.com)(address . 48205@debbugs.gnu.org)

Message-ID:87h7j936ox.fsf@gnu.org

Hi,

Morgan.J.Smith@outlook.com skribis:

Toggle quote (5 lines)

> From: Morgan Smith <Morgan.J.Smith@outlook.com>

> * gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use

> autoloads instead

[...]

Toggle quote (7 lines)

> + (add-after 'fix-org-version 'use-autoloads

> + (lambda _

> + (delete-file "org-loaddefs.el")

> + (substitute* (find-files "." ".*.el")

> + (("org-loaddefs.el") "org-autoloads.el"))

> + #t))

You can drop the trailing #t.

Could you explain in a comment in the phase what it does? It’s not

clear to me how this relates to making the build bit-reproducible.

Remember that ‘find-files’ takes a regexp as its second argument; thus,

to match .el files, you need to write "\\.el$".

Could you send an updated patch?

Thanks,

Ludo’.

Morgan.J.Smith wrote on 9 Jun 2021 17:32

[PATCH v2] gnu: emacs-org: Make build reproducible

Recipients:(address . ludo@gnu.org)

Message-ID:BYAPR05MB4023DFEC30DC46812C6AE7D1C5369@BYAPR05MB4023.namprd05.prod.outlook.com

* gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use

autoloads instead. Copy autoloads into build directory.

---

Thanks for the suggestions! Sorry about the delay. Also sorry about the

comment. It's probably a little too long :/. I need to work on my english and

my ability to be concise.

gnu/packages/emacs-xyz.scm | 21 +++++++++++++++++++++

1 file changed, 21 insertions(+)

Toggle diff (34 lines)diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 1d954ec5bd..180b2709fe 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -11761,6 +11761,27 @@ (define-public emacs-org
     (arguments
      `(#:phases
        (modify-phases %standard-phases
+         ;; Org ships with the file org-loaddefs.el that functions like our
+         ;; autoloads. Something about the interaction between the loaddefs
+         ;; and autoloads makes the build non-deterministic. We should opt for
+         ;; our autoloads system over the loaddefs since the loaddefs are
+         ;; autogenerated for the release tarball. org-loaddefs.el is
+         ;; mentioned by name in the source files so we have to do a blanket
+         ;; regex search and replace.
+         (add-after 'unpack 'use-autoloads
+           (lambda _
+             (delete-file "org-loaddefs.el")
+             (substitute* (find-files "." "\\.el$")
+               (("org-loaddefs.el") "org-autoloads.el"))))
+         ;; We create the autoloads directly in the install folder. During
+         ;; compilation org checks for the autoloads file so we need to copy
+         ;; it to the build directory.
+         (add-after 'make-autoloads 'copy-autoload-to-build-dir
+           (lambda* (#:key outputs #:allow-other-keys)
+             (copy-file (string-append
+                         (elpa-directory (assoc-ref outputs "out"))
+                         "/org-autoloads.el")
+                        "org-autoloads.el")))
          (add-after 'install 'install-documentation
            (lambda* (#:key outputs #:allow-other-keys)
              (let* ((share (string-append (assoc-ref outputs "out") "/share"))
-- 
2.32.0

Ludovic Courtès wrote on 12 Jun 2021 00:54

Recipients:(address . Morgan.J.Smith@outlook.com)(address . 48205@debbugs.gnu.org)

Message-ID:87a6nw3tcj.fsf@gnu.org

Hi,

Morgan.J.Smith@outlook.com skribis:

Toggle quote (5 lines)

> From: Morgan Smith <Morgan.J.Smith@outlook.com>

> * gnu/packages/emacs-xyz.scm (emacs-org)[phases]: Remove loaddefs and use

> autoloads instead. Copy autoloads into build directory.

[...]

Toggle quote (9 lines)

> (modify-phases %standard-phases

> + ;; Org ships with the file org-loaddefs.el that functions like our

> + ;; autoloads. Something about the interaction between the loaddefs

> + ;; and autoloads makes the build non-deterministic. We should opt for

> + ;; our autoloads system over the loaddefs since the loaddefs are

> + ;; autogenerated for the release tarball. org-loaddefs.el is

> + ;; mentioned by name in the source files so we have to do a blanket

> + ;; regex search and replace.

I wonder about the interaction.

Are you sure using org-autoloads.el everywhere is enough to fix the

problem? With current master, I see:

Toggle snippet (11 lines)$ guix challenge emacs-org --substitute-urls='https://ci.guix.gnu.org https://bayfront.guix.gnu.org'
/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6 contents differ:
  no local build for '/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6'
  https://ci.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 098vc0wpc0rp9qrmh8l75n77j0bd4xjya3q8vd317rxdsfs9dd4n
  https://bayfront.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 0zfzlf28rs7rjvvlq1d3b7cngcvz5grvrkjvsc5j9vp0ikn0zg9p
  differing files:
    /share/emacs/site-lisp/org-9.4.6/org-loaddefs.el
    /share/emacs/site-lisp/org-9.4.6/org-autoloads.elc
    /share/emacs/site-lisp/org-9.4.6/org-autoloads.el

And with diffoscope:

Toggle snippet (57 lines)$ guix challenge emacs-org --substitute-urls='https://ci.guix.gnu.org https://bayfront.guix.gnu.org' --diff=diffoscope
/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6 contents differ:
  no local build for '/gnu/store/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6'
  https://ci.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 098vc0wpc0rp9qrmh8l75n77j0bd4xjya3q8vd317rxdsfs9dd4n
  https://bayfront.guix.gnu.org/nar/lzip/2pny4z6mbi2aybgzzxz0yrzkds7hbpmq-emacs-org-9.4.6: 0zfzlf28rs7rjvvlq1d3b7cngcvz5grvrkjvsc5j9vp0ikn0zg9p
 bayfront.guix.gnu.org  1.7MiB                          7.9MiB/s 00:00 [##################] 100.0%--- /tmp/guix-directory.vqBGkT
+++ /tmp/guix-directory.H4HEMX
?   --- /tmp/guix-directory.vqBGkT/share
??? +++ /tmp/guix-directory.H4HEMX/share
? ?   --- /tmp/guix-directory.vqBGkT/share/emacs
? ??? +++ /tmp/guix-directory.H4HEMX/share/emacs
? ? ?   --- /tmp/guix-directory.vqBGkT/share/emacs/site-lisp
? ? ??? +++ /tmp/guix-directory.H4HEMX/share/emacs/site-lisp
? ? ? ?   --- /tmp/guix-directory.vqBGkT/share/emacs/site-lisp/org-9.4.6
? ? ? ??? +++ /tmp/guix-directory.H4HEMX/share/emacs/site-lisp/org-9.4.6
? ? ? ? ?   --- /tmp/guix-directory.vqBGkT/share/emacs/site-lisp/org-9.4.6/org-autoloads.el
? ? ? ? ??? +++ /tmp/guix-directory.H4HEMX/share/emacs/site-lisp/org-9.4.6/org-autoloads.el
? ? ? ? ? @@ -1071,28 +1071,52 @@
? ? ? ? ?  ;;;### (autoloads nil "org-mouse" "org-mouse.el" (0 1 0 0))
? ? ? ? ?  ;;; Generated autoloads from org-mouse.el
? ? ? ? ?  
? ? ? ? ?  (if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-mouse" '("org-mouse-")))
? ? ? ? ?  
? ? ? ? ?  ;;;***
? ? ? ? ?  
? ? ? ? ? +;;;### (autoloads "actual autoloads are elsewhere" "org-num" "org-num.el"
? ? ? ? ? +;;;;;;  (0 1 0 0))
? ? ? ? ? +;;; Generated autoloads from org-num.el
? ? ? ? ? +
? ? ? ? ? +(if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-num" '("org-num-")))
? ? ? ? ? +
? ? ? ? ? +;;;***
? ? ? ? ? +
? ? ? ? ?  ;;;### (autoloads nil "org-pcomplete" "org-pcomplete.el" (0 1 0 0))
? ? ? ? ?  ;;; Generated autoloads from org-pcomplete.el
? ? ? ? ?  
? ? ? ? ?  (if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-pcomplete" '("org-" "pcomplete/org-mode/")))
? ? ? ? ?  
? ? ? ? ?  ;;;***
? ? ? ? ?  
? ? ? ? ? +;;;### (autoloads "actual autoloads are elsewhere" "org-plot" "org-plot.el"
? ? ? ? ? +;;;;;;  (0 1 0 0))
? ? ? ? ? +;;; Generated autoloads from org-plot.el
? ? ? ? ? +
? ? ? ? ? +(if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-plot" '("org-plot")))
? ? ? ? ? +
? ? ? ? ? +;;;***
? ? ? ? ? +

? ? ? ? ?  ;;;### (autoloads nil "org-protocol" "org-protocol.el" (0 1 0 0))
? ? ? ? ?  ;;; Generated autoloads from org-protocol.el
? ? ? ? ?  
? ? ? ? ?  (if (fboundp 'register-definition-prefixes) (register-definition-prefixes "org-protocol" '("org-protocol-")))
? ? ? ? ?  
? ? ? ? ?  ;;;***

Thanks,

Ludo’.

Morgan.J.Smith wrote on 26 Nov 2021 06:12

[PATCH] gnu: emacs-org: Don't use release tar

Recipients:(name . Morgan Smith)(address . Morgan.J.Smith@outlook.com)

Message-ID:BYAPR05MB402332594AFD396371C61822C5639@BYAPR05MB4023.namprd05.prod.outlook.com

* gnu/packages/emacs-xyz.scm (emacs-org):

[source]: Use git repository

[arguments]: Added phases to help with building

[native-inputs]: Added texinfo

---

So I still have no clue why the current emacs-org package is not reproducible,

but I made a package that is reproducible. It also is building from the

development source so people can use package transformations easier and it

actually runs the tests.

gnu/packages/emacs-xyz.scm | 46 +++++++++++++++++++++++++++-----------

1 file changed, 33 insertions(+), 13 deletions(-)

Toggle diff (64 lines)diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 0cc66b6787..ab7b6fcf6a 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -12439,24 +12439,44 @@ (define-public emacs-org
     (version "9.5")
     (source
      (origin
-       (method url-fetch)
-       (uri (string-append "https://elpa.gnu.org/packages/org-" version ".tar"))
+       (method git-fetch)
+       (uri (git-reference
+             (url "https://git.savannah.gnu.org/git/emacs/org-mode.git")
+             (commit (string-append "release_" version))))
+       (file-name (git-file-name name version))
        (sha256
-        (base32 "16cflg5nms5nb8w86nvwkg49zkl0rvdhigkf4xpvbs0v7zb5y3ky"))))
+        (base32
+         "1gnzx53gkg2c6ljam31bvbibkra66lfx0w04jqmsv2gk8613527r"))))
     (build-system emacs-build-system)
     (arguments
-     `(#:phases
+     `(#:tests? #t
+       #:test-command '("make" "test-dirty")
+       #:phases
        (modify-phases %standard-phases
-         (add-after 'install 'install-documentation
+         (delete 'build)
+         (add-before 'check 'make
+           (lambda _
+             (invoke "make" (string-append "ORGVERSION=" ,version))))
+         (replace 'install
            (lambda* (#:key outputs #:allow-other-keys)
-             (let* ((share (string-append (assoc-ref outputs "out") "/share"))
-                    (info-dir (string-append share "/info"))
-                    (doc-dir (string-append share "/doc/" ,name "-" ,version)))
-               (install-file "org.info" info-dir)
-               (install-file "orgguide.info" info-dir)
-               ;; XXX: "orgcard.pdf" is not built in Org 9.5.
-               ;; (install-file "orgcard.pdf" doc-dir)
-               ))))))
+             (substitute* "local.mk"
+               (("^prefix.*") (string-append "prefix = " (assoc-ref outputs "out")))
+               (("^lispdir.*")
+                (string-append "lispdir = " (elpa-directory (assoc-ref outputs "out")))))
+             (invoke "make" "install")))
+         (add-after 'unpack 'fix-tests
+           (lambda _
+             ;; These files are modified during testing
+             (make-file-writable "testing/examples/babel.org")
+             (make-file-writable "testing/examples/ob-awk-test.org")
+             (make-file-writable "testing/examples/ob-sed-test.org")
+             (make-file-writable "testing/examples/ob-shell-test.org")
+
+             ;; Specify where sh is
+             (substitute* "testing/examples/babel.org"
+               (("/bin/sh") (which "sh"))))))))
+    (native-inputs
+     `(("texinfo" ,texinfo)))
     (home-page "https://orgmode.org/")
     (synopsis "Outline-based notes management and organizer")
     (description "Org is an Emacs mode for keeping notes, maintaining TODO
-- 
2.34.0

Nicolas Goaziou wrote on 28 Nov 2021 22:01

Recipients:(address . Morgan.J.Smith@outlook.com)

Message-ID:87pmqknge1.fsf@nicolasgoaziou.fr

Hello,

Morgan.J.Smith@outlook.com writes:

Toggle quote (6 lines)

> * gnu/packages/emacs-xyz.scm (emacs-org):

> [source]: Use git repository

> [arguments]: Added phases to help with building

> [native-inputs]: Added texinfo

> ---

I tweaked the definition a bit and applied your patch. Thank you.

There's an issue with the M-x org-version output, tho.

Regards,

Nicolas Goaziou

Closed

Michael Rohleder wrote on 2 Dec 2021 02:57

Re: bug#48205: [PATCH] gnu: emacs-org: Don't use release tar

Recipients:(address . 48205@debbugs.gnu.org)

Message-ID:8735nb4vl5.fsf@rohleder.de

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

Toggle quote (2 lines)

> There's an issue with the M-x org-version output, tho.

Looks like, this issue breaks emacs-org-re-reveal:

...

starting phase `build'

Checking /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/...

Compiling /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/org-re-reveal-autoloads.el...

Compiling /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/org-re-reveal.el...

Compiling /gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4/ox-re-reveal.el...

Invalid version syntax: ‘N/A’ (must start with a number)

command "/gnu/store/4w5rkbxb4l0q06bn07qjm5wps316s3sf-emacs-minimal-27.2/bin/emacs" "--quick" "--batch" "--eval=(eval '(progn (setq byte-compile-debug t) (byte-recompile-directory (file-name-as-directory \"/gnu/store/k6b8jwnd4l82h7g2dmlh1fcy41ycd0ih-emacs-org-re-reveal-3.12.4/share/emacs/site-lisp/org-re-reveal-3.12.4\") 0 1)) t)" failed with status 255

There are only three things certain in programmers life: taxes, death and parsing text.

Morgan.J.Smith wrote on 2 Dec 2021 18:38

[PATCH] gnu: emacs-org: Preserve version information

Recipients:(name . Morgan Smith)(address . Morgan.J.Smith@outlook.com)

Message-ID:BYAPR05MB40233F38A7720A3D6984D98DC5699@BYAPR05MB4023.namprd05.prod.outlook.com

* gnu/packages/emacs-xyz.scm (emacs-org)[#:phases]: Prevent install phase from

deleting the version information

---

Fixed it!

gnu/packages/emacs-xyz.scm | 4 ++++

1 file changed, 4 insertions(+)

Toggle diff (17 lines)diff --git a/gnu/packages/emacs-xyz.scm b/gnu/packages/emacs-xyz.scm
index 1d4c682482..9ced8198c7 100644
--- a/gnu/packages/emacs-xyz.scm
+++ b/gnu/packages/emacs-xyz.scm
@@ -12463,6 +12463,10 @@ (define-public emacs-org
                (("^lispdir.*")
                 (string-append "lispdir = "
                                (elpa-directory (assoc-ref outputs "out")))))
+             ;; The dependent targets for install perform cleanup that
+             ;; partially undoes our make phase
+             (substitute* "lisp/Makefile"
+               (("^install:.*") "install:\n"))
              (invoke "make" "install")))
          (add-after 'unpack 'fix-tests
            (lambda* (#:key inputs #:allow-other-keys)
-- 
2.34.0

Nicolas Goaziou wrote on 3 Dec 2021 21:15

Recipients:(address . Morgan.J.Smith@outlook.com)

Message-ID:87sfv9qwbn.fsf@nicolasgoaziou.fr

Hello,

Morgan.J.Smith@outlook.com writes:

Toggle quote (3 lines)

> * gnu/packages/emacs-xyz.scm (emacs-org)[#:phases]: Prevent install phase from

> deleting the version information

Applied. Thank you.

Regards,

Nicolas Goaziou

Closed

Your comment

This issue is archived.

To comment on this conversation send an email to 48205@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date