PatchELF can create broken ELF binaries

It appears I found a bug in guix triggered by certain binary data

present in an ELF header.

While running a validate runpath phase of a build-system for a new

package that is not a part of guix I encountered unusual errors

reported for certain binary files. Those binary files RUNPATHs were

modified by patchelf by adding a colon character followed by a

/gnu/store directory folder to runpath. As far as readelf/patchelf and

ld are concerned the binary files appear to contain a valid elf header

with valid DT_DYNAMIC section.

However, something in the ELF header triggers a bug in Guix's

file-runpath / file-dynamic-info procedure that result in the

following output:

scheme@(guix-user)> (file-dynamic-info

"/gnu/store/20z595j5jas5ri3nrza5465gbxwf9kmf-python-redacted/lib/python3.9/site-packages/torch/bin/FileStoreTest")

$13 = #<<elf-dynamic-info> soname: #f needed: ("" "" "" "" "" "" "" ""

"" "" "" "" "" "" "") rpath: () runpath: ()>

scheme@(guix-user)>

As can be seen above file-dynamic-info is unable to read the NEEDED

items, but correctly reports 8 of them. Additionally it is unable to

read the RUNPATH variable and returns an empty string. All those

values are populated in the header as shown below.

readelf -d reports the following correct DT_DYNAMIC:

[luk@archczop guix]$ readelf -d

/gnu/store/20z595j5jas5ri3nrza5465gbxwf9kmf-python-redacted/lib/python3.9/site-packages/torch/bin/FileStoreTest

Dynamic section at offset 0xfcf0 contains 40 entries:

Tag Type Name/Value

0x0000000000000001 (NEEDED) Shared library: [libtorch_cpu.so]

0x0000000000000001 (NEEDED) Shared library:

[libgtest_main.so.1.11.0]

0x0000000000000001 (NEEDED) Shared library: [libgtest.so.1.11.0]

0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0]

0x0000000000000001 (NEEDED) Shared library: [libprotobuf.so.28]

0x0000000000000001 (NEEDED) Shared library: [libc10.so]

0x0000000000000001 (NEEDED) Shared library:

[libmkl_intel_lp64.so.2]

0x0000000000000001 (NEEDED) Shared library:

[libmkl_gnu_thread.so.2]

0x0000000000000001 (NEEDED) Shared library: [libmkl_core.so.2]

0x0000000000000001 (NEEDED) Shared library: [libdl.so.2]

0x0000000000000001 (NEEDED) Shared library: [libstdc++.so.6]

0x0000000000000001 (NEEDED) Shared library: [libm.so.6]

0x0000000000000001 (NEEDED) Shared library: [libgomp.so.1]

0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1]

0x0000000000000001 (NEEDED) Shared library: [libc.so.6]

0x000000000000001d (RUNPATH) Library runpath:

[$ORIGIN/../lib:/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib:/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib:/gnu/store/mbzav28sik3zr3kbw1jyh4qk3zmkh6xn-googletest-1.11.0/lib:/gnu/store/9pyydl5w9xnz1qm56sxn1zh4qny6fkxz-protobuf-3.17.3/lib:/gnu/store/fj5npv9kpsiihrzpzhdlcz5q6bax15s8-mkl-2022.1.0/lib:/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../..:/gnu/store/janq8zcngwc7120gyj41cc2yysk7p9i5-nvidia-libs-515.65.01/lib]

0x000000000000000c (INIT) 0x405000

0x000000000000000d (FINI) 0x40bed4

0x0000000000000019 (INIT_ARRAY) 0x40f6c8

0x000000000000001b (INIT_ARRAYSZ) 16 (bytes)

0x000000000000001a (FINI_ARRAY) 0x40f6d8

0x000000000000001c (FINI_ARRAYSZ) 8 (bytes)

0x0000000000000004 (HASH) 0x402210

0x000000006ffffef5 (GNU_HASH) 0x401fa8

0x0000000000000005 (STRTAB) 0x3ff350

0x0000000000000006 (SYMTAB) 0x4011b0

0x000000000000000a (STRSZ) 7769 (bytes)

0x000000000000000b (SYMENT) 24 (bytes)

0x0000000000000015 (DEBUG) 0x0

0x0000000000000003 (PLTGOT) 0x410000

0x0000000000000002 (PLTRELSZ) 1680 (bytes)

0x0000000000000014 (PLTREL) RELA

0x0000000000000017 (JMPREL) 0x4039e0

0x0000000000000007 (RELA) 0x403710

0x0000000000000008 (RELASZ) 720 (bytes)

0x0000000000000009 (RELAENT) 24 (bytes)

0x000000006ffffffe (VERNEED) 0x403620

0x000000006fffffff (VERNEEDNUM) 4

0x000000006ffffff0 (VERSYM) 0x4034f0

0x0000000000000000 (NULL) 0x0

Therefore deeper analysis of what is in the binary header that

triggers the bug is required. I have a number of those binary files. I

attached the smallest one(74kB) base64 encoded to this email. It is

the binary named FileStoreTest used in the report above. If it gets

stripped from the message I'll reply by submitting it in the body of

the message.

After further troubleshooting it appears the elf binary might be

malformed. I've tried to run the executable on another machine and ld

complained like this:

$ ./FileStoreTest

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: symbol lookup error: ./FileStoreTest: undefined

symbol: , version

Further testing is needed to tell if this is indeed a bug in guix not

processing a correct header, or simply failing to display a meaningful

error message.

Regards,

Lukasz

Also, to ensure all the information is provided. This is the code that

resulted in the binary header being transformed:

(add-after 'install 'fix-issue-with-libs

(lambda* (#:key inputs outputs #:allow-other-keys)

(chdir "..")

(use-modules (ice-9 ftw)

(ice-9 regex)

(ice-9 rdelim)

(ice-9 popen)

(ice-9 textual-ports))

(let* ((libdir (string-append #$output "/lib")))

;; ------------------------------

;; patchelf

(define (get-rpaths file)

(format #t "Getting rpaths from ~a ...~%" file)

(let* ((port (open-input-pipe (string-append "patchelf --print-rpath " file)))

(str (read-line port))) ; from (ice-9 rdelim)

(close-pipe port)

str))

(define (patch-elf file)

(format #t "Patching ~a ...~%" file)

(define rpath (string-append (get-rpaths file) ":" #$extra-libs "/lib"))

(display (string-append "We're setting rpath:" rpath))

(invoke "patchelf" "--set-rpath" rpath file))

(for-each (lambda (file)

(when (elf-file? file)

(patch-elf file)))

(find-files #$output ".*")))))

I can run the patch-elf procedure in repl and it runs fine, but being

run during the build it results in the problematic elf header. The

build was run twice with the same result both times.

Regards,

Lukasz

After further troubleshooting it appears the elf binary might be

malformed by patchelf run during the build process. I've tried to run

the executable on another machine and ld

complained like this:

$ ./FileStoreTest

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: symbol lookup error: ./FileStoreTest: undefined

symbol: , version

This is the code that was supposed to simply add one extra path to the

rpath of binary files (I have added a copy-file step to make a backup

of each file too):

(add-after 'install 'fix-issue-with-libs

(lambda* (#:key inputs outputs #:allow-other-keys)

(chdir "..")

(use-modules (ice-9 ftw)

(ice-9 regex)

(ice-9 rdelim)

(ice-9 popen)

(ice-9 textual-ports))

(let* ((libdir (string-append #$output "/lib")))

;; ------------------------------

;; patchelf

(define (get-rpaths file)

(format #t "Getting rpaths from ~a ...~%" file)

(let* ((port (open-input-pipe (string-append "patchelf --print-rpath " file)))

(str (read-line port))) ; from (ice-9 rdelim)

(close-pipe port)

str))

(define (patch-elf file)

(format #t "Patching ~a ...~%" file)

(define rpath (string-append (get-rpaths file) ":" #$extra-libs "/lib"))

(display (string-append "We're setting rpath:" rpath))

(copy-file file (string-append #$output "/backup_" (car (last-pair

(string-split file #\/)))))

(invoke "patchelf" "--set-rpath" rpath file))

(for-each (lambda (file)

(when (elf-file? file)

(patch-elf file)))

(find-files #$output ".*")))))

When run in repl the patch-elf procedure shown above runs fine. It

adds the path in #$extra-libs to the rpath correctly. However when run

during the build process it most likely mangles the ELF header. At

this stage I can't be sure if it is a bug in guix (runpath check as

well as ld wrapper) or a bug in patchelf, or in the build system.

Therefore if anyone wants to have a look I attach both a binary with

the problematic elf header as well as a backup of the file above

procedure made before running patch-elf. Binaries are base64 encoded.

I used diffoscope to visualise the differences between both files, but

because of an extra offset it shows such a long output I couldn't find

an obvious difference.

Regards,

Lukasz

After further troubleshooting it appears the elf binary might be

malformed by patchelf run during the build process. I've tried to run

the executable on another machine and ld

complained like this:

$ ./FileStoreTest

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: ./FileStoreTest: no version information available

(required by ./FileStoreTest)

./FileStoreTest: symbol lookup error: ./FileStoreTest: undefined

symbol: , version

This is the code that was supposed to simply add one extra path to the

rpath of binary files (I have added a copy-file step to make a backup

of each file too):

(add-after 'install 'fix-issue-with-libs

(lambda* (#:key inputs outputs #:allow-other-keys)

(chdir "..")

(use-modules (ice-9 ftw)

(ice-9 regex)

(ice-9 rdelim)

(ice-9 popen)

(ice-9 textual-ports))

(let* ((libdir (string-append #$output "/lib")))

;; ------------------------------

;; patchelf

(define (get-rpaths file)

(format #t "Getting rpaths from ~a ...~%" file)

(let* ((port (open-input-pipe (string-append "patchelf --print-rpath "

file)))

(str (read-line port))) ; from (ice-9 rdelim)

(close-pipe port)

str))

(define (patch-elf file)

(format #t "Patching ~a ...~%" file)

(define rpath (string-append (get-rpaths file) ":" #$extra-libs "/lib"))

(display (string-append "We're setting rpath:" rpath))

(copy-file file (string-append #$output "/backup_" (car (last-pair

(string-split file #\/)))))

(invoke "patchelf" "--set-rpath" rpath file))

(for-each (lambda (file)

(when (elf-file? file)

(patch-elf file)))

(find-files #$output ".*")))))

When run in repl the patch-elf procedure shown above runs fine. It

adds the path in #$extra-libs to the rpath correctly. However when run

during the build process it most likely mangles the ELF header. At

this stage I can't be sure if it is a bug in guix (runpath check as

well as ld wrapper) or a bug in patchelf, or in the build system.

Therefore if anyone wants to have a look I attach both a binary with

the problematic elf header as well as a backup of the file above

procedure made before running patch-elf. Binaries are base64 encoded.

I used diffoscope to visualise the differences between both files, but

because of an extra offset it shows such a long output I couldn't find

an obvious difference.

Regards,

Lukasz

Sorry, for multiple emails with the same content. There was a delay of

over 10h in those emails showing up and I thought they were not

getting through.

Regarding the issue. It seems it has to do with the strip phase. I

managed to replicate the whole issue outside the build system by

taking binaries backed up after install.

Then if I run $ strip --strip-unneeded --enable-deterministic-archives

file the files can be run fine, but if I use patchelf to add an extra

folder to the rpath strip complains like this:

$ strip --strip-unneeded --enable-deterministic-archives

/home/luk/dev/backup_FileStoreTest

strip: /home/luk/dev/stt5WKN1: warning: allocated section `.dynstr'

not in segment

Then the binary has its elf header mangled as described previously.

By copying the unmodified file, modifying rpath and running strip a

couple of times I found that there is no problem if the rpath change

results in rpath of the same or shorter length, but adding even one

byte to it makes 'strip' later complain and mangle the binary.

Regards,

Lukasz

Hi Lukasz,

Lukasz Olszewski <email@lukaszolszewski.info> skribis:

I believe PatchELF has the potential to break binaries, especially when

trying to extend RUNPATH (the data has to fit in the string table;

PatchELF is supposed to be able to grow the string table as needed, but

there might be bugs.)

It looks like a workaround is to not run ‘strip’, right?

Ludo’.

retitle 58033 PatchELF can create broken ELF binaries

quit

I think I've found another instance of this bug.

Unlike https://issues.guix.gnu.org/58033#0, I'm getting an error

message from validate-runpath, in the form of a type error. Regardless

of whether PatchELF's output is correct, I think there shouldn't be any

type errors.

$ guix build -f guix.scm

[...]

validating RUNPATH of 1 binaries in

"/gnu/store/cn7m18zrxp3ba8dxp6hy1w8jdr0kqp43-gearhead2-0.1/bin"...

error: in phase 'validate-runpath': uncaught exception:

wrong-type-arg "struct-vtable" "Wrong type argument in position ~A

(expecting ~A): ~S" (1 "struct" #f) (#f)

phase `validate-runpath' failed after 0.0 seconds

Backtrace:

18 (primitive-load "/gnu/store/ygpsgmga4qsp042z5nf9av4fm3y…")

In guix/build/gnu-build-system.scm:

906:2 17 (gnu-build #:source _ #:outputs _ #:inputs _ #:phases . #)

In ice-9/boot-9.scm:

1752:10 16 (with-exception-handler _ _ #:unwind? _ # _)

In srfi/srfi-1.scm:

634:9 15 (for-each #<procedure 7fffeeb52420 at guix/build/gnu-b…> …)

In ice-9/boot-9.scm:

1752:10 14 (with-exception-handler _ _ #:unwind? _ # _)

In guix/build/gnu-build-system.scm:

927:23 13 (_)

567:16 12 (validate-runpath #:validate-runpath? _ # _ #:outputs _)

In guix/build/utils.scm:

677:23 11 (loop ("/gnu/store/cn7m18zrxp3ba8dxp6hy1w8jdr0kqp43-g…") …)

677:23 10 (loop ("/gnu/store/cn7m18zrxp3ba8dxp6hy1w8jdr0kqp43-g…") …)

In guix/build/gremlin.scm:

355:2 9 (validate-needed-in-runpath "/gnu/store/cn7m18zrxp3ba8…" …)

In ice-9/boot-9.scm:

1752:10 8 (with-exception-handler _ _ #:unwind? _ # _)

In guix/build/gremlin.scm:

368:20 7 (_)

228:20 6 (elf-dynamic-info #<<elf> bytes: #vu8(127 69 76 70 2 1 …>)

In srfi/srfi-1.scm:

586:17 5 (map1 (#<<dynamic-entry> type: 29 value: 905 offset:…> …))

In guix/build/gremlin.scm:

189:32 4 (interpret-dynamic-entry #<<dynamic-entry> type: 29 val…>)

166:14 3 (vma->offset #<<elf> bytes: #vu8(127 69 76 70 2 1 1 0 …> …)

In ice-9/boot-9.scm:

1685:16 2 (raise-exception _ #:continuable? _)

1683:16 1 (raise-exception _ #:continuable? _)

1685:16 0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:

In procedure struct-vtable: Wrong type argument in position 1 (expecting

struct): #f

builder for

`/gnu/store/3x0bsdyczsxjhmwynvb3mgh5x75i88s4-gearhead2-0.1.drv' failed

with exit code 1

The file "guix.scm" contains the following (remember to comment-out

#:strip-binaries? #false) (while there is a (license #f) line, that's

just because I didn't fill everything in yet; the software is actually

free):

(use-modules (guix gexp) (guix packages) (guix build-system gnu) (guix

utils) (guix hash) (gnu packages pascal)

(gnu packages sdl) (gnu packages elf))

(define (fpc-compilation-options)

;; -P and -T needs to be set for cross-compilation

#~`(#$(cond ((target-aarch64?) "-Paarch64")

((target-ppc32?) "-Ppowerpc")

((target-ppc64le?) "-Ppowerpc64")

((target-x86-64?) "-Px86_64")

((target-x86-32?) "-Pi386")

((target-arm32?) "-Parm")

(#true (error "unrecognised cross-compilation target")))

#$(cond ((target-linux?) "-Tlinux")

((target-hurd?) "-Tlinux") ; untested, hopefully it's

close enough

(#true (error "unrecognised cross-compilation target")))

"-O2" ; do some optimisations

;; Let fpc find libraries.

,@(append-map (lambda (x)

(list (string-append "-Fl" x) ; fpc's equivalent

of -L

(string-append "-Xr" x))) ; equivalent of

-Wl,rpath

(string-split (getenv #$(if (%current-target-system)

"CROSS_LIBRARY_PATH"

"LIBRARY_PATH")) #\:))))

(package

(name "gearhead2")

(version "0.1") ; TODO

(build-system gnu-build-system)

(source (local-file "../gearhead-2" #:recursive? #true

#:select? (negate vcs-file?)))

(arguments

(list #:modules `((srfi srfi-1) ,@%gnu-build-system-modules)

;; #:strip-binaries? #true causes type errors in

validate-runpath,

;; see https://issues.guix.gnu.org/58033.

#:strip-binaries? #false ; strip-binaries? #true causes type

errors in

#:phases

#~(modify-phases %standard-phases

(delete 'configure)

(replace 'build

(lambda _

(apply invoke "fpc" "gearhead2"

#$(fpc-compilation-options))

;; XXX: somehow the -Xr in fpc-compilation-options has no

;; effect. Work-around this.

(invoke "patchelf" "--set-rpath"

(getenv #$(if (%current-target-system)

"CROSS_LIBRARY_PATH"

"LIBRARY_PATH"))

"gearhead2")))

(delete 'check) ; no test suite exists

(replace 'install

(lambda _

(install-file "gearhead2" (string-append #$output

"/bin")))))))

(native-inputs (list patchelf fpc))

(inputs (list sdl sdl-image sdl-ttf))

(synopsis #f)

(description #f)

(home-page #f)

(license #f))