Bootstrapping Zig with no Binary Blobs

  • Open
  • quality assurance status badge
Details
5 participants
  • Efraim Flashner
  • Ekaitz Zarraga
  • Hilton Chain
  • Motiejus Jakštys
  • Noé Lopez
Owner
Somebody
Submitted by
Ekaitz Zarraga
Severity
normal
E
E
Ekaitz Zarraga wrote on 5 Nov 22:47 +0100
(name . bug-guix@gnu.org)(address . bug-guix@gnu.org)
af98e6c6-57fe-4db0-af71-f7635c939884@elenq.tech
Hi,

In order to include modern versions of Zig (Zig 0.12+) in Guix, we need
to remove the binary blobs.

I open this issue to track this effort and store information about the
process.

Some Guix user is trying to achieve the same goal:


And discussing about it here:


We could use that effort as a reference and package it to Guix.
E
E
Ekaitz Zarraga wrote on 5 Nov 23:05 +0100
(no subject)
(name . control@debbugs.gnu.org)(address . control@debbugs.gnu.org)
66921ed7-ff38-4b4c-a35f-0dc5622c451f@elenq.tech
owner 74217 !
block 72386 by 74217
thanks
H
H
Hilton Chain wrote on 7 Nov 02:19 +0100
Re: Bootstrapping Zig with no Binary Blobs
(name . Ekaitz Zarraga)(address . ekaitz@elenq.tech)
87cyj7g7s4.wl-hako@ultrarare.space
Hi Ekaitz,

On Wed, 06 Nov 2024 05:47:48 +0800,
Ekaitz Zarraga wrote:
Toggle quote (17 lines)
>
> Hi,
>
> In order to include modern versions of Zig (Zig 0.12+) in Guix, we need to
> remove the binary blobs.
>
> I open this issue to track this effort and store information about the process.
>
> Some Guix user is trying to achieve the same goal:
>
> https://git.jakstys.lt/motiejus/zig-repro
>
> And discussing about it here:
>
> https://ziggit.dev/t/building-self-hosted-from-the-original-c-implementation/6607/11


Great news, thanks for sharing!


Toggle quote (3 lines)
> We could use that effort as a reference and package it to Guix.


Excited to know Zig developers are willing to help with bootstrapping! We
shouldn't miss this chance :)

I'll look into replicating the current progress this weekend. Anyone reading
this mail wants to join in?


Thanks
N
N
Noé Lopez wrote on 7 Nov 23:06 +0100
(address . 74217@debbugs.gnu.org)
87ldxuk8bn.fsf@xn--no-cja.eu
Toggle quote (6 lines)
>Excited to know Zig developers are willing to help with bootstrapping! We
>shouldn't miss this chance :)
>
>I'll look into replicating the current progress this weekend. Anyone reading
>this mail wants to join in?

Hey Hilton and Ekaitz,

I’m interested in this :) From what I can see the current effort in the
zig-repro repository is very well made and we should just need to
replicate each step with the correct dependencies, hoping they exist ?

I can try to make the first few steps as packages tomorrow, WDYT?

Good evening,
Noé
E
E
Ekaitz Zarraga wrote on 7 Nov 23:09 +0100
Re: bug#74217: Bootstrapping Zig with no Binary Blobs
(name . Hilton Chain)(address . hako@ultrarare.space)
844f9d54-ef96-47f6-950e-655a9ccc9d47@elenq.tech
Hi

Toggle quote (2 lines)
> I can try to make the first few steps as packages tomorrow, WDYT?

Sure!
Send the patches to this issue and I'll review when I have time.

Thanks for the energy!
H
H
Hilton Chain wrote 6 days ago
[PATCH 0/2] Initial step on bootstrapping Zig.
(address . 74217@debbugs.gnu.org)(name . Hilton Chain)(address . hako@ultrarare.space)
cover.1731084096.git.hako@ultrarare.space
Finished step 00 from Motiejus Jakštys's script[1], this should make later
work on Guix side easier.


Hilton Chain (2):
gnu: Add zig-0.10.0-610-bootstrap.
gnu: Add zig-0.10.0-610.

gnu/local.mk | 1 +
...10.0-610-bootstrap-resolve-conflicts.patch | 87 ++++++++++
gnu/packages/zig.scm | 151 +++++++++++++++++-
3 files changed, 238 insertions(+), 1 deletion(-)
create mode 100644 gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch


base-commit: 2a6d96425eea57dc6dd48a2bec16743046e32e06
--
2.46.0
H
H
Hilton Chain wrote 6 days ago
[PATCH 1/2] gnu: Add zig-0.10.0-610-bootstrap.
(address . 74217@debbugs.gnu.org)(name . Hilton Chain)(address . hako@ultrarare.space)
81bba1694a0f1ff48967727855b158487340deb9.1731084096.git.hako@ultrarare.space
* gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch: New
file.
* gnu/local.mk (dist_patch_DATA): Regisiter it.
* gnu/packages/zig.scm (zig-0.10.0-538-source,zig-0.10.0-539-patch)
(zig-0.10.0-542-patch,zig-0.10.0-610-bootstrap): New variables.

Change-Id: I132bbad34f40b919b4573e02d0f40eb4a007a26c
---
gnu/local.mk | 1 +
...10.0-610-bootstrap-resolve-conflicts.patch | 87 +++++++++++++++
gnu/packages/zig.scm | 105 +++++++++++++++++-
3 files changed, 192 insertions(+), 1 deletion(-)
create mode 100644 gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch

Toggle diff (234 lines)
diff --git a/gnu/local.mk b/gnu/local.mk
index ae902a5ab2..2cc2ea4c81 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -2358,6 +2358,7 @@ dist_patch_DATA = \
%D%/packages/patches/xygrib-newer-proj.patch \
%D%/packages/patches/yggdrasil-extra-config.patch \
%D%/packages/patches/zig-0.9-riscv-support.patch \
+ %D%/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch \
%D%/packages/patches/zig-use-baseline-cpu-by-default.patch \
%D%/packages/patches/zig-use-system-paths.patch \
%D%/packages/patches/zsh-egrep-failing-test.patch \
diff --git a/gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch b/gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch
new file mode 100644
index 0000000000..5ad5ffc249
--- /dev/null
+++ b/gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch
@@ -0,0 +1,87 @@
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 1c03faf1e9..89406eb1b2 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -846,16 +846,17 @@ else()
+ endif()
+
+ set(ZIG_BUILD_ARGS
+- --zig-lib-dir "${CMAKE_SOURCE_DIR}/lib"
+- "-Dconfig_h=${ZIG_CONFIG_H_OUT}"
+- "-Denable-llvm"
+- ${ZIG_RELEASE_ARG}
+- ${ZIG_STATIC_ARG}
+- ${ZIG_NO_LIB_ARG}
+- ${ZIG_SINGLE_THREADED_ARG}
+- "-Dtarget=${ZIG_TARGET_TRIPLE}"
+- "-Dcpu=${ZIG_TARGET_MCPU}"
+- "-Dversion-string=${RESOLVED_ZIG_VERSION}"
++ --zig-lib-dir "${CMAKE_SOURCE_DIR}/lib"
++ "-Dconfig_h=${ZIG_CONFIG_H_OUT}"
++ "-Denable-llvm"
++ "-Denable-stage1"
++ ${ZIG_RELEASE_ARG}
++ ${ZIG_STATIC_ARG}
++ ${ZIG_NO_LIB_ARG}
++ ${ZIG_SINGLE_THREADED_ARG}
++ "-Dtarget=${ZIG_TARGET_TRIPLE}"
++ "-Dcpu=${ZIG_TARGET_MCPU}"
++ "-Dversion-string=${RESOLVED_ZIG_VERSION}"
+ )
+
+ add_custom_target(stage3 ALL
+diff --git a/build.zig b/build.zig
+index cf0e092326..7f80c3e1df 100644
+--- a/build.zig
++++ b/build.zig
+@@ -142,7 +142,8 @@ pub fn build(b: *Builder) !void {
+ const force_gpa = b.option(bool, "force-gpa", "Force the compiler to use GeneralPurposeAllocator") orelse false;
+ const link_libc = b.option(bool, "force-link-libc", "Force self-hosted compiler to link libc") orelse (enable_llvm or only_c);
+ const sanitize_thread = b.option(bool, "sanitize-thread", "Enable thread-sanitization") orelse false;
+- const strip = b.option(bool, "strip", "Omit debug information");
++ const strip = b.option(bool, "strip", "Omit debug information") orelse false;
++ const use_zig0 = b.option(bool, "zig0", "Bootstrap using zig0") orelse false;
+ const value_tracing = b.option(bool, "value-tracing", "Enable extra state tracking to help troubleshoot bugs in the compiler (using the std.debug.Trace API)") orelse false;
+
+ const mem_leak_frames: u32 = b.option(u32, "mem-leak-frames", "How many stack frames to print when a memory leak occurs. Tests get 2x this amount.") orelse blk: {
+@@ -151,7 +152,22 @@ pub fn build(b: *Builder) !void {
+ break :blk 4;
+ };
+
+- const exe = addCompilerStep(b);
++ if (only_c) {
++ target.ofmt = .c;
++ }
++
++ const main_file: ?[]const u8 = mf: {
++ if (!have_stage1) break :mf "src/main.zig";
++ if (use_zig0) break :mf null;
++ break :mf "src/stage1.zig";
++ };
++
++ const exe = b.addExecutable("zig", main_file);
++
++ const compile_step = b.step("compile", "Build the self-hosted compiler");
++ compile_step.dependOn(&exe.step);
++
++ exe.stack_size = stack_size;
+ exe.strip = strip;
+ exe.sanitize_thread = sanitize_thread;
+ exe.build_id = b.option(bool, "build-id", "Include a build id note") orelse false;
+diff --git a/src/translate_c/ast.zig b/src/translate_c/ast.zig
+index 20e4259725..bc0f002c21 100644
+--- a/src/translate_c/ast.zig
++++ b/src/translate_c/ast.zig
+@@ -1448,6 +1448,12 @@ fn renderNode(c: *Context, node: Node) Allocator.Error!NodeIndex {
+ .optional_type => return renderPrefixOp(c, node, .optional_type, .question_mark, "?"),
+ .address_of => {
+ const payload = node.castTag(.address_of).?.data;
++ if (c.zig_is_stage1 and payload.tag() == .fn_identifier)
++ return try c.addNode(.{
++ .tag = .identifier,
++ .main_token = try c.addIdentifier(payload.castTag(.fn_identifier).?.data),
++ .data = undefined,
++ });
+
+ const ampersand = try c.addToken(.ampersand, "&");
+ const base = if (payload.tag() == .fn_identifier)
diff --git a/gnu/packages/zig.scm b/gnu/packages/zig.scm
index 6994d48818..68907fd04e 100644
--- a/gnu/packages/zig.scm
+++ b/gnu/packages/zig.scm
@@ -23,13 +23,15 @@ (define-module (gnu packages zig)
#:use-module (guix gexp)
#:use-module (guix packages)
#:use-module (guix utils)
+ #:use-module (guix download)
#:use-module (guix git-download)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix build-system cmake)
#:use-module (gnu packages)
#:use-module (gnu packages compression)
#:use-module (gnu packages llvm)
- #:use-module (gnu packages llvm-meta))
+ #:use-module (gnu packages llvm-meta)
+ #:use-module (gnu packages web))
(define-public zig-0.9
(package
@@ -196,4 +198,105 @@ (define-public zig-0.10
(properties `((max-silent-time . 9600)
,@(clang-compiler-cpu-architectures "15")))))
+(define zig-0.10.0-538-source
+ ;; "std: added eql to DynamicBitSet and DynamicBitSetUnmanaged"
+ (let ((commit "bf316e550671cc71eb498b3cf799493627bb0fdc")
+ (revision "538"))
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/ziglang/zig")
+ (commit commit)))
+ (file-name (git-file-name "zig" (git-version "0.10" revision commit)))
+ (sha256
+ (base32 "1dchc2bp842jlw0byssqzindv8cigpqcj2hk3752667jrrww13vv")))))
+
+(define zig-0.10.0-539-patch
+ ;; "remove `-fstage1` option"
+ (let ((commit "28514476ef8c824c3d189d98f23d0f8d23e496ea"))
+ (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/ziglang/zig/commit/" commit ".patch"))
+ (sha256
+ (base32 "0qxxiafg2sd5rr4xhw0c12rygd7zh1rmf3x8hfialyxmsbi5pfxp")))))
+
+(define zig-0.10.0-542-patch
+ ;; "actually remove stage1"
+ (let ((commit "3ba916584db5485c38ebf2390e8d22bc6d81bf8e"))
+ (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/ziglang/zig/commit/" commit ".patch"))
+ (sha256
+ (base32 "1l09gmbr3vqzinb63kvaskgs1d0mvm1m7w3ai3ngwg5zlabyya35")))))
+
+;; Build zig1.wasm from source.
+(define zig-0.10.0-610-bootstrap
+ (let ((commit "e7d28344fa3ee81d6ad7ca5ce1f83d50d8502118")
+ (revision "610"))
+ (package
+ (inherit zig-0.10)
+ (name "zig")
+ (version (git-version "0.10.0" revision commit))
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/ziglang/zig")
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (modules '((guix build utils)))
+ (snippet '(delete-file "stage1/zig1.wasm.zst"))
+ (sha256
+ (base32
+ "08pm3f4hh6djl3szhqgm7fa3qisdl2xh9jrp18m0z7bk2vd0bzw7"))))
+ (arguments
+ (substitute-keyword-arguments (package-arguments zig-0.10)
+ ((#:phases phases '%standard-phases)
+ #~(modify-phases #$phases
+ (add-after 'unpack 'prepare-source
+ (lambda* (#:key native-inputs inputs #:allow-other-keys)
+ (copy-recursively "." "../source-backup")
+ ;; Revert "actually remove stage1".
+ (invoke "patch" "--reverse" "--strip=1"
+ "--input" #+zig-0.10.0-542-patch)
+ ;; Revert "remove `-fstage1` option".
+ (false-if-exception
+ (invoke "patch" "--reverse" "--strip=1"
+ "--input" #+zig-0.10.0-539-patch))
+ ;; Resolve conflicts in previous patching.
+ (invoke
+ "patch" "--forward" "--strip=1" "--input"
+ #+(local-file
+ (search-patch
+ "zig-0.10.0-610-bootstrap-resolve-conflicts.patch")))
+ ;; Restore build system.
+ (rename-file "stage1/config.zig.in" "src/config.zig.in")
+ (substitute* "src/config.zig.in"
+ (("(have_stage1 = )false" _ prefix)
+ (string-append prefix "true")))
+ (for-each
+ (lambda (file)
+ (copy-file (string-append #+zig-0.10.0-538-source "/" file)
+ file))
+ '("build.zig" "CMakeLists.txt"))))
+ (add-after 'install 'build-zig1
+ (lambda _
+ (copy-recursively "../source-backup" ".")
+ (invoke (string-append #$output "/bin/zig")
+ "build" "update-zig1" "--verbose")))
+ (add-after 'build-zig1 'install-zig1
+ (lambda _
+ (install-file "stage1/zig1.wasm.zst"
+ (string-append #$output:zig1 "/bin"))))
+ (replace 'check
+ (lambda* (#:key tests? #:allow-other-keys)
+ (when tests?
+ (invoke (string-append #$output "/bin/zig")
+ "test" "-I" "test" "test/behavior.zig"))))))))
+ (native-inputs
+ (modify-inputs (package-native-inputs zig-0.10)
+ (prepend binaryen)))
+ (outputs '("out" "zig1")))))
+
(define-public zig zig-0.10)
--
2.46.0
H
H
Hilton Chain wrote 6 days ago
[PATCH 2/2] gnu: Add zig-0.10.0-610.
(address . 74217@debbugs.gnu.org)(name . Hilton Chain)(address . hako@ultrarare.space)
91e88fbf01f28428bdf46464efd4912e3fd29f5d.1731084096.git.hako@ultrarare.space
* gnu/packages/zig.scm (zig-0.10.0-610): New variable.

Change-Id: I277a7f5e9781e89d7ad7cd108fec9afcf8cd23d9
---
gnu/packages/zig.scm | 46 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)

Toggle diff (57 lines)
diff --git a/gnu/packages/zig.scm b/gnu/packages/zig.scm
index 68907fd04e..4174dba38b 100644
--- a/gnu/packages/zig.scm
+++ b/gnu/packages/zig.scm
@@ -299,4 +299,50 @@ (define zig-0.10.0-610-bootstrap
(prepend binaryen)))
(outputs '("out" "zig1")))))
+;; Bootstrap with our zig1.wasm.
+(define zig-0.10.0-610
+ (let ((commit "e7d28344fa3ee81d6ad7ca5ce1f83d50d8502118")
+ (revision "610"))
+ (package
+ (inherit zig-0.10)
+ (name "zig")
+ (version (git-version "0.10.0" revision commit))
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/ziglang/zig")
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (modules '((guix build utils)))
+ (snippet '(delete-file "stage1/zig1.wasm.zst"))
+ (sha256
+ (base32
+ "08pm3f4hh6djl3szhqgm7fa3qisdl2xh9jrp18m0z7bk2vd0bzw7"))))
+ (arguments
+ (substitute-keyword-arguments (package-arguments zig-0.10)
+ ((#:phases phases '%standard-phases)
+ #~(modify-phases #$phases
+ (add-after 'unpack 'unpack-zig1
+ (lambda* (#:key native-inputs inputs #:allow-other-keys)
+ (install-file (search-input-file
+ (or native-inputs inputs) "bin/zig1.wasm.zst")
+ "stage1")))
+ (add-after 'install 'update-zig1
+ (lambda _
+ (invoke (string-append #$output "/bin/zig")
+ "build" "update-zig1" "--verbose")))
+ (add-after 'update-zig1 'install-zig1
+ (lambda _
+ (install-file "stage1/zig1.wasm.zst"
+ (string-append #$output:zig1 "/bin"))))
+ (replace 'check
+ (lambda* (#:key tests? #:allow-other-keys)
+ (when tests?
+ (invoke (string-append #$output "/bin/zig")
+ "test" "-I" "test" "test/behavior.zig"))))))))
+ (native-inputs
+ (modify-inputs (package-native-inputs zig-0.10)
+ (prepend binaryen `(,zig-0.10.0-610-bootstrap "zig1"))))
+ (outputs '("out" "zig1")))))
+
(define-public zig zig-0.10)
--
2.46.0
H
H
Hilton Chain wrote 5 days ago
Re: [PATCH 0/2] Initial step on bootstrapping Zig.
(address . 74217@debbugs.gnu.org)
871pzk1fpe.wl-hako@ultrarare.space
I have added a wip-zig-bootstrap[1] branch.

Variables are not exported, you can build them locally with the following
command:
Toggle snippet (9 lines)
ZIG=$(
sed 's/^\(define(-public)? (zig-.*-.*)/\2/p' gnu/packages/zig.scm \
--quiet --regexp-extended |
tail --lines=1
)

./pre-inst-env guix build --expression="(@@ (gnu packages zig) $ZIG)"

E
E
Efraim Flashner wrote 3 days ago
Re: Bootstrapping Zig with no Binary Blobs
(name . Hilton Chain)(address . hako@ultrarare.space)
ZzHtoV-P_QaxhSH5@3900XT
On Thu, Nov 07, 2024 at 09:19:23AM +0800, Hilton Chain wrote:
Toggle quote (33 lines)
> Hi Ekaitz,
>
> On Wed, 06 Nov 2024 05:47:48 +0800,
> Ekaitz Zarraga wrote:
> >
> > Hi,
> >
> > In order to include modern versions of Zig (Zig 0.12+) in Guix, we need to
> > remove the binary blobs.
> >
> > I open this issue to track this effort and store information about the process.
> >
> > Some Guix user is trying to achieve the same goal:
> >
> > https://git.jakstys.lt/motiejus/zig-repro
> >
> > And discussing about it here:
> >
> > https://ziggit.dev/t/building-self-hosted-from-the-original-c-implementation/6607/11
>
>
> Great news, thanks for sharing!
>
>
> > We could use that effort as a reference and package it to Guix.
>
>
> Excited to know Zig developers are willing to help with bootstrapping! We
> shouldn't miss this chance :)
>
> I'll look into replicating the current progress this weekend. Anyone reading
> this mail wants to join in?

Ok, you've convinced me. I guess I finally need to get zig@0.10
building correctly on riscv64.

Expect plenty of FIXUP and SQUASH commits in the future.

--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmcx7Z4ACgkQQarn3Mo9
g1H3VhAAjSvSv881uk3LrTDMsa8YOOkwHPMnxydZwefpzCr9pL1UAUyw0ZRO+GTk
tGmJt4yyA9vCJ/+bkQSnF8J8ttgTrTlgxNs7TGT/NlJM49WaqOIN+VYOEl51+Xaz
cfoYYitDxmSmZY2i9PE+SEiJudfCiJVuZrNsk2Yll2JzTAxYDSllO58xIVXVkkr5
Gkt+t7DWyuBo/LUCAmJY7BrA7yj6rzn3IDiCFPR8K4X6VsQwZa/LeoqzgwGfx+Id
jHBJV6zWmk1cm29aWBtwXSo5pmC0xenOtzOKlDpzbeFCF+Q3/QhhxpaJHErLQBOs
Rw71VJe8yNYEhZVD0XvoNFQ2MtCOv0etIXvyUKQXrmBA1sTF8RVfoJHU6xbHlKfb
t/Xpc/s1aDCNwDmEE3XpsqngtpgIL2xVZW299CdceWvi+AzvWiNrTDSBZ/hmTpCH
/nesxa5Wkc57+ECr0crt/2tOJbckoP7xQYGl4T7Q8eycqR3FWl4vJd+ct1PdZNgt
Y4MnByHkSrX1ilJH8vAUBHKAOm7tesFTabL8T6c+or8i4kY3jnL1PNI2FBi+t8Nr
GCQevedgyRHWYzaY/6B0sRVWlijdus0pNVXlF4uFgNkmrImUt4MJV1CBxzQzyh9D
5E2DHYp07opDQxPc58eMOcNgg5bB7doQGX3H1iX8tH50/PV/Oyc=
=Fjzq
-----END PGP SIGNATURE-----


H
H
Hilton Chain wrote 3 days ago
(name . Efraim Flashner)(address . efraim@flashner.co.il)
87v7wu7zm9.wl-hako@ultrarare.space
Hi Efraim,

On Mon, 11 Nov 2024 19:42:25 +0800,
Efraim Flashner wrote:
Toggle quote (6 lines)
>
> Ok, you've convinced me. I guess I finally need to get zig@0.10
> building correctly on riscv64.
>
> Expect plenty of FIXUP and SQUASH commits in the future.

I have something to fix with a force-push on wip-zig-bootstrap, I'll reply here
when the branch is ready to be worked on :)
E
E
Efraim Flashner wrote 3 days ago
(name . Hilton Chain)(address . hako@ultrarare.space)
ZzHyQOFBBiT2Rd3W@3900XT
On Mon, Nov 11, 2024 at 07:56:30PM +0800, Hilton Chain wrote:
Toggle quote (13 lines)
> Hi Efraim,
>
> On Mon, 11 Nov 2024 19:42:25 +0800,
> Efraim Flashner wrote:
> >
> > Ok, you've convinced me. I guess I finally need to get zig@0.10
> > building correctly on riscv64.
> >
> > Expect plenty of FIXUP and SQUASH commits in the future.
>
> I have something to fix with a force-push on wip-zig-bootstrap, I'll reply here
> when the branch is ready to be worked on :)

No worries. I'm still getting a look around the branch. And if you
force-push then my regular push will be rejected and I can rebase
whatever I have.

--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmcx8jsACgkQQarn3Mo9
g1GS3g/+JNYMc3E6h7EzJLixWVCpmJasMW2b+EFhs1LdlaNIK8c4ds/XpPle3Z4T
yYIoVQAClnabQqrTWgdNGsq2PQjfp/esB4bg4zFaV0ODG9o7gQMJ9K5Um5SQUoSL
z+hUOR8yrdW6dB3cSUXraTx1w0ToIoBl6H2QUpA+Lz8BSBb7PaiXv/AC27KP3t+g
j5CNvCbQvDacYNCvYUm+k9s0GHffYsFMuftpLg0VzRo84FAw92hs9MvC9wYJJ/Pl
YkoOTOiXnLXdJyPMX5acAGE61aFU6re/91uImWXSKWGOR4hZJ6DlY7+yMw9uA53K
WBWDfnrAe7XHeb4PPQRze2MIDsnVYEtbsi//vSYdFZHpqi2MOc9mAfm8vFZWeNjJ
kp0m6/413sFMc5i0Mjdd8X7foScKnvCKmtheO60ZB1fu/cL3Svfkhx9jq6ofE2Q1
zrkPhgP3+AdULqVctTw1M/BYngmaKuVObUWCdM8u095tMu50GEsPQ1ETTUZa0ikj
1NoOmMV5m2chQHj/qazuYhEib4sFwibdbOaBPLzcyB+wt/Kq0E6ymUb9jsgjmxbE
oi6DQRaIUk2I7P03XvtXqmTy8HqPvZvSyrCRkGJK6YIFLwXJFn9nkFjnxdAwkH7B
ZJaalCo93jdO2LrOjZH8j94ISTvrWNZpgGaXHWbJVgp7K+QB/MM=
=t5x9
-----END PGP SIGNATURE-----


H
H
Hilton Chain wrote 30 hours ago
(address . 74217@debbugs.gnu.org)
87ldxngjy4.wl-hako@ultrarare.space
Hello everyone,

With a new forced push (and new rebuilds, sorry), wip-zig-bootstrap is mostly
ready (for me) now. Please take a look at the first few commits, as I'm
changing Zig's behavior there, here are some additional notes:

To Efraim: Can adding pkg-config to native-inputs avoid the ncdu snippet?

I'm building this branch on my personal Cuirass instance[1][2], for x86_64-linux
and aarch64-linux (qemu-binfmt), in previous revisions I have indentified
reproduciblity issue, and aarch64 builds timed-out. I haven't investigated them
yet.

Then for what's the RUNPATH issue I have mentioned in commits:
+ For current zig@0.10 on Guix master: glibc is missing from RUNPATH, which
fails the validate-runpath check.
+ For zig@0.11, some other inputs are missing, making the binary failing to run on Guix[3].
+ dan also mentioned privately to me that they needed to add paths from
LIBRARY_PATH to RUNPATH for their own projects, so programs built by Zig is also
affected.

I think this is due to Zig's implemention of its own linking logic, which
bypasses our ld-wrapper.

I'm not going to implement ld-wrapper within Zig. :) So my proposed workaround
in wip-zig-bootstrap is to patch the handling logic added for Guix:

(In lib/std/zig/system/NativePaths.zig)
Toggle snippet (31 lines)
// Distros like guix don't use FHS, so they rely on environment
// variables to search for headers and libraries.
// We use os.getenv here since this part won't be executed on
// windows, to get rid of unnecessary error handling.
- if (std.posix.getenv("C_INCLUDE_PATH")) |c_include_path| {
+ if (std.posix.getenv("CROSS_C_INCLUDE_PATH") orelse std.posix.getenv("C_INCLUDE_PATH")) |c_include_path| {
var it = mem.tokenizeScalar(u8, c_include_path, ':');
while (it.next()) |dir| {
try self.addIncludeDir(dir);
}
}

- if (std.posix.getenv("CPLUS_INCLUDE_PATH")) |cplus_include_path| {
+ if (std.posix.getenv("CROSS_CPLUS_INCLUDE_PATH") orelse std.posix.getenv("CPLUS_INCLUDE_PATH")) |cplus_include_path| {
var it = mem.tokenizeScalar(u8, cplus_include_path, ':');
while (it.next()) |dir| {
try self.addIncludeDir(dir);
}
}

- if (std.posix.getenv("LIBRARY_PATH")) |library_path| {
+ if (std.posix.getenv("CROSS_LIBRARY_PATH") orelse std.posix.getenv("LIBRARY_PATH")) |library_path| {
var it = mem.tokenizeScalar(u8, library_path, ':');
while (it.next()) |dir| {
try self.addLibDir(dir);
+ try self.addRPath(dir);
}
}
}

Adding directories from CROSS_LIBRARY_PATH or LIBRARY_PATH to RUNPATH, "CROSS_"
part is for our cross toolchain, I haven't tested it yet.

I think this behavior change is reasonable since the search path
(CROSS_)?LIBRARY_PATH is only automatically set by our compilers.

I added this change to 0.9 as well to make all Zigs behave consistently. I also
used shrink-runpath phase from meson-build-system in Zig and zig-build-system.

I want to move shrink-runpath to (guix build utils) and export it too, so that
it can be used easier. But I'm not sure if this change will trigger rebuilds of
other packages, so I didn't do it.

Thanks to Guile, for builds not managed by guix-daemon, something like the
following script can be used, we can ship a program-file if we agree on this
workaround.
Toggle snippet (15 lines)
(use-modules (guix build meson-build-system))

(define shrink-runpath
(assoc-ref %standard-phases 'shrink-runpath))

(define (main directories)
(for-each (lambda (dir)
(false-if-exception
(shrink-runpath
#:elf-directories '(".")
#:outputs `(("out" . ,dir)))))
directories))

(main (cdr (command-line)))
Usage: guile <file-with-above-content> DIRECTORY...


Thanks
---
[2]: https://substitute.boiledscript.com, if you want to challenge it.
E
E
Efraim Flashner wrote 29 hours ago
(name . Hilton Chain)(address . hako@ultrarare.space)
ZzTrpMLuGg9cr70s@3900XT
Attachment: file
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmc066AACgkQQarn3Mo9
g1EnFQ/7BUV3KU9BDhqjdcN/3UHzaNFP0+zfL1eMsFykYB55ebJ+KXuSyB5pQSpI
QtXHeIKR0uk+WUfnSVZNvAz3LvwOFK84X5g7mpKiQJT2AacvkYzvU9ZzDEu9uRqW
D5LfqZNkbVirAWm3ObafPAbRUE1W7TQpgXyzQpFTuBrPEPzFEXYcnKMR29JW73GI
nvD4xlE25xNSQLp66U4QN9CciAehh2YDE/2VGkFIW6xObdmPPAl708gi2WTuW78V
4cqmikiq3ZuJSp9gC1JIKDrlOPVGQluNy8sAI58uyHkLqWF0xh3z6qS/6hGWwtgW
/AjlNmcAhC/0GkknG/AdEsxs2tYqH9Bbsc4TLHOpGl7wXjrGKV5UgjgarviMBcPA
DLr7VjKOOlHPy5tOUMsy0xDL2y1WGt1yhO/Hkumwui8Hd5Cka83rXPy1BkNIWlA0
B8A0g68GVYV9Uu7BfUk5+tmW7dXML8c/1M7tMJIkdCaV7z8WeP1b0QKcMUkO61Sz
1H9VZldUTOQjx8RfXv3Gwl/SDl6uOQibp0qMgGXIVKjgJwuEqwE7LMVOTvgWsQ3T
Ax14pycXy5qaGLOe1lYATr4PCbMsp6eN59bmtiuiKg4YaqFrOaUfmNvQgoX/8jHL
A+1JXT2cPLMXLmkZ+s9RkmARY3qMkJ2JwAwSxp3s6Hr0MRn1kQA=
=+4fE
-----END PGP SIGNATURE-----


H
H
Hilton Chain wrote 23 hours ago
878qtmvh2i.wl-hako@ultrarare.space
On Thu, 14 Nov 2024 02:10:44 +0800,
Efraim Flashner wrote:
Toggle quote (7 lines)
>
> > I think this is due to Zig's implemention of its own linking logic, which
> > bypasses our ld-wrapper.
>
> I wonder if switching from lld to make-lld-wrapper would make a
> difference here.

Thanks for mentioning! This should do the work!

I thought Zig was using lld as a library for linking. Just looked at
src/link/Elf.zig, this is not true, it invokes ld.lld.

I have more time today. I'll test this out and see if lld-as-ld-wrapper can
also add glibc RUNPATH.
H
H
Hilton Chain wrote 22 hours ago
(address . 74217@debbugs.gnu.org)
877c96vd3u.wl-hako@ultrarare.space
On Thu, 14 Nov 2024 07:40:05 +0800,
Hilton Chain wrote:
Toggle quote (18 lines)
>
> On Thu, 14 Nov 2024 02:10:44 +0800,
> Efraim Flashner wrote:
> >
> > > I think this is due to Zig's implemention of its own linking logic, which
> > > bypasses our ld-wrapper.
> >
> > I wonder if switching from lld to make-lld-wrapper would make a
> > difference here.
>
> Thanks for mentioning! This should do the work!
>
> I thought Zig was using lld as a library for linking. Just looked at
> src/link/Elf.zig, this is not true, it invokes ld.lld.
>
> I have more time today. I'll test this out and see if lld-as-ld-wrapper can
> also add glibc RUNPATH.

Zig manages linking to libc seperately, if this approach works, we have to
keep validate-runpath? off for Zig. Reference to ld.lld can't be patched
since it's both native-inputs (for building Zig) and inputs (for using the
built Zig), maybe we can add a zig-toolchain (zig + lld-wrapper) later?
H
H
Hilton Chain wrote 17 hours ago
(address . 74217@debbugs.gnu.org)
874j4auz7j.wl-hako@ultrarare.space
On Thu, 14 Nov 2024 09:05:41 +0800,
Hilton Chain wrote:
Toggle quote (15 lines)
>
> On Thu, 14 Nov 2024 07:40:05 +0800,
> Hilton Chain wrote:
> >
> > I thought Zig was using lld as a library for linking. Just looked at
> > src/link/Elf.zig, this is not true, it invokes ld.lld.
> >
> > I have more time today. I'll test this out and see if lld-as-ld-wrapper can
> > also add glibc RUNPATH.
>
> Zig manages linking to libc seperately, if this approach works, we have to
> keep validate-runpath? off for Zig. Reference to ld.lld can't be patched
> since it's both native-inputs (for building Zig) and inputs (for using the
> built Zig), maybe we can add a zig-toolchain (zig + lld-wrapper) later?

ld.lld is invoked in 'zig clang' route, sorry for the noise.

Currently I'm 1. modifying each-lib-rpath option of 'zig build'. 2. passing
libc to linker. I'll write details on this when succeed.
H
H
Hilton Chain wrote 13 hours ago
(address . 74217@debbugs.gnu.org)
87v7wqtbjq.wl-hako@ultrarare.space
On Thu, 14 Nov 2024 14:05:52 +0800,
Hilton Chain wrote:
Toggle quote (4 lines)
>
> Currently I'm 1. modifying each-lib-rpath option of 'zig build'. 2. passing
> libc to linker. I'll write details on this when succeed.

1. Modification about each-lib-rpath.
Toggle snippet (3 lines)
-feach-lib-rpath Ensure adding rpath for each used dynamic library

This option is on implicitly for native builds. This implicity is what our Zig
currently solely relies on.

I'm modifying it so that it's also on when CROSS_LIBRARY_PATH or LIBRARY_PATH is
set. This approach is better than my previous one since it only adds needed
libraries.


2. Pass libc to Zig's linker
This was the behavior in 0.9, but changed due to issue on macOS[1]. (btw, our
CPLUS_INCLUDE_PATH also has issue with macOS target[2]). RUNPATH for glibc was
missing because of this, since it's the linker handling each-lib-rpath.

Since we do not support macOS anyway, can we restore this behavior?


I also have concern for Zig's relying on /usr/bin/env (Zig uses an ELF file to
find dynamic linker, env is chosen for it's well-known). We have patched this
reference, not sure if it will cause issue for cross-building Zig.


Thanks
---
E
E
Efraim Flashner wrote 13 hours ago
(name . Hilton Chain)(address . hako@ultrarare.space)
ZzXFsMhXohkMutaw@3900XT
On Thu, Nov 14, 2024 at 05:22:17PM +0800, Hilton Chain wrote:
Toggle quote (26 lines)
> On Thu, 14 Nov 2024 14:05:52 +0800,
> Hilton Chain wrote:
> >
> > Currently I'm 1. modifying each-lib-rpath option of 'zig build'. 2. passing
> > libc to linker. I'll write details on this when succeed.
>
> 1. Modification about each-lib-rpath.
> --8<---------------cut here---------------start------------->8---
> -feach-lib-rpath Ensure adding rpath for each used dynamic library
> --8<---------------cut here---------------end--------------->8---
>
> This option is on implicitly for native builds. This implicity is what our Zig
> currently solely relies on.
>
> I'm modifying it so that it's also on when CROSS_LIBRARY_PATH or LIBRARY_PATH is
> set. This approach is better than my previous one since it only adds needed
> libraries.
>
>
> 2. Pass libc to Zig's linker
> This was the behavior in 0.9, but changed due to issue on macOS[1]. (btw, our
> CPLUS_INCLUDE_PATH also has issue with macOS target[2]). RUNPATH for glibc was
> missing because of this, since it's the linker handling each-lib-rpath.
>
> Since we do not support macOS anyway, can we restore this behavior?

At worst I could see adding a comment that it would likely break future
macOS cross-compiles. I don't see an issue with either putting it back
unconditionally or trying to make it conditional based on
(%current-system) or the contents of (%current-target-system).

Toggle quote (5 lines)
>
> I also have concern for Zig's relying on /usr/bin/env (Zig uses an ELF file to
> find dynamic linker, env is chosen for it's well-known). We have patched this
> reference, not sure if it will cause issue for cross-building Zig.

We use search-input-file in the replacement, so it should choose the
cross-binutils for the replacement /bin/env, so it shouldn't be a
problem.

Toggle quote (6 lines)
>
> Thanks
> ---
> [1]: https://github.com/ziglang/zig/issues/10765
> [2]: https://github.com/ziglang/zig/issues/18063

--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmc1xawACgkQQarn3Mo9
g1Hi9w/+MoKStnX0lS8NjHsUjTnNwMLx887hHQQXP/qYgWae/gb3bL4wCnxcfIek
BJ0T+dHaLG1T+YMxe91ElDh00aYo72IBaRPjL7fxjV15Frpg0WHW28wAXWYEmBE2
e//KDc04IyODHM36v6bU6V/oDeLL6h+Hc7Cr3j5l4LzEVFB2NHTKNSkfQ2g8sPx+
2rYp5hXfHcM8jWmpUgjE1bIx0G/GF7EVEN2mTYCTBXpdNO+jWG4Koob94+BSHwqJ
WOQcy0dT7bp7Ir6nm+ICu7FAVRPrgu1zlaDUv1QWQBwh7fS1E3jhYYWAUEgBrO8i
QuAYHrGvi0783XPDWuNwRltQV7CfwQpP/1w2Nay/3pi782oYEcRWmu7AttWTYNK2
XO6NKpK5c9MxNbI+dP3ppoToHsYOOUPO7Rbl4UDSnPnB+/auElVclObwdaiiymzo
CoC98hlKWqjwx5O8hrU57845X9rjZ/rnxfpfO21oJZyBP7RGX8503cvGKnLPn99b
8KUEkmI0TNiqcVLPuaBo3W5CSCBn0HWnUAsYb1td49sIu5phcENLuryOSairyfSU
bS5Ms5lJSttyQo1zB2+1VY+4SejWy43bsYD1djHH//ACs3ddgaVmYYV4ljCBpOS9
tZgojHKsQjKWUmQ8XO1lNKIZh3kIj7J2KvvbOEa3lsNFQOehbsA=
=C+D0
-----END PGP SIGNATURE-----


M
M
Motiejus Jakštys wrote 13 hours ago
(name . Hilton Chain)(address . hako@ultrarare.space)
CA+jRjx2GTNU3+Rw=aXFmq3f_K2TqKtw_i9ppy95oYdM9MV2o9g@mail.gmail.com
On Thu, Nov 14, 2024 at 11:22?AM Hilton Chain <hako@ultrarare.space> wrote:
Toggle quote (7 lines)
>
> On Thu, 14 Nov 2024 14:05:52 +0800,
> Hilton Chain wrote:
> I also have concern for Zig's relying on /usr/bin/env (Zig uses an ELF file to
> find dynamic linker, env is chosen for it's well-known). We have patched this
> reference, not sure if it will cause issue for cross-building Zig.

This file is only consulted when `-target=native`. I.e. when it needs
to compile for the host. If target is specified, it will not consult
that file.

Just verified with zig 0.13.0:

$ strace -f -e openat zig cc hello.c -o hello |& grep -w env
openat(AT_FDCWD,
"/nix/store/sf6y4arqcm100rnnl3dhpg732i774zp6-coreutils-9.5/bin/env",
O_RDONLY|O_NOCTTY|O_CLOEXEC) = 5
$ strace -f -e openat zig cc -target x86_64-linux-gnu.2.32 hello.c -o
hello |& grep -w env
$

Motiejus
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74217@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 74217
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch