Bootstrapping Zig with no Binary Blobs

  • Open
  • quality assurance status badge
Details
5 participants
  • Efraim Flashner
  • Ekaitz Zarraga
  • Hilton Chain
  • Motiejus Jakštys
  • Noé Lopez
Owner
Somebody
Submitted by
Ekaitz Zarraga
Severity
normal
E
E
Ekaitz Zarraga wrote on 5 Nov 22:47 +0100
(name . bug-guix@gnu.org)(address . bug-guix@gnu.org)
af98e6c6-57fe-4db0-af71-f7635c939884@elenq.tech
Hi,

In order to include modern versions of Zig (Zig 0.12+) in Guix, we need
to remove the binary blobs.

I open this issue to track this effort and store information about the
process.

Some Guix user is trying to achieve the same goal:


And discussing about it here:


We could use that effort as a reference and package it to Guix.
E
E
Ekaitz Zarraga wrote on 5 Nov 23:05 +0100
(no subject)
(name . control@debbugs.gnu.org)(address . control@debbugs.gnu.org)
66921ed7-ff38-4b4c-a35f-0dc5622c451f@elenq.tech
owner 74217 !
block 72386 by 74217
thanks
H
H
Hilton Chain wrote on 7 Nov 02:19 +0100
Re: Bootstrapping Zig with no Binary Blobs
(name . Ekaitz Zarraga)(address . ekaitz@elenq.tech)
87cyj7g7s4.wl-hako@ultrarare.space
Hi Ekaitz,

On Wed, 06 Nov 2024 05:47:48 +0800,
Ekaitz Zarraga wrote:
Toggle quote (17 lines)
>
> Hi,
>
> In order to include modern versions of Zig (Zig 0.12+) in Guix, we need to
> remove the binary blobs.
>
> I open this issue to track this effort and store information about the process.
>
> Some Guix user is trying to achieve the same goal:
>
> https://git.jakstys.lt/motiejus/zig-repro
>
> And discussing about it here:
>
> https://ziggit.dev/t/building-self-hosted-from-the-original-c-implementation/6607/11


Great news, thanks for sharing!


Toggle quote (3 lines)
> We could use that effort as a reference and package it to Guix.


Excited to know Zig developers are willing to help with bootstrapping! We
shouldn't miss this chance :)

I'll look into replicating the current progress this weekend. Anyone reading
this mail wants to join in?


Thanks
N
N
Noé Lopez wrote on 7 Nov 23:06 +0100
(address . 74217@debbugs.gnu.org)
87ldxuk8bn.fsf@xn--no-cja.eu
Toggle quote (6 lines)
>Excited to know Zig developers are willing to help with bootstrapping! We
>shouldn't miss this chance :)
>
>I'll look into replicating the current progress this weekend. Anyone reading
>this mail wants to join in?

Hey Hilton and Ekaitz,

I’m interested in this :) From what I can see the current effort in the
zig-repro repository is very well made and we should just need to
replicate each step with the correct dependencies, hoping they exist ?

I can try to make the first few steps as packages tomorrow, WDYT?

Good evening,
Noé
E
E
Ekaitz Zarraga wrote on 7 Nov 23:09 +0100
Re: bug#74217: Bootstrapping Zig with no Binary Blobs
(name . Hilton Chain)(address . hako@ultrarare.space)
844f9d54-ef96-47f6-950e-655a9ccc9d47@elenq.tech
Hi

Toggle quote (2 lines)
> I can try to make the first few steps as packages tomorrow, WDYT?

Sure!
Send the patches to this issue and I'll review when I have time.

Thanks for the energy!
H
H
Hilton Chain wrote on 8 Nov 18:43 +0100
[PATCH 0/2] Initial step on bootstrapping Zig.
(address . 74217@debbugs.gnu.org)(name . Hilton Chain)(address . hako@ultrarare.space)
cover.1731084096.git.hako@ultrarare.space
Finished step 00 from Motiejus Jakštys's script[1], this should make later
work on Guix side easier.


Hilton Chain (2):
gnu: Add zig-0.10.0-610-bootstrap.
gnu: Add zig-0.10.0-610.

gnu/local.mk | 1 +
...10.0-610-bootstrap-resolve-conflicts.patch | 87 ++++++++++
gnu/packages/zig.scm | 151 +++++++++++++++++-
3 files changed, 238 insertions(+), 1 deletion(-)
create mode 100644 gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch


base-commit: 2a6d96425eea57dc6dd48a2bec16743046e32e06
--
2.46.0
H
H
Hilton Chain wrote on 8 Nov 18:44 +0100
[PATCH 1/2] gnu: Add zig-0.10.0-610-bootstrap.
(address . 74217@debbugs.gnu.org)(name . Hilton Chain)(address . hako@ultrarare.space)
81bba1694a0f1ff48967727855b158487340deb9.1731084096.git.hako@ultrarare.space
* gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch: New
file.
* gnu/local.mk (dist_patch_DATA): Regisiter it.
* gnu/packages/zig.scm (zig-0.10.0-538-source,zig-0.10.0-539-patch)
(zig-0.10.0-542-patch,zig-0.10.0-610-bootstrap): New variables.

Change-Id: I132bbad34f40b919b4573e02d0f40eb4a007a26c
---
gnu/local.mk | 1 +
...10.0-610-bootstrap-resolve-conflicts.patch | 87 +++++++++++++++
gnu/packages/zig.scm | 105 +++++++++++++++++-
3 files changed, 192 insertions(+), 1 deletion(-)
create mode 100644 gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch

Toggle diff (234 lines)
diff --git a/gnu/local.mk b/gnu/local.mk
index ae902a5ab2..2cc2ea4c81 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -2358,6 +2358,7 @@ dist_patch_DATA = \
%D%/packages/patches/xygrib-newer-proj.patch \
%D%/packages/patches/yggdrasil-extra-config.patch \
%D%/packages/patches/zig-0.9-riscv-support.patch \
+ %D%/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch \
%D%/packages/patches/zig-use-baseline-cpu-by-default.patch \
%D%/packages/patches/zig-use-system-paths.patch \
%D%/packages/patches/zsh-egrep-failing-test.patch \
diff --git a/gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch b/gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch
new file mode 100644
index 0000000000..5ad5ffc249
--- /dev/null
+++ b/gnu/packages/patches/zig-0.10.0-610-bootstrap-resolve-conflicts.patch
@@ -0,0 +1,87 @@
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 1c03faf1e9..89406eb1b2 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -846,16 +846,17 @@ else()
+ endif()
+
+ set(ZIG_BUILD_ARGS
+- --zig-lib-dir "${CMAKE_SOURCE_DIR}/lib"
+- "-Dconfig_h=${ZIG_CONFIG_H_OUT}"
+- "-Denable-llvm"
+- ${ZIG_RELEASE_ARG}
+- ${ZIG_STATIC_ARG}
+- ${ZIG_NO_LIB_ARG}
+- ${ZIG_SINGLE_THREADED_ARG}
+- "-Dtarget=${ZIG_TARGET_TRIPLE}"
+- "-Dcpu=${ZIG_TARGET_MCPU}"
+- "-Dversion-string=${RESOLVED_ZIG_VERSION}"
++ --zig-lib-dir "${CMAKE_SOURCE_DIR}/lib"
++ "-Dconfig_h=${ZIG_CONFIG_H_OUT}"
++ "-Denable-llvm"
++ "-Denable-stage1"
++ ${ZIG_RELEASE_ARG}
++ ${ZIG_STATIC_ARG}
++ ${ZIG_NO_LIB_ARG}
++ ${ZIG_SINGLE_THREADED_ARG}
++ "-Dtarget=${ZIG_TARGET_TRIPLE}"
++ "-Dcpu=${ZIG_TARGET_MCPU}"
++ "-Dversion-string=${RESOLVED_ZIG_VERSION}"
+ )
+
+ add_custom_target(stage3 ALL
+diff --git a/build.zig b/build.zig
+index cf0e092326..7f80c3e1df 100644
+--- a/build.zig
++++ b/build.zig
+@@ -142,7 +142,8 @@ pub fn build(b: *Builder) !void {
+ const force_gpa = b.option(bool, "force-gpa", "Force the compiler to use GeneralPurposeAllocator") orelse false;
+ const link_libc = b.option(bool, "force-link-libc", "Force self-hosted compiler to link libc") orelse (enable_llvm or only_c);
+ const sanitize_thread = b.option(bool, "sanitize-thread", "Enable thread-sanitization") orelse false;
+- const strip = b.option(bool, "strip", "Omit debug information");
++ const strip = b.option(bool, "strip", "Omit debug information") orelse false;
++ const use_zig0 = b.option(bool, "zig0", "Bootstrap using zig0") orelse false;
+ const value_tracing = b.option(bool, "value-tracing", "Enable extra state tracking to help troubleshoot bugs in the compiler (using the std.debug.Trace API)") orelse false;
+
+ const mem_leak_frames: u32 = b.option(u32, "mem-leak-frames", "How many stack frames to print when a memory leak occurs. Tests get 2x this amount.") orelse blk: {
+@@ -151,7 +152,22 @@ pub fn build(b: *Builder) !void {
+ break :blk 4;
+ };
+
+- const exe = addCompilerStep(b);
++ if (only_c) {
++ target.ofmt = .c;
++ }
++
++ const main_file: ?[]const u8 = mf: {
++ if (!have_stage1) break :mf "src/main.zig";
++ if (use_zig0) break :mf null;
++ break :mf "src/stage1.zig";
++ };
++
++ const exe = b.addExecutable("zig", main_file);
++
++ const compile_step = b.step("compile", "Build the self-hosted compiler");
++ compile_step.dependOn(&exe.step);
++
++ exe.stack_size = stack_size;
+ exe.strip = strip;
+ exe.sanitize_thread = sanitize_thread;
+ exe.build_id = b.option(bool, "build-id", "Include a build id note") orelse false;
+diff --git a/src/translate_c/ast.zig b/src/translate_c/ast.zig
+index 20e4259725..bc0f002c21 100644
+--- a/src/translate_c/ast.zig
++++ b/src/translate_c/ast.zig
+@@ -1448,6 +1448,12 @@ fn renderNode(c: *Context, node: Node) Allocator.Error!NodeIndex {
+ .optional_type => return renderPrefixOp(c, node, .optional_type, .question_mark, "?"),
+ .address_of => {
+ const payload = node.castTag(.address_of).?.data;
++ if (c.zig_is_stage1 and payload.tag() == .fn_identifier)
++ return try c.addNode(.{
++ .tag = .identifier,
++ .main_token = try c.addIdentifier(payload.castTag(.fn_identifier).?.data),
++ .data = undefined,
++ });
+
+ const ampersand = try c.addToken(.ampersand, "&");
+ const base = if (payload.tag() == .fn_identifier)
diff --git a/gnu/packages/zig.scm b/gnu/packages/zig.scm
index 6994d48818..68907fd04e 100644
--- a/gnu/packages/zig.scm
+++ b/gnu/packages/zig.scm
@@ -23,13 +23,15 @@ (define-module (gnu packages zig)
#:use-module (guix gexp)
#:use-module (guix packages)
#:use-module (guix utils)
+ #:use-module (guix download)
#:use-module (guix git-download)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix build-system cmake)
#:use-module (gnu packages)
#:use-module (gnu packages compression)
#:use-module (gnu packages llvm)
- #:use-module (gnu packages llvm-meta))
+ #:use-module (gnu packages llvm-meta)
+ #:use-module (gnu packages web))
(define-public zig-0.9
(package
@@ -196,4 +198,105 @@ (define-public zig-0.10
(properties `((max-silent-time . 9600)
,@(clang-compiler-cpu-architectures "15")))))
+(define zig-0.10.0-538-source
+ ;; "std: added eql to DynamicBitSet and DynamicBitSetUnmanaged"
+ (let ((commit "bf316e550671cc71eb498b3cf799493627bb0fdc")
+ (revision "538"))
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/ziglang/zig")
+ (commit commit)))
+ (file-name (git-file-name "zig" (git-version "0.10" revision commit)))
+ (sha256
+ (base32 "1dchc2bp842jlw0byssqzindv8cigpqcj2hk3752667jrrww13vv")))))
+
+(define zig-0.10.0-539-patch
+ ;; "remove `-fstage1` option"
+ (let ((commit "28514476ef8c824c3d189d98f23d0f8d23e496ea"))
+ (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/ziglang/zig/commit/" commit ".patch"))
+ (sha256
+ (base32 "0qxxiafg2sd5rr4xhw0c12rygd7zh1rmf3x8hfialyxmsbi5pfxp")))))
+
+(define zig-0.10.0-542-patch
+ ;; "actually remove stage1"
+ (let ((commit "3ba916584db5485c38ebf2390e8d22bc6d81bf8e"))
+ (origin
+ (method url-fetch)
+ (uri (string-append
+ "https://github.com/ziglang/zig/commit/" commit ".patch"))
+ (sha256
+ (base32 "1l09gmbr3vqzinb63kvaskgs1d0mvm1m7w3ai3ngwg5zlabyya35")))))
+
+;; Build zig1.wasm from source.
+(define zig-0.10.0-610-bootstrap
+ (let ((commit "e7d28344fa3ee81d6ad7ca5ce1f83d50d8502118")
+ (revision "610"))
+ (package
+ (inherit zig-0.10)
+ (name "zig")
+ (version (git-version "0.10.0" revision commit))
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/ziglang/zig")
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (modules '((guix build utils)))
+ (snippet '(delete-file "stage1/zig1.wasm.zst"))
+ (sha256
+ (base32
+ "08pm3f4hh6djl3szhqgm7fa3qisdl2xh9jrp18m0z7bk2vd0bzw7"))))
+ (arguments
+ (substitute-keyword-arguments (package-arguments zig-0.10)
+ ((#:phases phases '%standard-phases)
+ #~(modify-phases #$phases
+ (add-after 'unpack 'prepare-source
+ (lambda* (#:key native-inputs inputs #:allow-other-keys)
+ (copy-recursively "." "../source-backup")
+ ;; Revert "actually remove stage1".
+ (invoke "patch" "--reverse" "--strip=1"
+ "--input" #+zig-0.10.0-542-patch)
+ ;; Revert "remove `-fstage1` option".
+ (false-if-exception
+ (invoke "patch" "--reverse" "--strip=1"
+ "--input" #+zig-0.10.0-539-patch))
+ ;; Resolve conflicts in previous patching.
+ (invoke
+ "patch" "--forward" "--strip=1" "--input"
+ #+(local-file
+ (search-patch
+ "zig-0.10.0-610-bootstrap-resolve-conflicts.patch")))
+ ;; Restore build system.
+ (rename-file "stage1/config.zig.in" "src/config.zig.in")
+ (substitute* "src/config.zig.in"
+ (("(have_stage1 = )false" _ prefix)
+ (string-append prefix "true")))
+ (for-each
+ (lambda (file)
+ (copy-file (string-append #+zig-0.10.0-538-source "/" file)
+ file))
+ '("build.zig" "CMakeLists.txt"))))
+ (add-after 'install 'build-zig1
+ (lambda _
+ (copy-recursively "../source-backup" ".")
+ (invoke (string-append #$output "/bin/zig")
+ "build" "update-zig1" "--verbose")))
+ (add-after 'build-zig1 'install-zig1
+ (lambda _
+ (install-file "stage1/zig1.wasm.zst"
+ (string-append #$output:zig1 "/bin"))))
+ (replace 'check
+ (lambda* (#:key tests? #:allow-other-keys)
+ (when tests?
+ (invoke (string-append #$output "/bin/zig")
+ "test" "-I" "test" "test/behavior.zig"))))))))
+ (native-inputs
+ (modify-inputs (package-native-inputs zig-0.10)
+ (prepend binaryen)))
+ (outputs '("out" "zig1")))))
+
(define-public zig zig-0.10)
--
2.46.0
H
H
Hilton Chain wrote on 8 Nov 18:44 +0100
[PATCH 2/2] gnu: Add zig-0.10.0-610.
(address . 74217@debbugs.gnu.org)(name . Hilton Chain)(address . hako@ultrarare.space)
91e88fbf01f28428bdf46464efd4912e3fd29f5d.1731084096.git.hako@ultrarare.space
* gnu/packages/zig.scm (zig-0.10.0-610): New variable.

Change-Id: I277a7f5e9781e89d7ad7cd108fec9afcf8cd23d9
---
gnu/packages/zig.scm | 46 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)

Toggle diff (57 lines)
diff --git a/gnu/packages/zig.scm b/gnu/packages/zig.scm
index 68907fd04e..4174dba38b 100644
--- a/gnu/packages/zig.scm
+++ b/gnu/packages/zig.scm
@@ -299,4 +299,50 @@ (define zig-0.10.0-610-bootstrap
(prepend binaryen)))
(outputs '("out" "zig1")))))
+;; Bootstrap with our zig1.wasm.
+(define zig-0.10.0-610
+ (let ((commit "e7d28344fa3ee81d6ad7ca5ce1f83d50d8502118")
+ (revision "610"))
+ (package
+ (inherit zig-0.10)
+ (name "zig")
+ (version (git-version "0.10.0" revision commit))
+ (source (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/ziglang/zig")
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (modules '((guix build utils)))
+ (snippet '(delete-file "stage1/zig1.wasm.zst"))
+ (sha256
+ (base32
+ "08pm3f4hh6djl3szhqgm7fa3qisdl2xh9jrp18m0z7bk2vd0bzw7"))))
+ (arguments
+ (substitute-keyword-arguments (package-arguments zig-0.10)
+ ((#:phases phases '%standard-phases)
+ #~(modify-phases #$phases
+ (add-after 'unpack 'unpack-zig1
+ (lambda* (#:key native-inputs inputs #:allow-other-keys)
+ (install-file (search-input-file
+ (or native-inputs inputs) "bin/zig1.wasm.zst")
+ "stage1")))
+ (add-after 'install 'update-zig1
+ (lambda _
+ (invoke (string-append #$output "/bin/zig")
+ "build" "update-zig1" "--verbose")))
+ (add-after 'update-zig1 'install-zig1
+ (lambda _
+ (install-file "stage1/zig1.wasm.zst"
+ (string-append #$output:zig1 "/bin"))))
+ (replace 'check
+ (lambda* (#:key tests? #:allow-other-keys)
+ (when tests?
+ (invoke (string-append #$output "/bin/zig")
+ "test" "-I" "test" "test/behavior.zig"))))))))
+ (native-inputs
+ (modify-inputs (package-native-inputs zig-0.10)
+ (prepend binaryen `(,zig-0.10.0-610-bootstrap "zig1"))))
+ (outputs '("out" "zig1")))))
+
(define-public zig zig-0.10)
--
2.46.0
H
H
Hilton Chain wrote on 9 Nov 18:26 +0100
Re: [PATCH 0/2] Initial step on bootstrapping Zig.
(address . 74217@debbugs.gnu.org)
871pzk1fpe.wl-hako@ultrarare.space
I have added a wip-zig-bootstrap[1] branch.

Variables are not exported, you can build them locally with the following
command:
Toggle snippet (9 lines)
ZIG=$(
sed 's/^\(define(-public)? (zig-.*-.*)/\2/p' gnu/packages/zig.scm \
--quiet --regexp-extended |
tail --lines=1
)

./pre-inst-env guix build --expression="(@@ (gnu packages zig) $ZIG)"

E
E
Efraim Flashner wrote on 11 Nov 12:42 +0100
Re: Bootstrapping Zig with no Binary Blobs
(name . Hilton Chain)(address . hako@ultrarare.space)
ZzHtoV-P_QaxhSH5@3900XT
On Thu, Nov 07, 2024 at 09:19:23AM +0800, Hilton Chain wrote:
Toggle quote (33 lines)
> Hi Ekaitz,
>
> On Wed, 06 Nov 2024 05:47:48 +0800,
> Ekaitz Zarraga wrote:
> >
> > Hi,
> >
> > In order to include modern versions of Zig (Zig 0.12+) in Guix, we need to
> > remove the binary blobs.
> >
> > I open this issue to track this effort and store information about the process.
> >
> > Some Guix user is trying to achieve the same goal:
> >
> > https://git.jakstys.lt/motiejus/zig-repro
> >
> > And discussing about it here:
> >
> > https://ziggit.dev/t/building-self-hosted-from-the-original-c-implementation/6607/11
>
>
> Great news, thanks for sharing!
>
>
> > We could use that effort as a reference and package it to Guix.
>
>
> Excited to know Zig developers are willing to help with bootstrapping! We
> shouldn't miss this chance :)
>
> I'll look into replicating the current progress this weekend. Anyone reading
> this mail wants to join in?

Ok, you've convinced me. I guess I finally need to get zig@0.10
building correctly on riscv64.

Expect plenty of FIXUP and SQUASH commits in the future.

--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmcx7Z4ACgkQQarn3Mo9
g1H3VhAAjSvSv881uk3LrTDMsa8YOOkwHPMnxydZwefpzCr9pL1UAUyw0ZRO+GTk
tGmJt4yyA9vCJ/+bkQSnF8J8ttgTrTlgxNs7TGT/NlJM49WaqOIN+VYOEl51+Xaz
cfoYYitDxmSmZY2i9PE+SEiJudfCiJVuZrNsk2Yll2JzTAxYDSllO58xIVXVkkr5
Gkt+t7DWyuBo/LUCAmJY7BrA7yj6rzn3IDiCFPR8K4X6VsQwZa/LeoqzgwGfx+Id
jHBJV6zWmk1cm29aWBtwXSo5pmC0xenOtzOKlDpzbeFCF+Q3/QhhxpaJHErLQBOs
Rw71VJe8yNYEhZVD0XvoNFQ2MtCOv0etIXvyUKQXrmBA1sTF8RVfoJHU6xbHlKfb
t/Xpc/s1aDCNwDmEE3XpsqngtpgIL2xVZW299CdceWvi+AzvWiNrTDSBZ/hmTpCH
/nesxa5Wkc57+ECr0crt/2tOJbckoP7xQYGl4T7Q8eycqR3FWl4vJd+ct1PdZNgt
Y4MnByHkSrX1ilJH8vAUBHKAOm7tesFTabL8T6c+or8i4kY3jnL1PNI2FBi+t8Nr
GCQevedgyRHWYzaY/6B0sRVWlijdus0pNVXlF4uFgNkmrImUt4MJV1CBxzQzyh9D
5E2DHYp07opDQxPc58eMOcNgg5bB7doQGX3H1iX8tH50/PV/Oyc=
=Fjzq
-----END PGP SIGNATURE-----


H
H
Hilton Chain wrote on 11 Nov 12:56 +0100
(name . Efraim Flashner)(address . efraim@flashner.co.il)
87v7wu7zm9.wl-hako@ultrarare.space
Hi Efraim,

On Mon, 11 Nov 2024 19:42:25 +0800,
Efraim Flashner wrote:
Toggle quote (6 lines)
>
> Ok, you've convinced me. I guess I finally need to get zig@0.10
> building correctly on riscv64.
>
> Expect plenty of FIXUP and SQUASH commits in the future.

I have something to fix with a force-push on wip-zig-bootstrap, I'll reply here
when the branch is ready to be worked on :)
E
E
Efraim Flashner wrote on 11 Nov 13:02 +0100
(name . Hilton Chain)(address . hako@ultrarare.space)
ZzHyQOFBBiT2Rd3W@3900XT
On Mon, Nov 11, 2024 at 07:56:30PM +0800, Hilton Chain wrote:
Toggle quote (13 lines)
> Hi Efraim,
>
> On Mon, 11 Nov 2024 19:42:25 +0800,
> Efraim Flashner wrote:
> >
> > Ok, you've convinced me. I guess I finally need to get zig@0.10
> > building correctly on riscv64.
> >
> > Expect plenty of FIXUP and SQUASH commits in the future.
>
> I have something to fix with a force-push on wip-zig-bootstrap, I'll reply here
> when the branch is ready to be worked on :)

No worries. I'm still getting a look around the branch. And if you
force-push then my regular push will be rejected and I can rebase
whatever I have.

--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmcx8jsACgkQQarn3Mo9
g1GS3g/+JNYMc3E6h7EzJLixWVCpmJasMW2b+EFhs1LdlaNIK8c4ds/XpPle3Z4T
yYIoVQAClnabQqrTWgdNGsq2PQjfp/esB4bg4zFaV0ODG9o7gQMJ9K5Um5SQUoSL
z+hUOR8yrdW6dB3cSUXraTx1w0ToIoBl6H2QUpA+Lz8BSBb7PaiXv/AC27KP3t+g
j5CNvCbQvDacYNCvYUm+k9s0GHffYsFMuftpLg0VzRo84FAw92hs9MvC9wYJJ/Pl
YkoOTOiXnLXdJyPMX5acAGE61aFU6re/91uImWXSKWGOR4hZJ6DlY7+yMw9uA53K
WBWDfnrAe7XHeb4PPQRze2MIDsnVYEtbsi//vSYdFZHpqi2MOc9mAfm8vFZWeNjJ
kp0m6/413sFMc5i0Mjdd8X7foScKnvCKmtheO60ZB1fu/cL3Svfkhx9jq6ofE2Q1
zrkPhgP3+AdULqVctTw1M/BYngmaKuVObUWCdM8u095tMu50GEsPQ1ETTUZa0ikj
1NoOmMV5m2chQHj/qazuYhEib4sFwibdbOaBPLzcyB+wt/Kq0E6ymUb9jsgjmxbE
oi6DQRaIUk2I7P03XvtXqmTy8HqPvZvSyrCRkGJK6YIFLwXJFn9nkFjnxdAwkH7B
ZJaalCo93jdO2LrOjZH8j94ISTvrWNZpgGaXHWbJVgp7K+QB/MM=
=t5x9
-----END PGP SIGNATURE-----


H
H
Hilton Chain wrote on 13 Nov 17:46 +0100
(address . 74217@debbugs.gnu.org)
87ldxngjy4.wl-hako@ultrarare.space
Hello everyone,

With a new forced push (and new rebuilds, sorry), wip-zig-bootstrap is mostly
ready (for me) now. Please take a look at the first few commits, as I'm
changing Zig's behavior there, here are some additional notes:

To Efraim: Can adding pkg-config to native-inputs avoid the ncdu snippet?

I'm building this branch on my personal Cuirass instance[1][2], for x86_64-linux
and aarch64-linux (qemu-binfmt), in previous revisions I have indentified
reproduciblity issue, and aarch64 builds timed-out. I haven't investigated them
yet.

Then for what's the RUNPATH issue I have mentioned in commits:
+ For current zig@0.10 on Guix master: glibc is missing from RUNPATH, which
fails the validate-runpath check.
+ For zig@0.11, some other inputs are missing, making the binary failing to run on Guix[3].
+ dan also mentioned privately to me that they needed to add paths from
LIBRARY_PATH to RUNPATH for their own projects, so programs built by Zig is also
affected.

I think this is due to Zig's implemention of its own linking logic, which
bypasses our ld-wrapper.

I'm not going to implement ld-wrapper within Zig. :) So my proposed workaround
in wip-zig-bootstrap is to patch the handling logic added for Guix:

(In lib/std/zig/system/NativePaths.zig)
Toggle snippet (31 lines)
// Distros like guix don't use FHS, so they rely on environment
// variables to search for headers and libraries.
// We use os.getenv here since this part won't be executed on
// windows, to get rid of unnecessary error handling.
- if (std.posix.getenv("C_INCLUDE_PATH")) |c_include_path| {
+ if (std.posix.getenv("CROSS_C_INCLUDE_PATH") orelse std.posix.getenv("C_INCLUDE_PATH")) |c_include_path| {
var it = mem.tokenizeScalar(u8, c_include_path, ':');
while (it.next()) |dir| {
try self.addIncludeDir(dir);
}
}

- if (std.posix.getenv("CPLUS_INCLUDE_PATH")) |cplus_include_path| {
+ if (std.posix.getenv("CROSS_CPLUS_INCLUDE_PATH") orelse std.posix.getenv("CPLUS_INCLUDE_PATH")) |cplus_include_path| {
var it = mem.tokenizeScalar(u8, cplus_include_path, ':');
while (it.next()) |dir| {
try self.addIncludeDir(dir);
}
}

- if (std.posix.getenv("LIBRARY_PATH")) |library_path| {
+ if (std.posix.getenv("CROSS_LIBRARY_PATH") orelse std.posix.getenv("LIBRARY_PATH")) |library_path| {
var it = mem.tokenizeScalar(u8, library_path, ':');
while (it.next()) |dir| {
try self.addLibDir(dir);
+ try self.addRPath(dir);
}
}
}

Adding directories from CROSS_LIBRARY_PATH or LIBRARY_PATH to RUNPATH, "CROSS_"
part is for our cross toolchain, I haven't tested it yet.

I think this behavior change is reasonable since the search path
(CROSS_)?LIBRARY_PATH is only automatically set by our compilers.

I added this change to 0.9 as well to make all Zigs behave consistently. I also
used shrink-runpath phase from meson-build-system in Zig and zig-build-system.

I want to move shrink-runpath to (guix build utils) and export it too, so that
it can be used easier. But I'm not sure if this change will trigger rebuilds of
other packages, so I didn't do it.

Thanks to Guile, for builds not managed by guix-daemon, something like the
following script can be used, we can ship a program-file if we agree on this
workaround.
Toggle snippet (15 lines)
(use-modules (guix build meson-build-system))

(define shrink-runpath
(assoc-ref %standard-phases 'shrink-runpath))

(define (main directories)
(for-each (lambda (dir)
(false-if-exception
(shrink-runpath
#:elf-directories '(".")
#:outputs `(("out" . ,dir)))))
directories))

(main (cdr (command-line)))
Usage: guile <file-with-above-content> DIRECTORY...


Thanks
---
[2]: https://substitute.boiledscript.com, if you want to challenge it.
E
E
Efraim Flashner wrote on 13 Nov 19:10 +0100
(name . Hilton Chain)(address . hako@ultrarare.space)
ZzTrpMLuGg9cr70s@3900XT
Attachment: file
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmc066AACgkQQarn3Mo9
g1EnFQ/7BUV3KU9BDhqjdcN/3UHzaNFP0+zfL1eMsFykYB55ebJ+KXuSyB5pQSpI
QtXHeIKR0uk+WUfnSVZNvAz3LvwOFK84X5g7mpKiQJT2AacvkYzvU9ZzDEu9uRqW
D5LfqZNkbVirAWm3ObafPAbRUE1W7TQpgXyzQpFTuBrPEPzFEXYcnKMR29JW73GI
nvD4xlE25xNSQLp66U4QN9CciAehh2YDE/2VGkFIW6xObdmPPAl708gi2WTuW78V
4cqmikiq3ZuJSp9gC1JIKDrlOPVGQluNy8sAI58uyHkLqWF0xh3z6qS/6hGWwtgW
/AjlNmcAhC/0GkknG/AdEsxs2tYqH9Bbsc4TLHOpGl7wXjrGKV5UgjgarviMBcPA
DLr7VjKOOlHPy5tOUMsy0xDL2y1WGt1yhO/Hkumwui8Hd5Cka83rXPy1BkNIWlA0
B8A0g68GVYV9Uu7BfUk5+tmW7dXML8c/1M7tMJIkdCaV7z8WeP1b0QKcMUkO61Sz
1H9VZldUTOQjx8RfXv3Gwl/SDl6uOQibp0qMgGXIVKjgJwuEqwE7LMVOTvgWsQ3T
Ax14pycXy5qaGLOe1lYATr4PCbMsp6eN59bmtiuiKg4YaqFrOaUfmNvQgoX/8jHL
A+1JXT2cPLMXLmkZ+s9RkmARY3qMkJ2JwAwSxp3s6Hr0MRn1kQA=
=+4fE
-----END PGP SIGNATURE-----


H
H
Hilton Chain wrote on 14 Nov 00:40 +0100
878qtmvh2i.wl-hako@ultrarare.space
On Thu, 14 Nov 2024 02:10:44 +0800,
Efraim Flashner wrote:
Toggle quote (7 lines)
>
> > I think this is due to Zig's implemention of its own linking logic, which
> > bypasses our ld-wrapper.
>
> I wonder if switching from lld to make-lld-wrapper would make a
> difference here.

Thanks for mentioning! This should do the work!

I thought Zig was using lld as a library for linking. Just looked at
src/link/Elf.zig, this is not true, it invokes ld.lld.

I have more time today. I'll test this out and see if lld-as-ld-wrapper can
also add glibc RUNPATH.
H
H
Hilton Chain wrote on 14 Nov 02:05 +0100
(address . 74217@debbugs.gnu.org)
877c96vd3u.wl-hako@ultrarare.space
On Thu, 14 Nov 2024 07:40:05 +0800,
Hilton Chain wrote:
Toggle quote (18 lines)
>
> On Thu, 14 Nov 2024 02:10:44 +0800,
> Efraim Flashner wrote:
> >
> > > I think this is due to Zig's implemention of its own linking logic, which
> > > bypasses our ld-wrapper.
> >
> > I wonder if switching from lld to make-lld-wrapper would make a
> > difference here.
>
> Thanks for mentioning! This should do the work!
>
> I thought Zig was using lld as a library for linking. Just looked at
> src/link/Elf.zig, this is not true, it invokes ld.lld.
>
> I have more time today. I'll test this out and see if lld-as-ld-wrapper can
> also add glibc RUNPATH.

Zig manages linking to libc seperately, if this approach works, we have to
keep validate-runpath? off for Zig. Reference to ld.lld can't be patched
since it's both native-inputs (for building Zig) and inputs (for using the
built Zig), maybe we can add a zig-toolchain (zig + lld-wrapper) later?
H
H
Hilton Chain wrote on 14 Nov 07:05 +0100
(address . 74217@debbugs.gnu.org)
874j4auz7j.wl-hako@ultrarare.space
On Thu, 14 Nov 2024 09:05:41 +0800,
Hilton Chain wrote:
Toggle quote (15 lines)
>
> On Thu, 14 Nov 2024 07:40:05 +0800,
> Hilton Chain wrote:
> >
> > I thought Zig was using lld as a library for linking. Just looked at
> > src/link/Elf.zig, this is not true, it invokes ld.lld.
> >
> > I have more time today. I'll test this out and see if lld-as-ld-wrapper can
> > also add glibc RUNPATH.
>
> Zig manages linking to libc seperately, if this approach works, we have to
> keep validate-runpath? off for Zig. Reference to ld.lld can't be patched
> since it's both native-inputs (for building Zig) and inputs (for using the
> built Zig), maybe we can add a zig-toolchain (zig + lld-wrapper) later?

ld.lld is invoked in 'zig clang' route, sorry for the noise.

Currently I'm 1. modifying each-lib-rpath option of 'zig build'. 2. passing
libc to linker. I'll write details on this when succeed.
H
H
Hilton Chain wrote on 14 Nov 10:22 +0100
(address . 74217@debbugs.gnu.org)
87v7wqtbjq.wl-hako@ultrarare.space
On Thu, 14 Nov 2024 14:05:52 +0800,
Hilton Chain wrote:
Toggle quote (4 lines)
>
> Currently I'm 1. modifying each-lib-rpath option of 'zig build'. 2. passing
> libc to linker. I'll write details on this when succeed.

1. Modification about each-lib-rpath.
Toggle snippet (3 lines)
-feach-lib-rpath Ensure adding rpath for each used dynamic library

This option is on implicitly for native builds. This implicity is what our Zig
currently solely relies on.

I'm modifying it so that it's also on when CROSS_LIBRARY_PATH or LIBRARY_PATH is
set. This approach is better than my previous one since it only adds needed
libraries.


2. Pass libc to Zig's linker
This was the behavior in 0.9, but changed due to issue on macOS[1]. (btw, our
CPLUS_INCLUDE_PATH also has issue with macOS target[2]). RUNPATH for glibc was
missing because of this, since it's the linker handling each-lib-rpath.

Since we do not support macOS anyway, can we restore this behavior?


I also have concern for Zig's relying on /usr/bin/env (Zig uses an ELF file to
find dynamic linker, env is chosen for it's well-known). We have patched this
reference, not sure if it will cause issue for cross-building Zig.


Thanks
---
E
E
Efraim Flashner wrote on 14 Nov 10:41 +0100
(name . Hilton Chain)(address . hako@ultrarare.space)
ZzXFsMhXohkMutaw@3900XT
On Thu, Nov 14, 2024 at 05:22:17PM +0800, Hilton Chain wrote:
Toggle quote (26 lines)
> On Thu, 14 Nov 2024 14:05:52 +0800,
> Hilton Chain wrote:
> >
> > Currently I'm 1. modifying each-lib-rpath option of 'zig build'. 2. passing
> > libc to linker. I'll write details on this when succeed.
>
> 1. Modification about each-lib-rpath.
> --8<---------------cut here---------------start------------->8---
> -feach-lib-rpath Ensure adding rpath for each used dynamic library
> --8<---------------cut here---------------end--------------->8---
>
> This option is on implicitly for native builds. This implicity is what our Zig
> currently solely relies on.
>
> I'm modifying it so that it's also on when CROSS_LIBRARY_PATH or LIBRARY_PATH is
> set. This approach is better than my previous one since it only adds needed
> libraries.
>
>
> 2. Pass libc to Zig's linker
> This was the behavior in 0.9, but changed due to issue on macOS[1]. (btw, our
> CPLUS_INCLUDE_PATH also has issue with macOS target[2]). RUNPATH for glibc was
> missing because of this, since it's the linker handling each-lib-rpath.
>
> Since we do not support macOS anyway, can we restore this behavior?

At worst I could see adding a comment that it would likely break future
macOS cross-compiles. I don't see an issue with either putting it back
unconditionally or trying to make it conditional based on
(%current-system) or the contents of (%current-target-system).

Toggle quote (5 lines)
>
> I also have concern for Zig's relying on /usr/bin/env (Zig uses an ELF file to
> find dynamic linker, env is chosen for it's well-known). We have patched this
> reference, not sure if it will cause issue for cross-building Zig.

We use search-input-file in the replacement, so it should choose the
cross-binutils for the replacement /bin/env, so it shouldn't be a
problem.

Toggle quote (6 lines)
>
> Thanks
> ---
> [1]: https://github.com/ziglang/zig/issues/10765
> [2]: https://github.com/ziglang/zig/issues/18063

--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmc1xawACgkQQarn3Mo9
g1Hi9w/+MoKStnX0lS8NjHsUjTnNwMLx887hHQQXP/qYgWae/gb3bL4wCnxcfIek
BJ0T+dHaLG1T+YMxe91ElDh00aYo72IBaRPjL7fxjV15Frpg0WHW28wAXWYEmBE2
e//KDc04IyODHM36v6bU6V/oDeLL6h+Hc7Cr3j5l4LzEVFB2NHTKNSkfQ2g8sPx+
2rYp5hXfHcM8jWmpUgjE1bIx0G/GF7EVEN2mTYCTBXpdNO+jWG4Koob94+BSHwqJ
WOQcy0dT7bp7Ir6nm+ICu7FAVRPrgu1zlaDUv1QWQBwh7fS1E3jhYYWAUEgBrO8i
QuAYHrGvi0783XPDWuNwRltQV7CfwQpP/1w2Nay/3pi782oYEcRWmu7AttWTYNK2
XO6NKpK5c9MxNbI+dP3ppoToHsYOOUPO7Rbl4UDSnPnB+/auElVclObwdaiiymzo
CoC98hlKWqjwx5O8hrU57845X9rjZ/rnxfpfO21oJZyBP7RGX8503cvGKnLPn99b
8KUEkmI0TNiqcVLPuaBo3W5CSCBn0HWnUAsYb1td49sIu5phcENLuryOSairyfSU
bS5Ms5lJSttyQo1zB2+1VY+4SejWy43bsYD1djHH//ACs3ddgaVmYYV4ljCBpOS9
tZgojHKsQjKWUmQ8XO1lNKIZh3kIj7J2KvvbOEa3lsNFQOehbsA=
=C+D0
-----END PGP SIGNATURE-----


M
M
Motiejus Jakštys wrote on 14 Nov 10:47 +0100
(name . Hilton Chain)(address . hako@ultrarare.space)
CA+jRjx2GTNU3+Rw=aXFmq3f_K2TqKtw_i9ppy95oYdM9MV2o9g@mail.gmail.com
On Thu, Nov 14, 2024 at 11:22?AM Hilton Chain <hako@ultrarare.space> wrote:
Toggle quote (7 lines)
>
> On Thu, 14 Nov 2024 14:05:52 +0800,
> Hilton Chain wrote:
> I also have concern for Zig's relying on /usr/bin/env (Zig uses an ELF file to
> find dynamic linker, env is chosen for it's well-known). We have patched this
> reference, not sure if it will cause issue for cross-building Zig.

This file is only consulted when `-target=native`. I.e. when it needs
to compile for the host. If target is specified, it will not consult
that file.

Just verified with zig 0.13.0:

$ strace -f -e openat zig cc hello.c -o hello |& grep -w env
openat(AT_FDCWD,
"/nix/store/sf6y4arqcm100rnnl3dhpg732i774zp6-coreutils-9.5/bin/env",
O_RDONLY|O_NOCTTY|O_CLOEXEC) = 5
$ strace -f -e openat zig cc -target x86_64-linux-gnu.2.32 hello.c -o
hello |& grep -w env
$

Motiejus
H
H
Hilton Chain wrote 6 days ago
(address . 74217@debbugs.gnu.org)
878qtl5g55.wl-hako@ultrarare.space
On Thu, 14 Nov 2024 17:41:04 +0800,
Efraim Flashner wrote:
Toggle quote (15 lines)
>
> [1 <text/plain; utf-8 (quoted-printable)>]
> On Thu, Nov 14, 2024 at 05:22:17PM +0800, Hilton Chain wrote:
> > 2. Pass libc to Zig's linker
> > This was the behavior in 0.9, but changed due to issue on macOS[1]. (btw, our
> > CPLUS_INCLUDE_PATH also has issue with macOS target[2]). RUNPATH for glibc was
> > missing because of this, since it's the linker handling each-lib-rpath.
> >
> > Since we do not support macOS anyway, can we restore this behavior?
>
> At worst I could see adding a comment that it would likely break future
> macOS cross-compiles. I don't see an issue with either putting it back
> unconditionally or trying to make it conditional based on
> (%current-system) or the contents of (%current-target-system).

Bad news: Zig 0.12 changed[1] behavior of each_lib_rpath, it won't filter
libraries now.

Good news: Thanks to this diff, I know how to add libc to RUNPATH now :)

Another forced push, I have ensured consistent behavior for
(CROSS_)?LIBRARY_PATH and added libc RUNPATH without restoring the behavior
passing '-lc' to linker.

Who said not going to implement a ld-wrapper within Zig? :P
Fortunately it was already there :)

BTW, adding pkg-config to native-inputs works for ncdu.

Toggle quote (9 lines)
> > I also have concern for Zig's relying on /usr/bin/env (Zig uses an ELF file to
> > find dynamic linker, env is chosen for it's well-known). We have patched this
> > reference, not sure if it will cause issue for cross-building Zig.
>
> We use search-input-file in the replacement, so it should choose the
> cross-binutils for the replacement /bin/env, so it shouldn't be a
> problem.


On Thu, 14 Nov 2024 17:47:23 +0800,
Motiejus Jakštys wrote:
Toggle quote (15 lines)
>
> This file is only consulted when `-target=native`. I.e. when it needs
> to compile for the host. If target is specified, it will not consult
> that file.
>
> Just verified with zig 0.13.0:
>
> $ strace -f -e openat zig cc hello.c -o hello |& grep -w env
> openat(AT_FDCWD,
> "/nix/store/sf6y4arqcm100rnnl3dhpg732i774zp6-coreutils-9.5/bin/env",
> O_RDONLY|O_NOCTTY|O_CLOEXEC) = 5
> $ strace -f -e openat zig cc -target x86_64-linux-gnu.2.32 hello.c -o
> hello |& grep -w env
> $

Thanks! Then it should be in inputs. I don't want to add coreutils to it, so I
patched reference of /usr/bin/env to clang++ (it's both in inputs and RUNPATH)
instead.

---
H
H
Hilton Chain wrote 6 days ago
(address . 74217@debbugs.gnu.org)
87jzd47enj.wl-hako@ultrarare.space
On Fri, 15 Nov 2024 11:29:10 +0800,
Hilton Chain wrote:
Toggle quote (12 lines)
>
> Good news: Thanks to this diff, I know how to add libc to RUNPATH now :)
>
> Another forced push, I have ensured consistent behavior for
> (CROSS_)?LIBRARY_PATH and added libc RUNPATH without restoring the behavior
> passing '-lc' to linker.
>
> Who said not going to implement a ld-wrapper within Zig? :P
> Fortunately it was already there :)
>
> BTW, adding pkg-config to native-inputs works for ncdu.

I have locally made the "use-system-paths" patch larger so that Zig can really
honor "CROSS_" environment variables.

The next issue is cross building with pkg-config. Zig only invokes
"pkg-config", but we don't have a "pkg-config" with search path for target
inputs. I can add a pkg-config-for-zig to workaround this, and then... It's
dynamic linker path, I'll look into it soon.

Also for reproducibility, bin/zig is the only file differs and here's the diff,
I don't know about this part so I currently have no idea on fixing it.
Toggle snippet (22 lines)
--- /gnu/store/gqdi4drfn3js5cwgfmlpkyfm2xf3l5b0-zig-0.10.1/bin/zig
+++ cuirass/gqdi4drfn3js5cwgfmlpkyfm2xf3l5b0-zig-0.10.1/bin/zig
??? readelf --wide --decompress --string-dump=.rodata {}
? @@ -77024,14 +77024,16 @@
? [149be0] +?&
? [149bf9] )&
? [149c12] %
? [149c28] VO$
? [149c40] D?(
? [149c59] >$
? [149c70] 8?%
? + [149c94] ;
? + [149ca0] ;
? [149ca8] '
? [149cb0] uespemos?odnarodarenegylsetybdet
? [149cf0] p??
? [149d11] O'
? [149d21] 5&
? [149d31] f&
? [149d40] SJ'

Toggle snippet (21 lines)
--- /gnu/store/466cm9xpjqg80iqracj4qirsrdha1rnk-zig-0.11.0/bin/zig
+++ cuirass/466cm9xpjqg80iqracj4qirsrdha1rnk-zig-0.11.0/bin/zig
??? readelf --wide --decompress --string-dump=.rodata {}
? @@ -64905,14 +64905,16 @@
? [ 5ae48] xpnt4win2kvistawin10ws2003win8_1win10_th2win10_rs1win10_rs2win10_rs3win10_rs4win10_rs5win10_19h1
? [ 5aef0]
? [ 5aef8] #
? [ 5af00] %
? [ 5af08] %
? [ 5af10] &
? [ 5af78] celfhexrawmachospirvdxcontainer
? + [ 5afc0] ;
? + [ 5aff8] ;
? [ 5b070] E
? [ 5b0b4] ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_
? [ 5b350]
? [ 5b380] @
? [ 5b3e0]
? [ 5b420] ]
? [ 5b5e0] %
H
H
Hilton Chain wrote 5 days ago
(address . 74217@debbugs.gnu.org)
87iksn7jnk.wl-hako@ultrarare.space
On Fri, 15 Nov 2024 22:30:40 +0800,
Hilton Chain wrote:
Toggle quote (9 lines)
>
> I have locally made the "use-system-paths" patch larger so that Zig can really
> honor "CROSS_" environment variables.
>
> The next issue is cross building with pkg-config. Zig only invokes
> "pkg-config", but we don't have a "pkg-config" with search path for target
> inputs. I can add a pkg-config-for-zig to workaround this, and then... It's
> dynamic linker path, I'll look into it soon.

Adding a file with content like the following and passing --libc <this file> to
zig works, RUNPATH is correct and no need to set CC then.

Toggle snippet (8 lines)
include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
sys_include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
crt_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/lib
msvc_lib_dir=
kernel32_lib_dir=
gcc_dir=

For cross builds interpreter path like /lib/ld-linux-aarch64.so.1 is used in
output binary, I'll find a way to fix it.

Toggle quote (3 lines)
> Also for reproducibility, bin/zig is the only file differs and here's the diff,
> I don't know about this part so I currently have no idea on fixing it.

This seem to be an upstream issue, Zig is reproducible only on the same machine.
I'll verify it and report to upstream.
M
M
Motiejus Jakštys wrote 5 days ago
(name . Hilton Chain)(address . hako@ultrarare.space)
CA+jRjx2OxbzATu3=ORp_Z31xvEzykuXpxfwUhj5VU=CwqHD2yA@mail.gmail.com
On Sat, Nov 16, 2024 at 8:55?AM Hilton Chain <hako@ultrarare.space> wrote:
Toggle quote (10 lines)
>
> On Fri, 15 Nov 2024 22:30:40 +0800,
> Hilton Chain wrote:
> >
> > Also for reproducibility, bin/zig is the only file differs and here's the diff,
> > I don't know about this part so I currently have no idea on fixing it.
>
> This seem to be an upstream issue, Zig is reproducible only on the same machine.
> I'll verify it and report to upstream.

Zig defaults to `-march=native` when building for the host. Try
ZIG_TARGET_MCPU=baseline when building stage3.

Motiejus
H
H
Hilton Chain wrote 5 days ago
(name . Motiejus Jakštys)(address . motiejus@jakstys.lt)
87frnr7ijm.wl-hako@ultrarare.space
On Sat, 16 Nov 2024 15:13:40 +0800,
Motiejus Jakštys wrote:
Toggle quote (15 lines)
>
> On Sat, Nov 16, 2024 at 8:55?AM Hilton Chain <hako@ultrarare.space> wrote:
> >
> > On Fri, 15 Nov 2024 22:30:40 +0800,
> > Hilton Chain wrote:
> > >
> > > Also for reproducibility, bin/zig is the only file differs and here's the diff,
> > > I don't know about this part so I currently have no idea on fixing it.
> >
> > This seem to be an upstream issue, Zig is reproducible only on the same machine.
> > I'll verify it and report to upstream.
>
> Zig defaults to `-march=native` when building for the host. Try
> ZIG_TARGET_MCPU=baseline when building stage3.

Yes, ZIG_TARGET_MCPU=baseline is passed to cmake.
E
E
Efraim Flashner wrote 5 days ago
(name . Hilton Chain)(address . hako@ultrarare.space)
ZzjQf-YCLUUT2Nnk@3900XT
On Sat, Nov 16, 2024 at 02:54:55PM +0800, Hilton Chain wrote:
Toggle quote (11 lines)
> On Fri, 15 Nov 2024 22:30:40 +0800,
> Hilton Chain wrote:
> >
> > I have locally made the "use-system-paths" patch larger so that Zig can really
> > honor "CROSS_" environment variables.
> >
> > The next issue is cross building with pkg-config. Zig only invokes
> > "pkg-config", but we don't have a "pkg-config" with search path for target
> > inputs. I can add a pkg-config-for-zig to workaround this, and then... It's
> > dynamic linker path, I'll look into it soon.

I tried adding pkg-config-for-build as a work-around but it wasn't
enough without touching the zig compiler's source too, which I didn't
attempt yesterday.

Toggle quote (12 lines)
> Adding a file with content like the following and passing --libc <this file> to
> zig works, RUNPATH is correct and no need to set CC then.
>
> --8<---------------cut here---------------start------------->8---
> include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
> sys_include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
> crt_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/lib
> msvc_lib_dir=
> kernel32_lib_dir=
> gcc_dir=
> --8<---------------cut here---------------end--------------->8---

Is this the layout of the file expected? That doesn't look too hard to
create in the build-system if necessary.

Toggle quote (3 lines)
> For cross builds interpreter path like /lib/ld-linux-aarch64.so.1 is used in
> output binary, I'll find a way to fix it.

I was going to say to take a look at gcc-2.95, where we point all the
linkers for all the architectures to whatever the target architecture
is, but that won't work here since we have 1 zig binary and it can
compile for any architecture.

I'm going to suggest against adding a cross-libc for all the different
architectures as an input, that would be crazy.

(ins)efraim@3900XT ~/workspace/zig$ git grep 'ld-linux-aarch64.so.1'
lib/libc/include/aarch64-linux-gnu/gnu/lib-names-lp64.h:#define LD_LINUX_AARCH64_SO "ld-linux-aarch64.so.1"
lib/libc/include/aarch64-linux-gnu/gnu/lib-names-lp64.h:#define LD_SO "ld-linux-aarch64.so.1"
lib/std/Build.zig:/// that contains the path `aarch64-linux-gnu/lib/ld-linux-aarch64.so.1`.
lib/std/Target.zig: .aarch64 => init("/lib/ld-linux-aarch64.so.1"),
Binary file stage1/zig1.wasm matches

Would it be possible to change the init("/path/to/ld.so") part to the
zig equivalent of (search-input-file inputs "/path/to/ld.so"), and then
when it is used from Guix the cross-libc will already be in the PATH and
therefore findable from zig's search through the vector¹ of the paths
inside PATH?

¹ I know it's not true but in my mind a vector and an array are the same
thing.

Toggle quote (6 lines)
> > Also for reproducibility, bin/zig is the only file differs and here's the diff,
> > I don't know about this part so I currently have no idea on fixing it.
>
> This seem to be an upstream issue, Zig is reproducible only on the same machine.
> I'll verify it and report to upstream.

--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmc40HwACgkQQarn3Mo9
g1FDKBAAsOjAtHrgqW3V3e2O+kURryk4zOC+D7b18vVDbz4vygQnJ5ysG2lAZyXl
8X/C9Yzffu5LAwXZbW2cS+HZXi+URXLxzfXwXblxGhGd0C6Qc/FK5N6K+loCnG7p
R628weaFJuvQuGouu+0oa99wgmQjAYBYq5n+laPCvw6BOx5lAIilBjJcPk6Qf892
PId6I0P6TPs+MxRrWuKpVpMPh2/Su5PnH10qcHW3zSAQ81z08lvYGSBQB1y1ZLik
2/obfqjHhVz7HV4WTN/ysyqkot0JH5r1tG0RWwB0IGooX8audRG6anZiyHzKYpp8
h45JVbeIAgiFL20ka92GS4nXGxDeToJEDCpPFctDWCvnHwbYXa2v9aHVx+liJyTw
+WAwMKKQ713LaNQVMaFvlSg5frTKLwqzw6r7MIVMZqakm4z9HbqH2i+uYPkNAwWc
CMHvWEidwA7+KNQ7cAlb1Fx79e/dt7nopNw4Tcmqw8QT/uPd6sgg2kOjlWKmFDTY
mgSWa0g5BwSLmQjuVUae7SkveOG5F2fu6QIymwQnvyXJqvNR8DK3SyCj/HiGYEP/
OcEvhnH000QUEiXp/DNdFH5cBrqFmzfxhhz5eEE3I87rJUb0UnR5sgCR4G75M/8b
0vheaTxUnSy0QRe1iu9ev5kEcdkTfpJRRz1+S170Oi2rX2lgLiA=
=Fqzp
-----END PGP SIGNATURE-----


H
H
Hilton Chain wrote 5 days ago
87ed3b6m39.wl-hako@ultrarare.space
On Sun, 17 Nov 2024 01:03:59 +0800,
Efraim Flashner wrote:
Toggle quote (60 lines)
>
> [1 <text/plain; utf-8 (quoted-printable)>]
> On Sat, Nov 16, 2024 at 02:54:55PM +0800, Hilton Chain wrote:
> > On Fri, 15 Nov 2024 22:30:40 +0800,
> > Hilton Chain wrote:
> > >
> > > I have locally made the "use-system-paths" patch larger so that Zig can really
> > > honor "CROSS_" environment variables.
> > >
> > > The next issue is cross building with pkg-config. Zig only invokes
> > > "pkg-config", but we don't have a "pkg-config" with search path for target
> > > inputs. I can add a pkg-config-for-zig to workaround this, and then... It's
> > > dynamic linker path, I'll look into it soon.
>
> I tried adding pkg-config-for-build as a work-around but it wasn't
> enough without touching the zig compiler's source too, which I didn't
> attempt yesterday.
>
> > Adding a file with content like the following and passing --libc <this file> to
> > zig works, RUNPATH is correct and no need to set CC then.
> >
> > --8<---------------cut here---------------start------------->8---
> > include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
> > sys_include_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/include
> > crt_dir=/gnu/store/dfx90sc16nphh6bd07sjyri6x4s51zni-glibc-cross-aarch64-linux-gnu-2.39/lib
> > msvc_lib_dir=
> > kernel32_lib_dir=
> > gcc_dir=
> > --8<---------------cut here---------------end--------------->8---
>
> Is this the layout of the file expected? That doesn't look too hard to
> create in the build-system if necessary.
>
> > For cross builds interpreter path like /lib/ld-linux-aarch64.so.1 is used in
> > output binary, I'll find a way to fix it.
>
> I was going to say to take a look at gcc-2.95, where we point all the
> linkers for all the architectures to whatever the target architecture
> is, but that won't work here since we have 1 zig binary and it can
> compile for any architecture.
>
> I'm going to suggest against adding a cross-libc for all the different
> architectures as an input, that would be crazy.
>
> (ins)efraim@3900XT ~/workspace/zig$ git grep 'ld-linux-aarch64.so.1'
> lib/libc/include/aarch64-linux-gnu/gnu/lib-names-lp64.h:#define LD_LINUX_AARCH64_SO "ld-linux-aarch64.so.1"
> lib/libc/include/aarch64-linux-gnu/gnu/lib-names-lp64.h:#define LD_SO "ld-linux-aarch64.so.1"
> lib/std/Build.zig:/// that contains the path `aarch64-linux-gnu/lib/ld-linux-aarch64.so.1`.
> lib/std/Target.zig: .aarch64 => init("/lib/ld-linux-aarch64.so.1"),
> Binary file stage1/zig1.wasm matches
>
> Would it be possible to change the init("/path/to/ld.so") part to the
> zig equivalent of (search-input-file inputs "/path/to/ld.so"), and then
> when it is used from Guix the cross-libc will already be in the PATH and
> therefore findable from zig's search through the vector¹ of the paths
> inside PATH?
>
> ¹ I know it's not true but in my mind a vector and an array are the same
> thing.

I have added a GUIX_ZIG_LIBC_DIR environment variable, to be set as output path
of cross-libc or libc by zig-build-system, patched Zig to search it and
concatenate it with "/lib/ld...".

Also added the file for --libc option in zig-build-system. Cross compilation is
available now. (only available in 0.12 for now, I'll port the patches to other
versions when I get up.)
H
H
Hilton Chain wrote 4 days ago
(address . 74217@debbugs.gnu.org)
87bjye7i6b.wl-hako@ultrarare.space
On Sat, 16 Nov 2024 14:54:55 +0800,
Hilton Chain wrote:
Toggle quote (7 lines)
>
> > Also for reproducibility, bin/zig is the only file differs and here's the diff,
> > I don't know about this part so I currently have no idea on fixing it.
>
> This seem to be an upstream issue, Zig is reproducible only on the same machine.
> I'll verify it and report to upstream.

E
E
Efraim Flashner wrote 4 days ago
(name . Hilton Chain)(address . hako@ultrarare.space)
ZzmYPYaie5Av09gX@3900XT
On Fri, Nov 15, 2024 at 10:30:40PM +0800, Hilton Chain wrote:
Toggle quote (22 lines)
> On Fri, 15 Nov 2024 11:29:10 +0800,
> Hilton Chain wrote:
> >
> > Good news: Thanks to this diff, I know how to add libc to RUNPATH now :)
> >
> > Another forced push, I have ensured consistent behavior for
> > (CROSS_)?LIBRARY_PATH and added libc RUNPATH without restoring the behavior
> > passing '-lc' to linker.
> >
> > Who said not going to implement a ld-wrapper within Zig? :P
> > Fortunately it was already there :)
> >
> > BTW, adding pkg-config to native-inputs works for ncdu.
>
> I have locally made the "use-system-paths" patch larger so that Zig can really
> honor "CROSS_" environment variables.
>
> The next issue is cross building with pkg-config. Zig only invokes
> "pkg-config", but we don't have a "pkg-config" with search path for target
> inputs. I can add a pkg-config-for-zig to workaround this, and then... It's
> dynamic linker path, I'll look into it soon.

I found a patch after the 0.13.0 release that switches from hardcoding
pkg-config to using the PKG_CONFIG environment variable and falling back
to pkg-config, so I backported it to 0.12 and was able to use that and
guix's regular pkg-config package. I've added those patches to the
wip-zig-bootstrap tree.

We now have a couple of phases that are before the 'build phase, do you
think it'd be better to consolidate them into a 'configure phase?
There's no 'configure' script to run, but it does do a lot of
preparation before the actual 'build phase...

--
Efraim Flashner <efraim@flashner.co.il> ????? ?????
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmc5mDoACgkQQarn3Mo9
g1EjVg//ZNxsbIucSXqCBp2nFQQdw5hBU8FA4jrS5AsZ2ZYBxVBV70/h99EX1AkJ
LLyOvBlA0dmK8pQQRtzmVJSk/fNoTW3nmJ3VwsT3NZtNG94+CZcLnIzxtJLcFqBu
hpC2+v8s1r70HnHUVZyd/jNo6w8JHouk0X8TywFC1C0mNqPlxckRjYhb7SyBbogQ
N0v2LdFOFf8D5bZofe6bnTYpNEP+JLsx68IiU8ujQMiCOSYrQvEZJVxRoJ3ZWXao
sQKGCwY0te53rZJR3R/fr2YYHhB7VRCYkCQa/m67ZWsdur/L2yQdCoVNq0UYSKOv
eYGbtXL7T+LY1gtzF/jR8PWidIvnMCQ7qocOrElZdbqWvq9IHPU5t4TwYRaRbwNA
htg1BQQRYx1pXMCG7w/BvQh/8UmmPnK1xF6jk2RCFBbDn1P+UYhnZ4GmAP2oOVfe
OuKwn5WSVpBl7B3pAgHiluKDE0H4GFFn3Wlupd2Dy8yxDrdt+OBrG1a9V4OxE//s
k+tCkuIDEsj+hdPVIA5LJ4bzzldqOcAPZV2pUCusBBBwtxh2jdFyGScCmYfaAc2v
ns71zbHBHn2ptG0bN0cfEXpqtgI/tsszhVe74AqtBq95SwoyTRu7rOoH7GIpNvui
WZ1iVo+CP3OpuGeUvyMJIoEHqje8Y5OUdPK5tjF2f4jgTOswYIk=
=xUP5
-----END PGP SIGNATURE-----


H
H
Hilton Chain wrote 4 days ago
(address . 74217@debbugs.gnu.org)
87a5dx7w1n.wl-hako@ultrarare.space
On Sun, 17 Nov 2024 15:16:13 +0800,
Efraim Flashner wrote:
Toggle quote (31 lines)
>
> [1 <text/plain; utf-8 (quoted-printable)>]
> On Fri, Nov 15, 2024 at 10:30:40PM +0800, Hilton Chain wrote:
> > On Fri, 15 Nov 2024 11:29:10 +0800,
> > Hilton Chain wrote:
> > >
> > > Good news: Thanks to this diff, I know how to add libc to RUNPATH now :)
> > >
> > > Another forced push, I have ensured consistent behavior for
> > > (CROSS_)?LIBRARY_PATH and added libc RUNPATH without restoring the behavior
> > > passing '-lc' to linker.
> > >
> > > Who said not going to implement a ld-wrapper within Zig? :P
> > > Fortunately it was already there :)
> > >
> > > BTW, adding pkg-config to native-inputs works for ncdu.
> >
> > I have locally made the "use-system-paths" patch larger so that Zig can really
> > honor "CROSS_" environment variables.
> >
> > The next issue is cross building with pkg-config. Zig only invokes
> > "pkg-config", but we don't have a "pkg-config" with search path for target
> > inputs. I can add a pkg-config-for-zig to workaround this, and then... It's
> > dynamic linker path, I'll look into it soon.
>
> I found a patch after the 0.13.0 release that switches from hardcoding
> pkg-config to using the PKG_CONFIG environment variable and falling back
> to pkg-config, so I backported it to 0.12 and was able to use that and
> guix's regular pkg-config package. I've added those patches to the
> wip-zig-bootstrap tree.

Thanks, I have ported all patches and pushed. GUIX_ZIG_LIBC_DIR is changed to
GUIX_ZIG_GLIBC_LINKER and is set as full path in Guix side because I don't want
mess with strings in Zig side...

Toggle quote (5 lines)
> We now have a couple of phases that are before the 'build phase, do you
> think it'd be better to consolidate them into a 'configure phase?
> There's no 'configure' script to run, but it does do a lot of
> preparation before the actual 'build phase...

I have merged these phases into configure, forgot to change commit log though.


The reproducibility issue is related to kernel version from target ("native" by
default) information, to address this we need to specify a target for native
builds too.[1]

---
H
H
Hilton Chain wrote 3 days ago
(address . 74217@debbugs.gnu.org)
87a5dwsqex.wl-hako@ultrarare.space
On Sun, 17 Nov 2024 22:51:48 +0800,
Hilton Chain wrote:
Toggle quote (5 lines)
>
> The reproducibility issue is related to kernel version from target ("native" by
> default) information, to address this we need to specify a target for native
> builds too.

Specifying a target makes Zig behave like in cross-compilation, fortunately
workaround for zig-build-system still applies here too.

Marked the commit as DRAFT since I'm still building it.
H
H
Hilton Chain wrote 45 hours ago
(address . 74217@debbugs.gnu.org)
87frnns6wh.wl-hako@ultrarare.space
On Sun, 17 Nov 2024 22:51:48 +0800,
Hilton Chain wrote:
Toggle quote (5 lines)
>
> Thanks, I have ported all patches and pushed. GUIX_ZIG_LIBC_DIR is changed to
> GUIX_ZIG_GLIBC_LINKER and is set as full path in Guix side because I don't want
> mess with strings in Zig side...

Reworked this, patched Zig to search dynamic linker in CROSS_LIBRARY_PATH or
LIBRARY_PATH (added in the use-system-paths patch), no extra environment
variable introduced now.

Toggle quote (7 lines)
> > We now have a couple of phases that are before the 'build phase, do you
> > think it'd be better to consolidate them into a 'configure phase?
> > There's no 'configure' script to run, but it does do a lot of
> > preparation before the actual 'build phase...
>
> I have merged these phases into configure, forgot to change commit log though.

Moved zig-target and zig-build-system's configure phase to (guix build
zig-utils) to avoid dependency issue, as building Zig now uses them too.

Removed nonfree files (IETF RFC documents) from Zig source.

I'll take a look at Zig package manager support.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74217@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 74217
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch