Guix Inferiors: Curious incorrect derivation output bug

DoneSubmitted by Carl Dong.
Details
2 participants
  • Carl Dong
  • Ludovic Courtès
Owner
unassigned
Severity
important
C
C
Carl Dong wrote on 24 Jul 2019 03:09
(name . bug-guix@gnu.org)(address . bug-guix@gnu.org)
TifHDa-L4paOnZQruSSv3YhpIdqMvwlk7pwnElq9azsMZwEdFv3-0usXgSQkV4nIOuKoJzv-BwIurwdecWYX-BTGUn2xZXAqQzefQp85W7M=@carldong.me
Hi all,
I've been working on improving the Guix build support on Bitcoin Core so thatit'll be ready to use for official releases.
One of the things that I wanted to do was to use a combination of channels andinferiors to have reproducible builds across time. I have most of it set up, butam running into some trouble. Here are the details:
I have a Guix channel set up at https://github.com/dongcarl/bitcoin-guix,whereI pin the Guix version to 6869b6635afd93872b8f0d9f2db0db4c0d765a86 in the.guix-channel, and declare all my packages in packages.scm.
I have my manifest.scm here:https://github.com/dongcarl/bitcoin/blob/2019-06-guix-channels-and-inferiors/contrib/guix/manifest.scm,and it references the aforementioned Guix channel.
What I expect to happen is that now when I change my default profile's Guixversion by 'guix pull'ing, it won't affect the environment that's generated bythe manifest.scm. I've tested this with differing versions of Guix as my defaultprofile, and this seems to work with 7304d5623ab5cc35289cb1535cbd0d8a37691facand 7f1c69f5d32bee6b8b6b902a9ce445e04aa9d07d being my default profile. However,I tried an older version b6dc08393e6a8313b88ce422fc3c1e4e9c0efc6f, and got thefollowing error:
```guix environment: error: derivation `/gnu/store/r641vpqc9rfjhljf7rzfzwmkmpa642ls-info-dir.drv' has incorrect output `/gnu/store/q9hkdidycz3wq28xxgjq47bzx5s39k52-info-dir', should be `/gnu/store/z5hh2nl0h58b9f6hdxfwm00gjyxfcc3n-info-dir'```
I remember my previous attempts at integrating channels and inferiors into ourworkflow yielded a similar result, so I would like to know if this is a bug, andhow I might go about fixing this so that I can feel comfortable using inferiorsfor our builds.
Nevertheless, thank you all for your hard work. The fact that we have inferiorsat all is marvelous!
Cheers,Carl Dongcontact@carldong.me"I fight for the users"
L
L
Ludovic Courtès wrote on 26 Jul 2019 09:50
(name . Carl Dong)(address . contact@carldong.me)(address . 36777@debbugs.gnu.org)
875znps08a.fsf@gnu.org
Hello Carl,
Carl Dong <contact@carldong.me> skribis:
Toggle quote (4 lines)> I have my manifest.scm here:> https://github.com/dongcarl/bitcoin/blob/2019-06-guix-channels-and-inferiors/contrib/guix/manifest.scm,> and it references the aforementioned Guix channel.
This one appears to work for me (I interrupted it before it was donecompiling all the toolchains, though.)
Toggle quote (8 lines)> What I expect to happen is that now when I change my default profile's Guix> version by 'guix pull'ing, it won't affect the environment that's generated by> the manifest.scm. I've tested this with differing versions of Guix as my default> profile, and this seems to work with 7304d5623ab5cc35289cb1535cbd0d8a37691fac> and 7f1c69f5d32bee6b8b6b902a9ce445e04aa9d07d being my default profile. However,> I tried an older version b6dc08393e6a8313b88ce422fc3c1e4e9c0efc6f, and got the> following error:
When I put b6dc08393e6a8313b88ce422fc3c1e4e9c0efc6f in .guix-channel anduse that as my channel, it also works fine (well, I commented out thetoolchains as well.)
Toggle quote (4 lines)> ```> guix environment: error: derivation `/gnu/store/r641vpqc9rfjhljf7rzfzwmkmpa642ls-info-dir.drv' has incorrect output `/gnu/store/q9hkdidycz3wq28xxgjq47bzx5s39k52-info-dir', should be `/gnu/store/z5hh2nl0h58b9f6hdxfwm00gjyxfcc3n-info-dir'> ```
That definitely looks like a bug.
Could you send the faulty info-dir.drv file?
Thanks,Ludo’.
C
C
Carl Dong wrote on 26 Jul 2019 21:58
(name . Ludovic Courtès)(address . ludo@gnu.org)(name . 36777@debbugs.gnu.org)(address . 36777@debbugs.gnu.org)
1WSz93jYl9XxYywM_b-gQYV4SiIG9VyTXLYpj4bueFcBZByRMIdXQNnVY4YDnbH_hPe4y_ZDjl7lIgyvynH8J85NtO93namyeEVMU7Lj9WY=@carldong.me
Hi all,
I did some more digging, and have included a git-bisect log, the -info-dir.drv,and -info-dir-builder here:
https://gist.github.com/dongcarl/0a305badf20c9b5cfae738147ca416af
Please let me know if I can provide more information.
Cheers,Carl Dongcontact@carldong.me"I fight for the users"
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐On Friday, July 26, 2019 7:50 AM, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (32 lines)> Hello Carl,>> Carl Dong contact@carldong.me skribis:>> > I have my manifest.scm here:> > https://github.com/dongcarl/bitcoin/blob/2019-06-guix-channels-and-inferiors/contrib/guix/manifest.scm,> > and it references the aforementioned Guix channel.>> This one appears to work for me (I interrupted it before it was done> compiling all the toolchains, though.)>> > What I expect to happen is that now when I change my default profile's Guix> > version by 'guix pull'ing, it won't affect the environment that's generated by> > the manifest.scm. I've tested this with differing versions of Guix as my default> > profile, and this seems to work with 7304d5623ab5cc35289cb1535cbd0d8a37691fac> > and 7f1c69f5d32bee6b8b6b902a9ce445e04aa9d07d being my default profile. However,> > I tried an older version b6dc08393e6a8313b88ce422fc3c1e4e9c0efc6f, and got the> > following error:>> When I put b6dc08393e6a8313b88ce422fc3c1e4e9c0efc6f in .guix-channel and> use that as my channel, it also works fine (well, I commented out the> toolchains as well.)>> > guix environment: error: derivation `/gnu/store/r641vpqc9rfjhljf7rzfzwmkmpa642ls-info-dir.drv' has incorrect output `/gnu/store/q9hkdidycz3wq28xxgjq47bzx5s39k52-info-dir', should be `/gnu/store/z5hh2nl0h58b9f6hdxfwm00gjyxfcc3n-info-dir'> >>> That definitely looks like a bug.>> Could you send the faulty info-dir.drv file?>> Thanks,> Ludo’.
L
L
Ludovic Courtès wrote on 27 Jul 2019 23:30
(name . Carl Dong)(address . contact@carldong.me)(name . 36777@debbugs.gnu.org)(address . 36777@debbugs.gnu.org)
87wog3i2sj.fsf@gnu.org
Hi Carl,
Carl Dong <contact@carldong.me> skribis:
Toggle quote (5 lines)> I did some more digging, and have included a git-bisect log, the -info-dir.drv,> and -info-dir-builder here:>> https://gist.github.com/dongcarl/0a305badf20c9b5cfae738147ca416af
For future reference, the bisect output is:
# first bad commit: [5cf4b26d52bcea382d98fb4becce89be9ee37b55] derivations: <derivation-input> now aggregates a <derivation>.
and ‘info-dir.drv’ is attached below.
I looked at the offending commit. I don’t doubt it has the potential tointroduce such a bug :-), but so far I haven’t seen anything fishy as Iinspected it with a fresh eye.
Could you come up with a reduced reproducer for this? Ideally a smallmanifest, the commit of the Guix you’re using (not just that of theinferior), maybe a manifest that doesn’t even use inferiors, and so on.
Alternately, since the problem is that the output path of the derivationis incorrectly computed, you could add ‘pk’ calls to print the value of‘drv-masked’ in (guix derivations) or similar. That’s not going to beconvenient, though. :-/
Ludo’.
PS: I’ll be away from keyboard so I won’t be able to look into it until some time.
Derive([("out","/gnu/store/q9hkdidycz3wq28xxgjq47bzx5s39k52-info-dir","","")],[("/gnu/store/1lyb9v2fnbflccrawnzjw2n87ycaicia-glibc-cross-riscv64-linux-gnu-2.27.drv",["out","static"]),("/gnu/store/3wxvpcvxr5qgs2i1wx0cqyspv3in1kws-linux-libre-headers-cross-riscv64-linux-gnu-4.19.59.drv",["out"]),("/gnu/store/4d4si2aiviy0ldihj7ymjkxrvx6c7rvq-i686-linux-gnu-toolchain-9.1.0.drv",["out"]),("/gnu/store/4imc4bigz499427hqd2yvz38y361vq5k-gcc-cross-arm-linux-gnueabihf-9.1.0.drv",["out"]),("/gnu/store/5i49zz5l4dzh59fi2b8fiq2llw0rkn8s-python-3.7.0.drv",["out"]),("/gnu/store/60x9nffmmdahdfmd0argjkfcvkzcm4c3-linux-libre-headers-cross-x86_64-linux-gnu-4.19.59.drv",["out"]),("/gnu/store/7s7lml5cgign328i6nk13fxz0r2vykng-binutils-cross-x86_64-linux-gnu-2.31.1.drv",["out"]),("/gnu/store/7zc3ms7wxjg9ypg7r4p23ni18f43m6p1-linux-libre-headers-cross-aarch64-linux-gnu-4.19.59.drv",["out"]),("/gnu/store/82r3931cfjm1864mc81wzzlfmb5rlgx1-util-linux-2.32.1.drv",["out"]),("/gnu/store/8wgjmm3cks3xaabhm3m7w25i9wwx50z2-sed-4.5.drv",["out"]),("/gnu/store/9dgy50xx4d9f2pnh4s0pdla7n7j61mjg-findutils-4.6.0.drv",["out"]),("/gnu/store/arvm0v20n93mra1s2la59by3kbm3cbsd-guile-2.2.4.drv",["out"]),("/gnu/store/c0ngi4rm0avjlp3nb8j4yvmdlf598vr2-gzip-1.9.drv",["out"]),("/gnu/store/c0pcf8c405my5g7nsc99gxpq66hzynn8-gcc-cross-riscv64-linux-gnu-9.1.0.drv",["out"]),("/gnu/store/c59fjany683kns33dhb44hfc598654xh-libtool-2.4.6.drv",["out"]),("/gnu/store/d6inhd29h4mp3zfah4dq6mh0hi1bwsd6-gcc-glibc-2.27-toolchain-9.1.0.drv",["out"]),("/gnu/store/dwjr2d9ds00bg8dvmsmljygz6k9y6b5h-tar-1.30.drv",["out"]),("/gnu/store/fgv06c04vkf52s8idc5jm6c10nsa2ph1-bash-4.4.23.drv",["out"]),("/gnu/store/gj3b2r65jfgrz83b6nni44028xciakfy-libfaketime-0.9.7.drv",["out"]),("/gnu/store/iddbw29p9fq2ccygwhyvfc5r7yq0qlyr-grep-3.1.drv",["out"]),("/gnu/store/iv231dbdxg300rk9lqmgkdm855hdxzgr-binutils-cross-aarch64-linux-gnu-2.31.1.drv",["out"]),("/gnu/store/jdzrl9rrdxvyg7y639igda091w1aja1b-module-import-compiled.drv",["out"]),("/gnu/store/k9y9vjpv62nvksm5pgbxfy2ddgkrmwmv-linux-libre-headers-cross-i686-linux-gnu-4.19.59.drv",["out"]),("/gnu/store/kx4f618mcrmkizp5gxyb00fwn1hp4cbw-gcc-cross-aarch64-linux-gnu-9.1.0.drv",["out"]),("/gnu/store/ljjw2dnwpdq73fg3rjh58yw27qalyssc-linux-libre-headers-cross-arm-linux-gnueabihf-4.19.59.drv",["out"]),("/gnu/store/lm5x5sz3s5h4drn0n2l0xrbnqr9xyqzd-gcc-cross-x86_64-linux-gnu-9.1.0.drv",["out"]),("/gnu/store/m3fqaak2rc24ic0aisbl1fdafy14fb45-gcc-cross-i686-linux-gnu-9.1.0.drv",["out"]),("/gnu/store/maiy0a31cqlsls1g139hikwabv7wwvbh-zlib-1.2.11.drv",["out"]),("/gnu/store/n05hlghlgw64ljc8hgbimbajnlapssiw-glibc-cross-aarch64-linux-gnu-2.27.drv",["out","static"]),("/gnu/store/n8a9mj131ddip42cmcci3s12a3y01zj3-texinfo-6.5.drv",["out"]),("/gnu/store/nas9cw7lk7737ha4icmdw5yhxx4j9453-which-2.21.drv",["out"]),("/gnu/store/ngj6zhpajm7693ys419yxmkdpcqwkzg9-tcsh-6.20.00.drv",["out"]),("/gnu/store/nz7vl4xn2s3xwx2f7vlvhl8sp9h4lkgx-glibc-utf8-locales-2.28.drv",["out"]),("/gnu/store/nzq7zmb44hn9wzm997pn0cb0aak7nmxq-binutils-cross-arm-linux-gnueabihf-2.31.1.drv",["out"]),("/gnu/store/ph6l1b0ixbvghvs41dpm06wqf4pw5677-glibc-cross-i686-linux-gnu-2.27.drv",["out","static"]),("/gnu/store/pjnibkx89y2nq3r9wb6dkscdb9rs915z-pkg-config-0.29.2.drv",["out"]),("/gnu/store/pwfrz20z33lhgkvy3y1s9gm2ijwcacrk-coreutils-8.30.drv",["out"]),("/gnu/store/q0l8y15xksvrm6fpn6miivk1dy88ka1a-riscv64-linux-gnu-toolchain-9.1.0.drv",["out"]),("/gnu/store/q1v24z3zfnym7fqbbh4z4jcybaaiwzlv-m4-1.4.18.drv",["out"]),("/gnu/store/q24ic25gr3azaxlabh9pj842kybs8h4m-file-5.33.drv",["out"]),("/gnu/store/qk22gsq1vrkzhrpmmlr2084f4vvlsyg4-make-4.2.1.drv",["out"]),("/gnu/store/qm501xmx0p6h2qyk2jfg6i35n2g2iblw-autoconf-2.69.drv",["out"]),("/gnu/store/qxs84gr9g0vinyq433ym226d54nr668a-diffutils-3.6.drv",["out"]),("/gnu/store/r3ih0hhv74bp4qkg6zfi6wd4bqzlq7df-perl-5.28.0.drv",["out"]),("/gnu/store/rmqarqss73akz9yaad4a02zdvp62ijx2-gzip-1.9.drv",["out"]),("/gnu/store/rwz69w3nnbxvzwnkrrhq1q5fqf1d6rm1-glibc-cross-x86_64-linux-gnu-2.27.drv",["out","static"]),("/gnu/store/s5ajxc1mc2j6njkjs2bj6ssfzb43bvan-arm-linux-gnueabihf-toolchain-9.1.0.drv",["out"]),("/gnu/store/s66bj62iwlh9fndcyv06z997zp6ga7fy-aarch64-linux-gnu-toolchain-9.1.0.drv",["out"]),("/gnu/store/vi1jc3vkpw13nigqzw7iiqqcq9c1xinz-glibc-cross-arm-linux-gnueabihf-2.27.drv",["out","static"]),("/gnu/store/vxh3p0y6v577gzfp016v1xz5xs5lg47m-x86_64-linux-gnu-toolchain-9.1.0.drv",["out"]),("/gnu/store/w1wcggyikprn54y2c2hf1wk9bkn9ryja-automake-1.16.1.drv",["out"]),("/gnu/store/wan77da4mrf8x2fwc0yqndpz6v0fwqja-gawk-4.2.1.drv",["out"]),("/gnu/store/wcnmh5krgmz8a9x59r2gzpd81ybn5b7v-xz-5.2.4.drv",["out"]),("/gnu/store/wgfz82qfkh9vgqlpa5icj3bzqclyg8rj-bzip2-1.0.6.drv",["out"]),("/gnu/store/x9kynl44rpqxd78vbv8j2z58j1vmar0r-binutils-cross-i686-linux-gnu-2.31.1.drv",["out"]),("/gnu/store/xk4qgfc77rd4k4c863fxsyxkxfi9c0ja-binutils-cross-riscv64-linux-gnu-2.31.1.drv",["out"]),("/gnu/store/yvm7rgcskmgsjx6wnxx2gjp71gz6ghjh-patch-2.7.6.drv",["out"])],["/gnu/store/ak3y80w6xjr9fq22bw8d577zkm9qq005-info-dir-builder","/gnu/store/p5q59xqmhwg60y2437fnny76nw7l6gbz-module-import"],"x86_64-linux","/gnu/store/r658y3cgpnf99nxjxqgjiaizx20ac4k0-guile-2.2.4/bin/guile",["--no-auto-compile","-L","/gnu/store/p5q59xqmhwg60y2437fnny76nw7l6gbz-module-import","-C","/gnu/store/syp239hi3xjgxdm8r4hvhhzh67i5ixgl-module-import-compiled","/gnu/store/ak3y80w6xjr9fq22bw8d577zkm9qq005-info-dir-builder"],[("allowSubstitutes","0"),("guix properties","((type . profile-hook) (hook . info-dir))"),("out","/gnu/store/q9hkdidycz3wq28xxgjq47bzx5s39k52-info-dir"),("preferLocalBuild","1")])
L
L
Ludovic Courtès wrote on 27 Jul 2019 23:48
(name . Carl Dong)(address . contact@carldong.me)(name . 36777@debbugs.gnu.org)(address . 36777@debbugs.gnu.org)
87r26bi1x2.fsf@gnu.org
Ludovic Courtès <ludo@gnu.org> skribis:
Toggle quote (4 lines)> For future reference, the bisect output is:>> # first bad commit: [5cf4b26d52bcea382d98fb4becce89be9ee37b55] derivations: <derivation-input> now aggregates a <derivation>.
Does the patch below have any effect?
My understanding is that it shouldn’t have any effect (because inputshave already been coalesced at that point), but that’s one place where5cf4b26d52bcea382d98fb4becce89be9ee37b55 introduced a difference.
Thanks in advance,Ludo’.
Toggle diff (33 lines)diff --git a/guix/derivations.scm b/guix/derivations.scmindex 92d50503ce..eb94fea55e 100644--- a/guix/derivations.scm+++ b/guix/derivations.scm@@ -239,12 +239,17 @@ the store." "Return a list of inputs, such that when INPUTS contains the same DRV twice, they are coalesced, with their sub-derivations merged. This is needed because Nix itself keeps only one of them."+ (define (derivation-file-name* obj)+ (if (derivation? obj)+ (derivation-file-name obj)+ obj))+ (fold (lambda (input result) (match input- (($ <derivation-input> (= derivation-file-name path) sub-drvs)+ (($ <derivation-input> (= derivation-file-name* path) sub-drvs) ;; XXX: quadratic (match (find (match-lambda- (($ <derivation-input> (= derivation-file-name p)+ (($ <derivation-input> (= derivation-file-name* p) s) (string=? p path))) result)@@ -685,7 +690,7 @@ name of each input with that input's hash." (make-derivation-input hash sub-drvs)))) inputs))) (make-derivation outputs- (sort inputs+ (sort (coalesce-duplicate-inputs inputs) (lambda (drv1 drv2) (string<? (derivation-input-derivation drv1) (derivation-input-derivation drv2))))
L
L
Ludovic Courtès wrote on 27 Jul 2019 23:49
control message for bug #36777
(address . control@debbugs.gnu.org)
87pnlvi1wt.fsf@gnu.org
severity 36777 importantquit
C
C
Carl Dong wrote on 30 Jul 2019 20:36
Re: bug#36777: Guix Inferiors: Curious incorrect derivation output bug
(name . Ludovic Courtès)(address . ludo@gnu.org)(name . 36777\@debbugs.gnu.org)(address . 36777@debbugs.gnu.org)
rvQNShnNXKeE-TRrrP9SzUqXX4dJEB481poZeW9Rh_RZ7tgj7BRuBFnV049S2IfmM6v32jY1T8WM_zn5ZDX-2Kyfw16jm4wSaRshHnntHJQ=@carldong.me
All,
Yes! The patch actually fixed the problem when applied on top of 5cf4b26d52bcea382d98fb4becce89be9ee37b55! I was also able to come up with a minimal reproducing manifest:
```(use-modules (guix inferior) (guix channels) (srfi srfi-1)) ;for 'first'
(define channels (list (channel (name 'guix) (url "https://git.savannah.gnu.org/git/guix.git") (commit "6869b6635afd93872b8f0d9f2db0db4c0d765a86"))))
(define inferior ;; An inferior representing the above revision. (inferior-for-channels channels))
(packages->manifest (map (lambda (x) (first (lookup-inferior-packages inferior x))) '("gzip")))```
It seems that the `gzip` package is what's causing this.
Not sure what the next steps are for this, but I'd very much like to understand where this went wrong. Perhaps we could write tests for this so it doesn't happen in the future for releases.
Cheers,Carl Dongcontact@carldong.me"I fight for the users"
L
L
Ludovic Courtès wrote on 16 Aug 2019 23:03
(name . Carl Dong)(address . contact@carldong.me)(name . 36777@debbugs.gnu.org)(address . 36777-done@debbugs.gnu.org)
87h86gg6w4.fsf@gnu.org
Hi Carl,
Carl Dong <contact@carldong.me> skribis:
Toggle quote (2 lines)> Yes! The patch actually fixed the problem when applied on top of 5cf4b26d52bcea382d98fb4becce89be9ee37b55!
[...]
Toggle quote (2 lines)> Not sure what the next steps are for this, but I'd very much like to understand where this went wrong. Perhaps we could write tests for this so it doesn't happen in the future for releases.
Yup, I understood when this could happen (if multiple inputs of aderivation are “fixed-output” derivations leading to the same output),wrote a test for that, and came up with a simpler fix in commit268896444bed7b958add74b2e1e86ff802c5f5cb.
Let me know if anything is amiss!
Thanks for testing the patch,Ludo’.
Closed
?
Your comment

This issue is archived.

To comment on this conversation send email to 36777@debbugs.gnu.org