core-update: Raw origin tarballs are not handled correctly.

  • Done
  • quality assurance status badge
Details
2 participants
  • Maxim Cournoyer
  • Mathieu Othacehe
Owner
unassigned
Submitted by
Mathieu Othacehe
Severity
important
M
M
Mathieu Othacehe wrote on 15 Aug 2021 10:42
(address . bug-guix@gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87bl5z85ga.fsf@gnu.org
Hello,

When building "emacs-org-contrib" on core-updates-frozen, the following
message is printed:

Toggle snippet (12 lines)
tar: Exiting with failure status due to previous errors
error: in phase 'unpack': uncaught exception:
%exception #<&invoke-error program: "tar" arguments: ("xvf" "/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar") exit-status: 2 term-signal: #f stop-signal: #f>
phase `unpack' failed after 0.0 seconds
command "tar" "xvf" "/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar" failed with status 2
note: keeping build directory `/tmp/guix-build-emacs-org-contrib-20210809.drv-6'
builder for `/gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv' failed with exit code 1
build of /gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv failed
View build log at '/var/log/guix/drvs/52/jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv.bz2'.
guix build: error: build of `/gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv' failed

That's because
"/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar"
is empty. I believe this is a consequence of:
cfcead2e515c0dae02127e5a76496463898be6b6, and in particular this
section:

Toggle snippet (12 lines)
;; If SOURCE is a directory (such as a checkout), return a
;; directory. Otherwise create a tarball.
(cond
((file-is-directory? #+source)
(copy-recursively directory #$output
#:log (%make-void-port "w")))
((not #+comp)
(copy-file file #$output)) <---------- We should call repack instead here for raw tarballs?
(else
(repack directory #$output)))))))

Maxim, could you please have a look?

Thanks,

Mathieu
M
M
Maxim Cournoyer wrote on 27 Aug 2021 06:19
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . bug-guix@gnu.org)
87lf4nsene.fsf@gmail.com
Hello Mathieu,

Mathieu Othacehe <othacehe@gnu.org> writes:

Toggle quote (36 lines)
> Hello,
>
> When building "emacs-org-contrib" on core-updates-frozen, the following
> message is printed:
>
> tar: Exiting with failure status due to previous errors
> error: in phase 'unpack': uncaught exception:
> %exception #<&invoke-error program: "tar" arguments: ("xvf" "/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar") exit-status: 2 term-signal: #f stop-signal: #f>
> phase `unpack' failed after 0.0 seconds
> command "tar" "xvf" "/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar" failed with status 2
> note: keeping build directory `/tmp/guix-build-emacs-org-contrib-20210809.drv-6'
> builder for `/gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv' failed with exit code 1
> build of /gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv failed
> View build log at '/var/log/guix/drvs/52/jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv.bz2'.
> guix build: error: build of `/gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv' failed
>
>
> That's because
> "/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar"
> is empty. I believe this is a consequence of:
> cfcead2e515c0dae02127e5a76496463898be6b6, and in particular this
> section:
>
> ;; If SOURCE is a directory (such as a checkout), return a
> ;; directory. Otherwise create a tarball.
> (cond
> ((file-is-directory? #+source)
> (copy-recursively directory #$output
> #:log (%make-void-port "w")))
> ((not #+comp)
> (copy-file file #$output)) <---------- We should call repack instead here for raw tarballs?
> (else
> (repack directory #$output)))))))
>
> Maxim, could you please have a look?

Thanks for the report; it looks like a bug indeed. I had probably
overlooked that case (where there is no compression but we are still
dealing with an archive format). The annoying thing is that touching
this code will trigger a world rebuild; so unless we find other big
problems and batch the fixes, it may not be worth it.

The site.py problem you reported may be one, if I find a solution.

Other things on my mind:

1. rustc bootstrap from 1.39 (the itch to make it faster comes back
everytime I have to build it ;-)).

2. fontconfig update that should allow per-profile fonts management via
XDG_DATA_DIRS (I have the commit ready, but failed to test it due to
having to build rust and getting distracted by 1. :-))

Not a very useful reply in itself, but I felt that communicating what
I've been busy with was better than a longer silence :-).

Thank you,

Maxim
M
M
Maxim Cournoyer wrote on 27 Aug 2021 19:03
control message for bug #50066
(address . control@debbugs.gnu.org)
87k0k6stuw.fsf@gmail.com
severity 50066 important
quit
M
M
Maxim Cournoyer wrote on 31 Aug 2021 05:50
Re: bug#50066: core-update: Raw origin tarballs are not handled correctly.
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50066@debbugs.gnu.org)
874kb6jmrn.fsf@gmail.com
Hello Mathieu,

[...]

Toggle quote (13 lines)
> Thanks for the report; it looks like a bug indeed. I had probably
> overlooked that case (where there is no compression but we are still
> dealing with an archive format). The annoying thing is that touching
> this code will trigger a world rebuild; so unless we find other big
> problems and batch the fixes, it may not be worth it.
>
> The site.py problem you reported may be one, if I find a solution.
>
> Other things on my mind:
>
> 1. rustc bootstrap from 1.39 (the itch to make it faster comes back
> everytime I have to build it ;-)).

I'm nearing completion on this front.

Toggle quote (4 lines)
> 2. fontconfig update that should allow per-profile fonts management via
> XDG_DATA_DIRS (I have the commit ready, but failed to test it due to
> having to build rust and getting distracted by 1. :-))

I'll propose a few patches to include in the 'frozen' core-updates
branch since we've found a world-rebuilding change worthy of addressing
anyway (the patch below).

1 file changed, 4 insertions(+), 4 deletions(-)
guix/packages.scm | 8 ++++----

modified guix/packages.scm
@@ -836,10 +836,10 @@ specifies modules in scope when evaluating SNIPPET."
((file-is-directory? #+source)
(copy-recursively directory #$output
#:log (%make-void-port "w")))
- ((not #+comp)
- (copy-file file #$output))
- (else
- (repack directory #$output)))))))
+ ((or #+comp (tarball? #+source))
+ (repack directory #$output))
+ (else ;single uncompressed file
+ (copy-file file #$output)))))))
(let ((name (if (or (checkout? original-file-name)
(not (compressor original-file-name)))

Sadly the plain .tar file was/is not covered by tests (and not easily
so), but if his was my only omission, it should be correct (TM).
Attentive review needed :-).

Thank you!

Maxim
M
M
Maxim Cournoyer wrote on 31 Aug 2021 15:47
control message for bug #50066
(address . control@debbugs.gnu.org)
87v93liv4s.fsf@gmail.com
tags 50066 + patch
quit
M
M
Mathieu Othacehe wrote on 1 Sep 2021 09:09
Re: bug#50066: core-update: Raw origin tarballs are not handled correctly.
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 50066@debbugs.gnu.org)
87lf4gvkkd.fsf@gnu.org
Hello Maxim,

Toggle quote (5 lines)
>> 1. rustc bootstrap from 1.39 (the itch to make it faster comes back
>> everytime I have to build it ;-)).
>
> I'm nearing completion on this front.

That will be a welcomed improvement!

Toggle quote (8 lines)
>> 2. fontconfig update that should allow per-profile fonts management via
>> XDG_DATA_DIRS (I have the commit ready, but failed to test it due to
>> having to build rust and getting distracted by 1. :-))
>
> I'll propose a few patches to include in the 'frozen' core-updates
> branch since we've found a world-rebuilding change worthy of addressing
> anyway (the patch below).

Great! As we are going for a full rebuild, maybe we should also consider
some of the Thiago fixes targeting core-updates-frozen such as

Toggle quote (4 lines)
> Sadly the plain .tar file was/is not covered by tests (and not easily
> so), but if his was my only omission, it should be correct (TM).
> Attentive review needed :-).

This seems reasonable and I'm currently rebuilding the world to test
this fix and the Python fix. I should maybe apply your Rust bootstrap
patch if you have it around to gain some time.

Thanks for all those fixes!

Mathieu
M
M
Maxim Cournoyer wrote on 1 Sep 2021 14:57
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50066@debbugs.gnu.org)
87y28gcv2j.fsf@gmail.com
Hi Mathieu,

Mathieu Othacehe <othacehe@gnu.org> writes:

Toggle quote (21 lines)
> Hello Maxim,
>
>>> 1. rustc bootstrap from 1.39 (the itch to make it faster comes back
>>> everytime I have to build it ;-)).
>>
>> I'm nearing completion on this front.
>
> That will be a welcomed improvement!
>
>>> 2. fontconfig update that should allow per-profile fonts management via
>>> XDG_DATA_DIRS (I have the commit ready, but failed to test it due to
>>> having to build rust and getting distracted by 1. :-))
>>
>> I'll propose a few patches to include in the 'frozen' core-updates
>> branch since we've found a world-rebuilding change worthy of addressing
>> anyway (the patch below).
>
> Great! As we are going for a full rebuild, maybe we should also consider
> some of the Thiago fixes targeting core-updates-frozen such as
> https://issues.guix.gnu.org/50239.

Yes, I think we should cherry-pick anything valuable before the big
rebuild of the frozen branch (perhaps we should simply sync with
core-updates?)

Toggle quote (8 lines)
>> Sadly the plain .tar file was/is not covered by tests (and not easily
>> so), but if his was my only omission, it should be correct (TM).
>> Attentive review needed :-).
>
> This seems reasonable and I'm currently rebuilding the world to test
> this fix and the Python fix. I should maybe apply your Rust bootstrap
> patch if you have it around to gain some time.

It's currently building Rust 1.47 (it takes about 15 minutes per Rust).
I'll build it all until 1.54, then recheck a few ones for
reproducibility (at least the first and last ones). If all is good I'll
send the patches I have.

Toggle quote (2 lines)
> Thanks for all those fixes!

Thank you for the testing/reports!

Maxim
M
M
Maxim Cournoyer wrote on 3 Sep 2021 15:30
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50066@debbugs.gnu.org)
877dfxdbwq.fsf@gmail.com
Hi Mathieu,

Mathieu Othacehe <othacehe@gnu.org> writes:

Toggle quote (29 lines)
> Hello Maxim,
>
>>> 1. rustc bootstrap from 1.39 (the itch to make it faster comes back
>>> everytime I have to build it ;-)).
>>
>> I'm nearing completion on this front.
>
> That will be a welcomed improvement!
>
>>> 2. fontconfig update that should allow per-profile fonts management via
>>> XDG_DATA_DIRS (I have the commit ready, but failed to test it due to
>>> having to build rust and getting distracted by 1. :-))
>>
>> I'll propose a few patches to include in the 'frozen' core-updates
>> branch since we've found a world-rebuilding change worthy of addressing
>> anyway (the patch below).
>
> Great! As we are going for a full rebuild, maybe we should also consider
> some of the Thiago fixes targeting core-updates-frozen such as
> https://issues.guix.gnu.org/50239.
>
>> Sadly the plain .tar file was/is not covered by tests (and not easily
>> so), but if his was my only omission, it should be correct (TM).
>> Attentive review needed :-).
>
> This seems reasonable and I'm currently rebuilding the world to test
> this fix and the Python fix. I should maybe apply your Rust bootstrap
> patch if you have it around to gain some time.

successfully built /gnu/store/vag2a53byhgrkpchviqz78lilall6412-emacs-org-contrib-20210519.drv
/gnu/store/40hl0dli745i2ggs1aqhn2drh0fkmrbx-emacs-org-contrib-20210519

:-).

I'm trying to fix one last thing with my local patch set (mozjs not
building with rust@1.54), but if takes too much time, I'll send what I
have and we can figure it out together.

Thanks,

Maxim
M
M
Mathieu Othacehe wrote on 19 Dec 2021 17:40
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 50066-done@debbugs.gnu.org)
874k74r1id.fsf@gnu.org
Hey Maxim,

Toggle quote (3 lines)
> successfully built /gnu/store/vag2a53byhgrkpchviqz78lilall6412-emacs-org-contrib-20210519.drv
> /gnu/store/40hl0dli745i2ggs1aqhn2drh0fkmrbx-emacs-org-contrib-20210519

Thanks for fixing this one,

Closing,

Mathieu
Closed
?