core-update: Raw origin tarballs are not handled correctly.

  • Done
  • quality assurance status badge
Details
2 participants
  • Maxim Cournoyer
  • Mathieu Othacehe
Owner
unassigned
Submitted by
Mathieu Othacehe
Severity
important
M
M
Mathieu Othacehe wrote on 15 Aug 2021 10:42
(address . bug-guix@gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
87bl5z85ga.fsf@gnu.org
Hello,

When building "emacs-org-contrib" on core-updates-frozen, the following
message is printed:

Toggle snippet (12 lines)
tar: Exiting with failure status due to previous errors
error: in phase 'unpack': uncaught exception:
%exception #<&invoke-error program: "tar" arguments: ("xvf" "/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar") exit-status: 2 term-signal: #f stop-signal: #f>
phase `unpack' failed after 0.0 seconds
command "tar" "xvf" "/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar" failed with status 2
note: keeping build directory `/tmp/guix-build-emacs-org-contrib-20210809.drv-6'
builder for `/gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv' failed with exit code 1
build of /gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv failed
View build log at '/var/log/guix/drvs/52/jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv.bz2'.
guix build: error: build of `/gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv' failed

That's because
"/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar"
is empty. I believe this is a consequence of:
cfcead2e515c0dae02127e5a76496463898be6b6, and in particular this
section:

Toggle snippet (12 lines)
;; If SOURCE is a directory (such as a checkout), return a
;; directory. Otherwise create a tarball.
(cond
((file-is-directory? #+source)
(copy-recursively directory #$output
#:log (%make-void-port "w")))
((not #+comp)
(copy-file file #$output)) <---------- We should call repack instead here for raw tarballs?
(else
(repack directory #$output)))))))

Maxim, could you please have a look?

Thanks,

Mathieu
M
M
Maxim Cournoyer wrote on 27 Aug 2021 06:19
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . bug-guix@gnu.org)
87lf4nsene.fsf@gmail.com
Hello Mathieu,

Mathieu Othacehe <othacehe@gnu.org> writes:

Toggle quote (36 lines)
> Hello,
>
> When building "emacs-org-contrib" on core-updates-frozen, the following
> message is printed:
>
> tar: Exiting with failure status due to previous errors
> error: in phase 'unpack': uncaught exception:
> %exception #<&invoke-error program: "tar" arguments: ("xvf" "/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar") exit-status: 2 term-signal: #f stop-signal: #f>
> phase `unpack' failed after 0.0 seconds
> command "tar" "xvf" "/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar" failed with status 2
> note: keeping build directory `/tmp/guix-build-emacs-org-contrib-20210809.drv-6'
> builder for `/gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv' failed with exit code 1
> build of /gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv failed
> View build log at '/var/log/guix/drvs/52/jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv.bz2'.
> guix build: error: build of `/gnu/store/52jcx6rwnf0wrkrpcwkyz766fzg6asjy-emacs-org-contrib-20210809.drv' failed
>
>
> That's because
> "/gnu/store/r3v4gkqqbxp2w739a0llvk9nrda9xn89-org-plus-contrib-20210809.tar"
> is empty. I believe this is a consequence of:
> cfcead2e515c0dae02127e5a76496463898be6b6, and in particular this
> section:
>
> ;; If SOURCE is a directory (such as a checkout), return a
> ;; directory. Otherwise create a tarball.
> (cond
> ((file-is-directory? #+source)
> (copy-recursively directory #$output
> #:log (%make-void-port "w")))
> ((not #+comp)
> (copy-file file #$output)) <---------- We should call repack instead here for raw tarballs?
> (else
> (repack directory #$output)))))))
>
> Maxim, could you please have a look?

Thanks for the report; it looks like a bug indeed. I had probably
overlooked that case (where there is no compression but we are still
dealing with an archive format). The annoying thing is that touching
this code will trigger a world rebuild; so unless we find other big
problems and batch the fixes, it may not be worth it.

The site.py problem you reported may be one, if I find a solution.

Other things on my mind:

1. rustc bootstrap from 1.39 (the itch to make it faster comes back
everytime I have to build it ;-)).

2. fontconfig update that should allow per-profile fonts management via
XDG_DATA_DIRS (I have the commit ready, but failed to test it due to
having to build rust and getting distracted by 1. :-))

Not a very useful reply in itself, but I felt that communicating what
I've been busy with was better than a longer silence :-).

Thank you,

Maxim
M
M
Maxim Cournoyer wrote on 27 Aug 2021 19:03
control message for bug #50066
(address . control@debbugs.gnu.org)
87k0k6stuw.fsf@gmail.com
severity 50066 important
quit
M
M
Maxim Cournoyer wrote on 31 Aug 2021 05:50
Re: bug#50066: core-update: Raw origin tarballs are not handled correctly.
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50066@debbugs.gnu.org)
874kb6jmrn.fsf@gmail.com
Hello Mathieu,

[...]

Toggle quote (13 lines)
> Thanks for the report; it looks like a bug indeed. I had probably
> overlooked that case (where there is no compression but we are still
> dealing with an archive format). The annoying thing is that touching
> this code will trigger a world rebuild; so unless we find other big
> problems and batch the fixes, it may not be worth it.
>
> The site.py problem you reported may be one, if I find a solution.
>
> Other things on my mind:
>
> 1. rustc bootstrap from 1.39 (the itch to make it faster comes back
> everytime I have to build it ;-)).

I'm nearing completion on this front.

Toggle quote (4 lines)
> 2. fontconfig update that should allow per-profile fonts management via
> XDG_DATA_DIRS (I have the commit ready, but failed to test it due to
> having to build rust and getting distracted by 1. :-))

I'll propose a few patches to include in the 'frozen' core-updates
branch since we've found a world-rebuilding change worthy of addressing
anyway (the patch below).

1 file changed, 4 insertions(+), 4 deletions(-)
guix/packages.scm | 8 ++++----

modified guix/packages.scm
@@ -836,10 +836,10 @@ specifies modules in scope when evaluating SNIPPET."
((file-is-directory? #+source)
(copy-recursively directory #$output
#:log (%make-void-port "w")))
- ((not #+comp)
- (copy-file file #$output))
- (else
- (repack directory #$output)))))))
+ ((or #+comp (tarball? #+source))
+ (repack directory #$output))
+ (else ;single uncompressed file
+ (copy-file file #$output)))))))
(let ((name (if (or (checkout? original-file-name)
(not (compressor original-file-name)))

Sadly the plain .tar file was/is not covered by tests (and not easily
so), but if his was my only omission, it should be correct (TM).
Attentive review needed :-).

Thank you!

Maxim
M
M
Maxim Cournoyer wrote on 31 Aug 2021 15:47
control message for bug #50066
(address . control@debbugs.gnu.org)
87v93liv4s.fsf@gmail.com
tags 50066 + patch
quit
M
M
Mathieu Othacehe wrote on 1 Sep 2021 09:09
Re: bug#50066: core-update: Raw origin tarballs are not handled correctly.
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 50066@debbugs.gnu.org)
87lf4gvkkd.fsf@gnu.org
Hello Maxim,

Toggle quote (5 lines)
>> 1. rustc bootstrap from 1.39 (the itch to make it faster comes back
>> everytime I have to build it ;-)).
>
> I'm nearing completion on this front.

That will be a welcomed improvement!

Toggle quote (8 lines)
>> 2. fontconfig update that should allow per-profile fonts management via
>> XDG_DATA_DIRS (I have the commit ready, but failed to test it due to
>> having to build rust and getting distracted by 1. :-))
>
> I'll propose a few patches to include in the 'frozen' core-updates
> branch since we've found a world-rebuilding change worthy of addressing
> anyway (the patch below).

Great! As we are going for a full rebuild, maybe we should also consider
some of the Thiago fixes targeting core-updates-frozen such as

Toggle quote (4 lines)
> Sadly the plain .tar file was/is not covered by tests (and not easily
> so), but if his was my only omission, it should be correct (TM).
> Attentive review needed :-).

This seems reasonable and I'm currently rebuilding the world to test
this fix and the Python fix. I should maybe apply your Rust bootstrap
patch if you have it around to gain some time.

Thanks for all those fixes!

Mathieu
M
M
Maxim Cournoyer wrote on 1 Sep 2021 14:57
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50066@debbugs.gnu.org)
87y28gcv2j.fsf@gmail.com
Hi Mathieu,

Mathieu Othacehe <othacehe@gnu.org> writes:

Toggle quote (21 lines)
> Hello Maxim,
>
>>> 1. rustc bootstrap from 1.39 (the itch to make it faster comes back
>>> everytime I have to build it ;-)).
>>
>> I'm nearing completion on this front.
>
> That will be a welcomed improvement!
>
>>> 2. fontconfig update that should allow per-profile fonts management via
>>> XDG_DATA_DIRS (I have the commit ready, but failed to test it due to
>>> having to build rust and getting distracted by 1. :-))
>>
>> I'll propose a few patches to include in the 'frozen' core-updates
>> branch since we've found a world-rebuilding change worthy of addressing
>> anyway (the patch below).
>
> Great! As we are going for a full rebuild, maybe we should also consider
> some of the Thiago fixes targeting core-updates-frozen such as
> https://issues.guix.gnu.org/50239.

Yes, I think we should cherry-pick anything valuable before the big
rebuild of the frozen branch (perhaps we should simply sync with
core-updates?)

Toggle quote (8 lines)
>> Sadly the plain .tar file was/is not covered by tests (and not easily
>> so), but if his was my only omission, it should be correct (TM).
>> Attentive review needed :-).
>
> This seems reasonable and I'm currently rebuilding the world to test
> this fix and the Python fix. I should maybe apply your Rust bootstrap
> patch if you have it around to gain some time.

It's currently building Rust 1.47 (it takes about 15 minutes per Rust).
I'll build it all until 1.54, then recheck a few ones for
reproducibility (at least the first and last ones). If all is good I'll
send the patches I have.

Toggle quote (2 lines)
> Thanks for all those fixes!

Thank you for the testing/reports!

Maxim
M
M
Maxim Cournoyer wrote on 3 Sep 2021 15:30
(name . Mathieu Othacehe)(address . othacehe@gnu.org)(address . 50066@debbugs.gnu.org)
877dfxdbwq.fsf@gmail.com
Hi Mathieu,

Mathieu Othacehe <othacehe@gnu.org> writes:

Toggle quote (29 lines)
> Hello Maxim,
>
>>> 1. rustc bootstrap from 1.39 (the itch to make it faster comes back
>>> everytime I have to build it ;-)).
>>
>> I'm nearing completion on this front.
>
> That will be a welcomed improvement!
>
>>> 2. fontconfig update that should allow per-profile fonts management via
>>> XDG_DATA_DIRS (I have the commit ready, but failed to test it due to
>>> having to build rust and getting distracted by 1. :-))
>>
>> I'll propose a few patches to include in the 'frozen' core-updates
>> branch since we've found a world-rebuilding change worthy of addressing
>> anyway (the patch below).
>
> Great! As we are going for a full rebuild, maybe we should also consider
> some of the Thiago fixes targeting core-updates-frozen such as
> https://issues.guix.gnu.org/50239.
>
>> Sadly the plain .tar file was/is not covered by tests (and not easily
>> so), but if his was my only omission, it should be correct (TM).
>> Attentive review needed :-).
>
> This seems reasonable and I'm currently rebuilding the world to test
> this fix and the Python fix. I should maybe apply your Rust bootstrap
> patch if you have it around to gain some time.

successfully built /gnu/store/vag2a53byhgrkpchviqz78lilall6412-emacs-org-contrib-20210519.drv
/gnu/store/40hl0dli745i2ggs1aqhn2drh0fkmrbx-emacs-org-contrib-20210519

:-).

I'm trying to fix one last thing with my local patch set (mozjs not
building with rust@1.54), but if takes too much time, I'll send what I
have and we can figure it out together.

Thanks,

Maxim
M
M
Mathieu Othacehe wrote on 19 Dec 2021 17:40
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 50066-done@debbugs.gnu.org)
874k74r1id.fsf@gnu.org
Hey Maxim,

Toggle quote (3 lines)
> successfully built /gnu/store/vag2a53byhgrkpchviqz78lilall6412-emacs-org-contrib-20210519.drv
> /gnu/store/40hl0dli745i2ggs1aqhn2drh0fkmrbx-emacs-org-contrib-20210519

Thanks for fixing this one,

Closing,

Mathieu
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 50066@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 50066
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch