low performance when building profile of already built packages

  • Open
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Ricardo Wurmus
Owner
unassigned
Submitted by
Ricardo Wurmus
Severity
important
Merged with
R
R
Ricardo Wurmus wrote on 24 Apr 2021 22:38
(address . bug-guix@gnu.org)
87zgxnpgd9.fsf@elephly.net
Here is a manifest that isn’t unusual for a bioinformatics
workflow:

Toggle snippet (34 lines)
(define packages
(list "bash-minimal"
"fastqc"
"gzip"
"htseq"
"multiqc"
"r-corrplot"
"r-crosstalk"
"r-data-table"
"r-deseq2"
"r-dt"
"r-genomicalignments"
"r-genomicranges"
"r-ggplot2"
"r-ggrepel"
"r-gprofiler"
"r-knitr"
"r-pheatmap"
"r-plotly"
"r-reshape2"
"r-rmarkdown"
"r-rsamtools"
"r-rtracklayer"
"r-s4vectors"
"r-scales"
"r-summarizedexperiment"
"r-tximport"
"samtools"
"sed"
"trim-galore"))

(specifications->manifest packages)

When all these packages already exist and nothing is to be done
when Guix is asked to instantiate this manifest, more than 13
seconds pass.

Toggle snippet (59 lines)
$ [env] export GUIX_PROFILING="object-cache
add-data-to-store-cache rpc"
$ [env] time guix package -m manifest-test.scm -p /tmp/foo
The following packages will be installed:
bash-minimal 5.0.16
fastqc 0.11.5
gzip 1.10
htseq 0.9.1
multiqc 1.5
r-corrplot 0.84
r-crosstalk 1.1.1
r-data-table 1.14.0
r-deseq2 1.30.1
r-dt 0.17
r-genomicalignments 1.26.0
r-genomicranges 1.42.0
r-ggplot2 3.3.3
r-ggrepel 0.9.1
r-gprofiler 0.7.0
r-knitr 1.31
r-pheatmap 1.0.12
r-plotly 4.9.3
r-reshape2 1.4.4
r-rmarkdown 2.7
r-rsamtools 2.6.0
r-rtracklayer 1.50.0
r-s4vectors 0.28.1
r-scales 1.1.1
r-summarizedexperiment 1.20.0
r-tximport 1.18.0
samtools 1.11
sed 4.8
trim-galore 0.6.6
[[most of the time passes here]]

nothing to be done
Store object cache:
fresh caches: 2
lookups: 45493
hits: 43943 (96.6%)
cache size: 1549 entries

'add-data-to-store' cache:
lookups: 7755
hits: 3411 (44.0%)
.drv files: 4357 (56.2%)
Scheme files: 3343 (43.1%)
Remote procedure call summary: 5283 RPCs
built-in-builders ... 1
add-to-store/tree ... 27
add-to-store ... 213
query-references ... 698
add-text-to-store ... 4344

real 0m13.379s
user 0m16.414s
sys 0m0.253s

Some of these R packages have a lot of transitive inputs, such as
r-deseq2 with 81 transitive propagated inputs. So there’s a lot
of propagation here, which I suspect contributes to things being
so unexpectedly slow.

This report may be related to performance issues reported for
“guix environment” in https://issues.guix.gnu.org/41702,which
also involves R packages (such as r-learnr with 46 transitive
propagated inputs).

--
Ricardo
L
L
Ludovic Courtès wrote on 30 Apr 2021 17:45
control message for bug #48005
(address . control@debbugs.gnu.org)
87lf8zbwsv.fsf@gnu.org
severity 48005 important
quit
L
L
Ludovic Courtès wrote on 27 May 2021 18:32
control message for bug #41702
(address . control@debbugs.gnu.org)
87wnrkjfwv.fsf@gnu.org
merge 41702 48005
quit
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 48005@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 48005
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch