'add-to-store' RPC loads files in memory (daemon-side)

OpenSubmitted by Andreas Enge.
Details
3 participants
  • Andreas Enge
  • Leo Famulari
  • Ludovic Courtès
Owner
unassigned
Severity
normal
Merged with
A
A
Andreas Enge wrote on 31 May 2016 21:56
guix download fails for large files
(address . bug-guix@gnu.org)
20160531195631.GA22897@solar
Hello,
since I am regularly rebuilding texlive (who is not!), I would like to keepthe source tarball on my machine, and whenever I garbage collect it from thestore, add it back via guix download file:texlive-20150523-texmf.tar.xz
On my Novena with a 32 bit processor and 4 GB of main memory, this fails with guix download: error: sendfile: Broken pipe
The file itself has 1,8 GB.
Downloading the file via http as part of "guix build" works, and"guix download file:" also works on my x86_64 machine with 8 GB of RAM.
Andreas
L
L
Leo Famulari wrote on 31 May 2016 22:26
(name . Andreas Enge)(address . andreas@enge.fr)(address . 23666@debbugs.gnu.org)
20160531202651.GA19070@jasmine
On Tue, May 31, 2016 at 09:56:31PM +0200, Andreas Enge wrote:
Toggle quote (13 lines)> since I am regularly rebuilding texlive (who is not!), I would like to keep> the source tarball on my machine, and whenever I garbage collect it from the> store, add it back via> guix download file:texlive-20150523-texmf.tar.xz>> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with> guix download: error: sendfile: Broken pipe> > The file itself has 1,8 GB.> > Downloading the file via http as part of "guix build" works, and> "guix download file:" also works on my x86_64 machine with 8 GB of RAM.
As a temporary workaround, you could make the tarball a gcroot:
# ln -sf /gnu/store/...-texlive-20150523-texmf.tar.xz \/var/guix/gcroots/texlive-texmf.tar.xz
L
L
Ludovic Courtès wrote on 1 Jun 2016 14:39
(name . Andreas Enge)(address . andreas@enge.fr)(address . 23666@debbugs.gnu.org)
878typkrqd.fsf@gnu.org
Andreas Enge <andreas@enge.fr> skribis:
Toggle quote (5 lines)> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with> guix download: error: sendfile: Broken pipe>> The file itself has 1,8 GB.
This is implemented using the ‘add-to-store’ RPC, which, after all theseyears, is still implemented like this (nix/libstore/local-store.cc):
Toggle snippet (11 lines)Path LocalStore::addToStore(const string & name, const Path & _srcPath, bool recursive, HashType hashAlgo, PathFilter & filter, bool repair){ Path srcPath(absPath(_srcPath)); debug(format("adding `%1%' to the store") % srcPath);
/* Read the whole path into memory. This is not a very scalable method for very large paths, but `copyPath' is mainly used for small files. */
This is undoubtedly what’s causing the failure.
Until this is fixed, you could wrap the file in a nar and import the nar(nar import does not have this limitation, see ’LocalStore::importPath’and co.).
To do that, you could do:
guix archive --export $(guix build -S texlive-texmf-whatever) > t.nar
and later:
guix archive --import < t.nar
HTH,Ludo’.
L
L
Ludovic Courtès wrote on 1 Jun 2016 14:40
retitle
(address . request@debbugs.gnu.org)
874m9dkroo.fsf@gnu.org
retitle 23666 'add-to-store' RPC loads files in memory (daemon-side)thanks
A
A
Andreas Enge wrote on 5 Jun 2016 22:58
Re: bug#23666: guix download fails for large files
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 23666@debbugs.gnu.org)
20160605205839.GB20831@solar
Thanks for your suggestions, Leo and Ludovic! I still see this as a bug;should I report it upstream to Nix?
On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote:
Toggle quote (15 lines)> This is implemented using the ‘add-to-store’ RPC, which, after all these> years, is still implemented like this (nix/libstore/local-store.cc):> > --8<---------------cut here---------------start------------->8---> Path LocalStore::addToStore(const string & name, const Path & _srcPath,> bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)> {> Path srcPath(absPath(_srcPath));> debug(format("adding `%1%' to the store") % srcPath);> > /* Read the whole path into memory. This is not a very scalable> method for very large paths, but `copyPath' is mainly used for> small files. */> --8<---------------cut here---------------end--------------->8---
Something that mainly does not fail could indeed be seen as a bug...But how come that "guix download http://" succeeds, where"guix download file://" fails?
Andreas
L
L
Ludovic Courtès wrote on 6 Jun 2016 09:06
(name . Andreas Enge)(address . andreas@enge.fr)(address . 23666@debbugs.gnu.org)
87k2i27q5c.fsf@gnu.org
Andreas Enge <andreas@enge.fr> skribis:
Toggle quote (20 lines)> On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote:>> This is implemented using the ‘add-to-store’ RPC, which, after all these>> years, is still implemented like this (nix/libstore/local-store.cc):>> >> --8<---------------cut here---------------start------------->8--->> Path LocalStore::addToStore(const string & name, const Path & _srcPath,>> bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)>> {>> Path srcPath(absPath(_srcPath));>> debug(format("adding `%1%' to the store") % srcPath);>> >> /* Read the whole path into memory. This is not a very scalable>> method for very large paths, but `copyPath' is mainly used for>> small files. */>> --8<---------------cut here---------------end--------------->8--->> Something that mainly does not fail could indeed be seen as a bug...> But how come that "guix download http://" succeeds, where> "guix download file://" fails?
Both use the ‘add-to-store’ RPC, so both have equal chances of failing.
Note that, fortunately, fixed-output derivations (‘guix build -S foo’)do not use ‘add-to-store’. In general, except in the case of ‘guixdownload’, ‘add-to-store’ is only used with small files like when adding.scm files to the store, etc.
Ludo’.
L
L
Ludovic Courtès wrote on 24 Jul 2016 16:03
control message for bug #24053
(address . control@debbugs.gnu.org)
87oa5ncevv.fsf@gnu.org
merge 24053 23666
?