'add-to-store' RPC loads files in memory (daemon-side)

OpenSubmitted by Andreas Enge.
Details
3 participants
  • Andreas Enge
  • Leo Famulari
  • Ludovic Courtès
Owner
unassigned
Severity
normal
Merged with
A
A
Andreas Enge wrote on 31 May 2016 21:56
Download
guix download fails for large files
(address . bug-guix@gnu.org)
20160531195631.GA22897@solar
Download
Hello,

since I am regularly rebuilding texlive (who is not!), I would like to keep
the source tarball on my machine, and whenever I garbage collect it from the
store, add it back via
guix download file:texlive-20150523-texmf.tar.xz

On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
guix download: error: sendfile: Broken pipe

The file itself has 1,8 GB.

Downloading the file via http as part of "guix build" works, and
"guix download file:" also works on my x86_64 machine with 8 GB of RAM.

Andreas
L
L
Leo Famulari wrote on 31 May 2016 22:26
Download
(name . Andreas Enge)(address . andreas@enge.fr)(address . 23666@debbugs.gnu.org)
20160531202651.GA19070@jasmine
Download
On Tue, May 31, 2016 at 09:56:31PM +0200, Andreas Enge wrote:
Toggle quote (13 lines)
> since I am regularly rebuilding texlive (who is not!), I would like to keep
> the source tarball on my machine, and whenever I garbage collect it from the
> store, add it back via
> guix download file:texlive-20150523-texmf.tar.xz
>
> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
> guix download: error: sendfile: Broken pipe
>
> The file itself has 1,8 GB.
>
> Downloading the file via http as part of "guix build" works, and
> "guix download file:" also works on my x86_64 machine with 8 GB of RAM.

As a temporary workaround, you could make the tarball a gcroot:

# ln -sf /gnu/store/...-texlive-20150523-texmf.tar.xz \
/var/guix/gcroots/texlive-texmf.tar.xz
L
L
Ludovic Courtès wrote on 1 Jun 2016 14:39
Download
(name . Andreas Enge)(address . andreas@enge.fr)(address . 23666@debbugs.gnu.org)
878typkrqd.fsf@gnu.org
Download
Andreas Enge <andreas@enge.fr> skribis:

Toggle quote (5 lines)
> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
> guix download: error: sendfile: Broken pipe
>
> The file itself has 1,8 GB.

This is implemented using the ‘add-to-store’ RPC, which, after all these
years, is still implemented like this (nix/libstore/local-store.cc):

Toggle snippet (11 lines)
Path LocalStore::addToStore(const string & name, const Path & _srcPath,
bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
{
Path srcPath(absPath(_srcPath));
debug(format("adding `%1%' to the store") % srcPath);

/* Read the whole path into memory. This is not a very scalable
method for very large paths, but `copyPath' is mainly used for
small files. */

This is undoubtedly what’s causing the failure.

Until this is fixed, you could wrap the file in a nar and import the nar
(nar import does not have this limitation, see ’LocalStore::importPath’
and co.).

To do that, you could do:

guix archive --export $(guix build -S texlive-texmf-whatever) > t.nar

and later:

guix archive --import < t.nar

HTH,
Ludo’.
L
L
Ludovic Courtès wrote on 1 Jun 2016 14:40
Download
retitle
(address . request@debbugs.gnu.org)
874m9dkroo.fsf@gnu.org
Download
retitle 23666 'add-to-store' RPC loads files in memory (daemon-side)
thanks
A
A
Andreas Enge wrote on 5 Jun 2016 22:58
Download
Re: bug#23666: guix download fails for large files
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 23666@debbugs.gnu.org)
20160605205839.GB20831@solar
Download
Thanks for your suggestions, Leo and Ludovic! I still see this as a bug;
should I report it upstream to Nix?

On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote:
Toggle quote (15 lines)
> This is implemented using the ‘add-to-store’ RPC, which, after all these
> years, is still implemented like this (nix/libstore/local-store.cc):
>
> --8<---------------cut here---------------start------------->8---
> Path LocalStore::addToStore(const string & name, const Path & _srcPath,
> bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
> {
> Path srcPath(absPath(_srcPath));
> debug(format("adding `%1%' to the store") % srcPath);
>
> /* Read the whole path into memory. This is not a very scalable
> method for very large paths, but `copyPath' is mainly used for
> small files. */
> --8<---------------cut here---------------end--------------->8---

Something that mainly does not fail could indeed be seen as a bug...
But how come that "guix download http://" succeeds, where
"guix download file://" fails?

Andreas
L
L
Ludovic Courtès wrote on 6 Jun 2016 09:06
Download
(name . Andreas Enge)(address . andreas@enge.fr)(address . 23666@debbugs.gnu.org)
87k2i27q5c.fsf@gnu.org
Download
Andreas Enge <andreas@enge.fr> skribis:

Toggle quote (20 lines)
> On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote:
>> This is implemented using the ‘add-to-store’ RPC, which, after all these
>> years, is still implemented like this (nix/libstore/local-store.cc):
>>
>> --8<---------------cut here---------------start------------->8---
>> Path LocalStore::addToStore(const string & name, const Path & _srcPath,
>> bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
>> {
>> Path srcPath(absPath(_srcPath));
>> debug(format("adding `%1%' to the store") % srcPath);
>>
>> /* Read the whole path into memory. This is not a very scalable
>> method for very large paths, but `copyPath' is mainly used for
>> small files. */
>> --8<---------------cut here---------------end--------------->8---
>
> Something that mainly does not fail could indeed be seen as a bug...
> But how come that "guix download http://" succeeds, where
> "guix download file://" fails?

Both use the ‘add-to-store’ RPC, so both have equal chances of failing.

Note that, fortunately, fixed-output derivations (‘guix build -S foo’)
do not use ‘add-to-store’. In general, except in the case of ‘guix
download’, ‘add-to-store’ is only used with small files like when adding
.scm files to the store, etc.

Ludo’.
L
L
Ludovic Courtès wrote on 24 Jul 2016 16:03
Download
control message for bug #24053
(address . control@debbugs.gnu.org)
87oa5ncevv.fsf@gnu.org
Download
merge 24053 23666
?
You may also send email to 23666@debbugs.gnu.org to comment.