guix does not resume download on FTP timeouts

  • Open
  • quality assurance status badge
Details
3 participants
  • Leo Famulari
  • Ludovic Courtès
  • Sergei Trofimovich
Owner
unassigned
Submitted by
Sergei Trofimovich
Severity
normal
S
S
Sergei Trofimovich wrote on 7 May 2017 10:31
(address . bug-guix@gnu.org)
20170507093159.170a100b@sf
I have slow internet :)

It looks like ftp://tug.org has a FTP session timeout of 2 hours 11 minutes, or something like that.
Over night I tried to build sbcl and failed. Note how download silently stopped at ~80% [1].

Would be nice if guix had a mechanism to reconnect to FTP and continue download after timeout.

Thanks!

[1]: $ while :; do ./pre-inst-env guix build sbcl --no-substitutes; done

@ build-started /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - x86_64-linux /var/log/guix/drvs/yw//pxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv.bz2

Starting download of /gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz
From ftp://tug.org/historic/systems/texlive/2016/texlive-20160523b-texmf.tar.xz...
…3b-texmf.tar.xz 1.93GiB 213KiB/s 02:11:32 [################ ] 83.0%
output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `062qbbhlin1ys9lihax34kq503ig1fl55br84j94vwawbk04p6j0'
@ build-failed /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - 1 output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `062qbbhlin1ys9lihax34kq503ig1fl55br84j94vwawbk04p6j0'
@ build-started /gnu/store/in6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv - x86_64-linux /var/log/guix/drvs/in//6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv.bz2
cannot build derivation `/gnu/store/q9s2wdsaqvrllkcsf8w1a8424zd9ybc1-texlive-20160523b-texmf.tar.xz.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/b7abk0aarqv0ylxp2g4s2sf9j75xdy04-texlive-texmf-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/kpjy0c1pl8vzxbkys6p45ij3za6a92gj-texlive-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv': 1 dependencies couldn't be built
guix build: error: build failed: build of `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv' failed
@ build-started /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - x86_64-linux /var/log/guix/drvs/yw//pxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv.bz2

Starting download of /gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz
From ftp://tug.org/historic/systems/texlive/2016/texlive-20160523b-texmf.tar.xz...
…3b-texmf.tar.xz 1.93GiB 213KiB/s 02:11:46 [################ ] 83.3%
output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `1l296b4ypphffv1zpamp01g6a633kqlmfbf9l4x9411rlblb1hfd'
@ build-failed /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - 1 output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `1l296b4ypphffv1zpamp01g6a633kqlmfbf9l4x9411rlblb1hfd'
@ build-started /gnu/store/in6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv - x86_64-linux /var/log/guix/drvs/in//6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv.bz2
cannot build derivation `/gnu/store/q9s2wdsaqvrllkcsf8w1a8424zd9ybc1-texlive-20160523b-texmf.tar.xz.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/b7abk0aarqv0ylxp2g4s2sf9j75xdy04-texlive-texmf-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/kpjy0c1pl8vzxbkys6p45ij3za6a92gj-texlive-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv': 1 dependencies couldn't be built
guix build: error: build failed: build of `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv' failed
@ build-started /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - x86_64-linux /var/log/guix/drvs/yw//pxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv.bz2

Starting download of /gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz
From ftp://tug.org/historic/systems/texlive/2016/texlive-20160523b-texmf.tar.xz...
…3b-texmf.tar.xz 1.93GiB 213KiB/s 02:11:31 [################ ] 83.1%
output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `143zan8sh0lw2jjhdpvc7w9yjh7cw75w1ry60qq4g1n3hjmfrwi3'
@ build-failed /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - 1 output path `/gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz' should have sha256 hash `1dv8vgfzpczqw82hv9g7a8djhhyzywljmrarlcyy6g2qi5q51glr', instead has `143zan8sh0lw2jjhdpvc7w9yjh7cw75w1ry60qq4g1n3hjmfrwi3'
@ build-started /gnu/store/in6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv - x86_64-linux /var/log/guix/drvs/in//6vixj9fb9c8f9z3jf42zz9v40dbjxi-serf-1.3.8.drv.bz2
cannot build derivation `/gnu/store/q9s2wdsaqvrllkcsf8w1a8424zd9ybc1-texlive-20160523b-texmf.tar.xz.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/b7abk0aarqv0ylxp2g4s2sf9j75xdy04-texlive-texmf-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/kpjy0c1pl8vzxbkys6p45ij3za6a92gj-texlive-2016.drv': 1 dependencies couldn't be built
cannot build derivation `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv': 1 dependencies couldn't be built
guix build: error: build failed: build of `/gnu/store/clnamqpbkkn2k0ji1clbnznx2fyyig2c-sbcl-1.3.7.drv' failed
@ build-started /gnu/store/ywpxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv - x86_64-linux /var/log/guix/drvs/yw//pxpl2gp6z3sjvm7sn8gs6n4mhm2f0g-texlive-20160523b-texmf.tar.xz.drv.bz2

--

Sergei
-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQSZKa0VG5avZRlY01hxoe52YR/zqgUCWQ7bfwAKCRBxoe52YR/z
qgEqAJ9tlNFeHTMr59uH+ueIEs4JkNhdJgCeNpv+DeEteGnirZH3GWGxma658/g=
=QAGe
-----END PGP SIGNATURE-----


L
L
Leo Famulari wrote on 7 May 2017 20:25
(name . Sergei Trofimovich)(address . slyfox@inbox.ru)(address . 26811@debbugs.gnu.org)
20170507182543.GB6096@jasmine
On Sun, May 07, 2017 at 09:31:59AM +0100, Sergei Trofimovich wrote:
Toggle quote (7 lines)
> I have slow internet :)
>
> It looks like ftp://tug.org has a FTP session timeout of 2 hours 11 minutes, or something like that.
> Over night I tried to build sbcl and failed. Note how download silently stopped at ~80% [1].
>
> Would be nice if guix had a mechanism to reconnect to FTP and continue download after timeout.

In the meantime, you can download the files with a resumable tool like
wget, and then use `guix download file:///path/to/file` to work around
this missing feature. Note that both the client *and* the server must
support resuming downloads for this to work.
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlkPZqcACgkQJkb6MLrK
fwi9mhAA2DCfs3cCsz/B+6CquWTR8irPhT1Yt3ozcj1sGUVyYtEo4sZlWPkX4Ez9
q+jZhymrPoQ2QrViKNTY911ZPitcN2KvJbD0Qs8gFEqj3Nzh2E7IKyn3D+jVQaYZ
d+r7ih5LsLXL1Hn2c9aVOFQuvX2SQHQS3iQ0ZBCIfrHC3QgZWbuqGMwfN3Btf3XE
QkaHIlMxX793WxMxKxDYRELsglmDnRbrg5CKPNxNNqG7hoLYEuVAAhxBBwUP+7WX
9pHI6r14u6jMHjHt6VK8EYe+QkfCgxA5Trs2KeQ8clKLe5fzLk/pHhpjPDzJf9Ps
vjla5VzKAZaB1iymI9u8214n765iCmBXmc1VUd+lG8VnN1sxeXCad5aRCjBX4DDb
PJ/MkagnbOMLZGfrkiJvS/Sb4TyamQFiTMyzVGS5k/IV71IxCk7RDB3skBmJhjtl
CYfzgmNcgJ1XX/+1xASZczch2fzCgUfzS44zlR5QYVdVIKm7rKhNJF6SR6Q/auIx
/CGt3FOPlSn6PAOpXx+7sgo91b3w0AyUzvDhOYUqGXkF2oMoUSI5EhxA+FHsIMrm
VNKsqQENrtCQxNrjl4skTYirG7iWxDzY6lTEjCOIHXgx/Gpsip03qmQ05vDTKcjv
WG9jpMrNTKuxipCYFP2vBW1E8Ln/NSuAavoX4HyPbu0Gio3i/ok=
=pO2o
-----END PGP SIGNATURE-----


S
S
Sergei Trofimovich wrote on 7 May 2017 20:49
(name . Leo Famulari)(address . leo@famulari.name)(address . 26811@debbugs.gnu.org)
20170507194932.213380f8@sf
On Sun, 7 May 2017 14:25:43 -0400
Leo Famulari <leo@famulari.name> wrote:

Toggle quote (13 lines)
> On Sun, May 07, 2017 at 09:31:59AM +0100, Sergei Trofimovich wrote:
> > I have slow internet :)
> >
> > It looks like ftp://tug.org has a FTP session timeout of 2 hours 11 minutes, or something like that.
> > Over night I tried to build sbcl and failed. Note how download silently stopped at ~80% [1].
> >
> > Would be nice if guix had a mechanism to reconnect to FTP and continue download after timeout.
>
> In the meantime, you can download the files with a resumable tool like
> wget, and then use `guix download file:///path/to/file` to work around
> this missing feature. Note that both the client *and* the server must
> support resuming downloads for this to work.

The workaround worked.

I've noticed it took guix-daemon 2GB of RAM to process this file locally.
Does guix keep all the file in RAM until download finishes?

Thank you!

--

Sergei
-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQSZKa0VG5avZRlY01hxoe52YR/zqgUCWQ9sPAAKCRBxoe52YR/z
qkq6AJ9AcgPoLkED6BqIgeH1DoCoQRv5awCfbL4QzW39v89l8wiXRCUsm1LVV7k=
=8cHI
-----END PGP SIGNATURE-----


L
L
Leo Famulari wrote on 7 May 2017 20:58
(name . Sergei Trofimovich)(address . slyfox@inbox.ru)(address . 26811@debbugs.gnu.org)
20170507185817.GB27949@jasmine
On Sun, May 07, 2017 at 07:49:32PM +0100, Sergei Trofimovich wrote:
Toggle quote (21 lines)
> On Sun, 7 May 2017 14:25:43 -0400
> Leo Famulari <leo@famulari.name> wrote:
>
> > On Sun, May 07, 2017 at 09:31:59AM +0100, Sergei Trofimovich wrote:
> > > I have slow internet :)
> > >
> > > It looks like ftp://tug.org has a FTP session timeout of 2 hours 11 minutes, or something like that.
> > > Over night I tried to build sbcl and failed. Note how download silently stopped at ~80% [1].
> > >
> > > Would be nice if guix had a mechanism to reconnect to FTP and continue download after timeout.
> >
> > In the meantime, you can download the files with a resumable tool like
> > wget, and then use `guix download file:///path/to/file` to work around
> > this missing feature. Note that both the client *and* the server must
> > support resuming downloads for this to work.
>
> The workaround worked.
>
> I've noticed it took guix-daemon 2GB of RAM to process this file locally.
> Does guix keep all the file in RAM until download finishes?

I haven't looked, but that's how I'd expect it be coded. The OS can
handle paging things in and out as needed, right?
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAlkPbkgACgkQJkb6MLrK
fwjY2BAAuj9CyqbTveJQGTVTugc9TaOBSoBgIRV6tFzU7KYobEes5WRUSsrYWVqq
Lhxl+CsKaKiTQ7Gb3rfNb5hlnAfLKcfpCeSZ0M7+oWPJdB1t/hEYIoSFKmjpse1+
wUHMBVHnnQMOxtvE7HGxIq1/56e8A4SgvxGVNiSM1oTIL6+XyW+BJqiI3qedrkwi
CvAUNIoHjaqIND/b88TYIF4jstawyudk6kq3aXo3VgFoIQwQzK79K9B/MX3VWbqF
My9QSgNY4G866vsFmhp4pWcJ6lanC51lYB3GqTdVmR7rZHhv9VzVDr9f78Suyv3p
AtZFbrs/ihcQoFs7TidtTb3gQZmA67za3zqpRPegK75yfR5cH6U4qwwOD6roueIq
nJ1H6+w1lEl0029a0Pw9h7adqh2sXdS8yYF4YR6afLG/nnbl79a5Sho8670rpsbo
iyi3Ev37bSZJdEbcaBxfrkBZdemabBly0CvWeCopj5EaT9mQeyTwYRKTxMG1kuqy
C/kVwGqeT15St8llyzFPimepW9N7w8mK/RpJ+mhaaXqCTuHXiCniFt/ewB36ClDb
oMVoK877nR59CCYQJqoQHRUBZ/GvIPVH9ikQLm/zKLIn+GtCygp0MZISsK+AuIHq
hJ5gwa6iOOPpYaNawgpcKNfULpVJmBBm8HTc9ec3Sh7NOyYHDX8=
=Z9/j
-----END PGP SIGNATURE-----


L
L
Ludovic Courtès wrote on 8 May 2017 16:27
(name . Sergei Trofimovich)(address . slyfox@inbox.ru)(address . 26811@debbugs.gnu.org)
8760hbmnk5.fsf@gnu.org
Sergei Trofimovich <slyfox@inbox.ru> skribis:

Toggle quote (3 lines)
> Starting download of /gnu/store/xdjl5wxdjghq85nbc9qacgyzh8d54f6y-texlive-20160523b-texmf.tar.xz
> From ftp://tug.org/historic/systems/texlive/2016/texlive-20160523b-texmf.tar.xz...

Fundamentally, I think we should just fix our ‘texlive’ package such
that it doesn’t download 4G at once…

Ludo’.
?