ci.guix.gnu.org is slow from my system

  • Open
  • quality assurance status badge
Details
6 participants
  • Julien Lepiller
  • Leo Famulari
  • Christopher Baines
  • Maxime Devos
  • raid5atemyhomework
  • zimoun
Owner
unassigned
Submitted by
raid5atemyhomework
Severity
normal
R
R
raid5atemyhomework wrote on 5 Mar 2021 12:22
(name . bug-guix@gnu.org)(address . bug-guix@gnu.org)
o8ZL0xD90cccAtyMQf2WaXcVlgCTjddqRp0u-xWYYxfncv-VGatQvHBMICTmnxofKzDm7-aiieRIICqY-nHK2RO91C6Z75qczNj_s0u3TaA=@protonmail.com
Downloading substitutes from ci.guix.gnu.org is slow from my two Guix-using computers.

One is a pure Guix System install without any channels, the other one is a foreign Guix install with the-channel-that-cannot-be-named.


```
wine64-6.0 54.4MiB 10KiB/s 01:55 [ ] 2.1%
```

This can be very slow, including as slow as 4KiB/s at times. This is fairly awful since sometimes I get this result:

```
wine64-6.0 54.4MiB 49KiB/s 06:01 [##### ] 31.9%guix substitute: error: TLS error in procedure 'read_from_session_record_port': Error decoding the received TLS packet.
substitution of /gnu/store/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 failed
guix package: error: some substitutes for the outputs of derivation `/gnu/store/wr9kf2bgcsvwxcmhnl9lf047nr8xcklc-wine64-6.0.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
```

Because of the failed incomplete download, I have to ***restart the download again from 0%***, which is awful, awful UX. It would be really nice if:

1. The guix download process would make an effort to ***retry*** this a few times.
2. Continue the previous download instead of restarting from 0%.

The problem is not on my ISP, or at least not solely on my ISP. Doing a `wget` from `github.com`:

```
Resolving github.com (github.com)... 13.229.188.59
Connecting to github.com (github.com)|13.229.188.59|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.108.154, 185.199.109.154, 185.199.110.154, ...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.108.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13114404 (13M) [application/octet-stream]
Saving to: ‘zfs-2.0.3.tar.gz’

zfs-2.0.3.tar.gz 100%[=====================================================================================================>] 12.51M 9.34MB/s in 1.3s

2021-03-05 17:39:49 (9.34 MB/s) - ‘zfs-2.0.3.tar.gz’ saved [13114404/13114404]

```

As can be seen above, I get 9.34MB/s elsewhere, which is better than the >60Mbit/s promised by my ISP.

It's possible that the problem is between my ISP and ci.guix.gnu.org. If I `wget` directly:

```
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57048561 (54M) [application/octet-stream]
Saving to: ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0’

1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 0%[ ] 167.79K 13.4KB/s eta 69m 20s^C
```

HOWEVER, if I `torify wget`:

```
Resolving ci.guix.gnu.org (ci.guix.gnu.org)... 141.80.181.40
Connecting to ci.guix.gnu.org (ci.guix.gnu.org)|141.80.181.40|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57048561 (54M) [application/octet-stream]
Saving to: ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.1’

1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.1 13%[============> ] 7.13M 683KB/s eta 72s ^C
```

Which is a big WHAT THE FUCK? Tor should not be 45x faster (683KB/s) than directly from my ISP (13KB/s). I am not using any special workarounds for Tor, like meek or obfs4, which makes me doubt that my ISP is attempting to censor --- if my ISP is the one doing the censoring, it would probably target Tor first before ci.guix.gnu.org, but I can access Tor just fine without special workarounds needed for very oppressive countries.

I've tried this a few times. Consistently, using `torify wget` is much faster than `wget` alone, I don't have to bang my head in boredom with `torify wget`. This is not my expectation given what I understand of the Tor network and the public network.

Is `ci.guix.gnu.org` doing some kind of per-IP or per-ISP or other throttling? This looks very suspicious given the massive speed difference when using Tor and not using Tor.

Doing `torify guix package -m manifest.scm` does not seem to perform the download inside Tor, which makes sense since it's probably `guix-daemon` that is doing the download.

So my questions are:

* WHY IS DIRECT DOWNLOAD FROM MY ISP SLOWER THAN TOR? What can I do to check if it's my ISP that is attempting to censor ci.guix.gnu.org?
* Is there a way to make `guix-daemon` use a Tor proxy? I have two systems using Guix, one is a Guix System, the other is using a foreign distro, and I'd like to adjust both to use Tor instead since it's faster.
* If not, is there a way to tell `guix-daemon` that I have these `nar` files and it can put them in its store somehow instead of me waiting for it to ***SLOOOOOOOOWLY*** download it?
* Are there any mirrors I can use for substitutes instead? How do I change my systems (both the Guix System and the foreign distro) to use the mirrors? Is there some configuration option to do this?
* How hard would it be to make the substitute download process at least make an effort to attempt to continue a failed download instead of failing immediately and forcing a restart from 0%? `texlive` is something like 400Mb, it takes hours at 10KiB/s, I once had it fail at >90% which was very painful because it had to restart from 0%, I had to do `guix package -m manifest.scm` in a loop over and over again until it finished download. It would probably cut down on people hammering on the server as well.

Thanks
raid5atemyhomework
C
C
Christopher Baines wrote on 5 Mar 2021 12:48
(name . raid5atemyhomework)(address . raid5atemyhomework@protonmail.com)(address . 46942@debbugs.gnu.org)
87o8fxn7mr.fsf@cbaines.net
raid5atemyhomework via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

Toggle quote (10 lines)
> Downloading substitutes from ci.guix.gnu.org is slow from my two Guix-using computers.
>
> One is a pure Guix System install without any channels, the other one is a foreign Guix install with the-channel-that-cannot-be-named.
>
>
> ```
> downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
> wine64-6.0 54.4MiB 10KiB/s 01:55 [ ] 2.1%
> ```

If you try this same download again, do you get the same speed?

guix publish, which is used on ci.guix.gnu.org can dynamically create
nars if they're not cached, which will probably result in slower
download speeds.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmBCGp5fFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XeaphAAotp1sc8bmjJe+4HmKG6w8Fckf7qeOCx1
8MWekR9pV1fi6/5ULUedMlAVL6XEsKaAYLw7+NWb83J4YBbMZFmvt6jTo19HJdaG
CIjgTSIMLvbdvwZFtFZDfTtxJkwoRZcRaPm4UMwy3IUrviHAu27olpDa6MygS4Ox
GWnEaJuuCBWMlHEAbuDLZ4Xprtih44nTXNX6FzvGLKql00UqVRlmpDK1a3AZbwg9
0U64M7eHKrIe0y0QuXHZvf0FGgEss/3B4QB14Cz1CaZD/bR0iHY2WxoZmNGQVdcZ
B+zYMKOBBBz9dF4QYtIHNfETWc7DIv9tdpOXEsaMB2nADnlvl0PRjKq9gFjNEwy+
5nmni+bH6MuNRqsjA3IEj3aaOPMCvGQuGMkBrS0UCXd2DHgvHvDcJyu5M6f8HnLB
kt8n9BnyiPX2uFU+713+0nvH39enrf67K7qcRfLo3IoDv38RxlbyGa1e9FiVEW56
HtqWcqESJzZd+VvpYIP2iw4fiHjCx6iBzM9arq5sLsIYiFqGhJzBYEXvlPXaJFSM
l3WK6vPZYyE4HOF5/Jvyao+1vo/YwluDE/iMNcM7PHHRuG/tmUt3EcGxmST2Z7ED
LoVtKYdh7XIh1vecQ2sUmxFaYca8MArlab9JTpYJPBJmxoQYxYUDgkHPd3sLGGW7
gGxnRHXH8x4=
=fuaX
-----END PGP SIGNATURE-----

R
R
raid5atemyhomework wrote on 5 Mar 2021 13:12
(name . Christopher Baines)(address . mail@cbaines.net)(name . 46942@debbugs.gnu.org)(address . 46942@debbugs.gnu.org)
D00R0PNVPHdI9gTJMtHRUkbwKIrlh1sZO_nRfHdT7xjLgaFwg4XyDlkfHxBTz4Pg00-LdKZNmDcyRqKu7BIt5m9wd9LF5-UVH-XZxFAqYh4=@protonmail.com
Hello Christopher,

Toggle quote (15 lines)
> raid5atemyhomework via Bug reports for GNU Guix bug-guix@gnu.org writes:
>
> > Downloading substitutes from ci.guix.gnu.org is slow from my two Guix-using computers.
> > One is a pure Guix System install without any channels, the other one is a foreign Guix install with the-channel-that-cannot-be-named.
> >
> > downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
> > wine64-6.0 54.4MiB 10KiB/s 01:55 [ ] 2.1%
> >
>
> If you try this same download again, do you get the same speed?
>
> guix publish, which is used on ci.guix.gnu.org can dynamically create
> nars if they're not cached, which will probably result in slower
> download speeds.

Yes, I have been suffering slow download speeds for a few weeks now. Even with repeated downloads, it is consistently slow. I only just now tried out using `torify wget`, which surprised me since it was much much faster --- one lucky download even got up to 6MB/s over frikking TOR before I ^Ced it. I've even tried doing `torify wget` and `wget` in parallel on two separate shells --- the `torify wget` was definitely faster, getting anywhere from 100kB/s to 700kB/s with a few outliers at >1MB/s, while `wget` was consistently below 40kB/s and eventually settling to ~11kB/s after a few seconds.

This holds even for packages I expect to be popular and thus likely to be in-cache with high probability, like `linux-libre`.

Thanks
raid5atemyhomework
R
R
raid5atemyhomework wrote on 5 Mar 2021 13:26
(name . 46942@debbugs.gnu.org)(address . 46942@debbugs.gnu.org)
gGJ5ufbP7AiW8HkFcWkPNmZl1dqnZ_74LuNAZ_Ey-y3TF8cDLEwQrK0P9aE7KmsZRZ0Lr32EiUB_0L8BD5MkoHdptYv49ImcU8_gaENiMRs=@protonmail.com
Toggle quote (2 lines)
> * Is there a way to make `guix-daemon` use a Tor proxy? I have two systems using Guix, one is a Guix System, the other is using a foreign distro, and I'd like to adjust both to use Tor instead since it's faster.

I saw that `guix-daemon` respects `http_proxy` and `https_proxy` envvars, but trying it out on my foreign-distro Guix computer, adding `https_proxy=socks5h://127.0.0.1:9050 http_proxy=socks5h://127.0.0.1:9050` to the `systemd` service file doesn't work.

```
guix substitute: error: TLS error in procedure 'handshake': The TLS connection was non-properly terminated.
substitution of /gnu/store/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 failed
guix package: error: some substitutes for the outputs of derivation `/gnu/store/wr9kf2bgcsvwxcmhnl9lf047nr8xcklc-wine64-6.0.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
```
Looking at the foreign distro's syslog:

```
Mar 5 19:52:03 developer guix-daemon[145182]: accepted connection from pid 145190, user raid5atemyhomework
Mar 5 19:52:05 developer guix-daemon[145200]: spurious SIGPOLL
Mar 5 19:52:07 developer Tor[1029]: Socks version 67 not recognized. (This port is not an HTTP proxy; did you want to use HTTPTunnelPort?)
```

So it looks to me that `guix-daemon` expects `https_proxy` to be an HTTPS proxy and not a SOCKS5/SOCKS5H proxy. I'll look into Tor's HTTPTunnelPort.

Thanks
raid5atemyhomework
R
R
raid5atemyhomework wrote on 5 Mar 2021 13:41
(name . 46942@debbugs.gnu.org)(address . 46942@debbugs.gnu.org)
B2rGBGXuPprLVfDpjZ9D8nyA6jXmGlXasaaPuk5XCsfizBziMqyN30BPDkTbvnTnYMBIPTPn52zh8XbqoBdKsukVvfDN2mbT4CfsmcvSFnw=@protonmail.com
Hi all,

Toggle quote (18 lines)
> > - Is there a way to make `guix-daemon` use a Tor proxy? I have two systems using Guix, one is a Guix System, the other is using a foreign distro, and I'd like to adjust both to use Tor instead since it's faster.
>
> I saw that`guix-daemon` respects `http_proxy` and `https_proxy` envvars, but trying it out on my foreign-distro Guix computer, adding `https_proxy=socks5h://127.0.0.1:9050 http_proxy=socks5h://127.0.0.1:9050` to the `systemd` service file doesn't work.
>
> guix substitute: error: TLS error in procedure 'handshake': The TLS connection was non-properly terminated.
> substitution of /gnu/store/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 failed
> guix package: error: some substitutes for the outputs of derivation `/gnu/store/wr9kf2bgcsvwxcmhnl9lf047nr8xcklc-wine64-6.0.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
>
>
> Looking at the foreign distro's syslog:
>
> Mar 5 19:52:03 developer guix-daemon[145182]: accepted connection from pid 145190, user raid5atemyhomework
> Mar 5 19:52:05 developer guix-daemon[145200]: spurious SIGPOLL
> Mar 5 19:52:07 developer Tor[1029]: Socks version 67 not recognized. (This port is not an HTTP proxy; did you want to use HTTPTunnelPort?)
>
>
> So it looks to me that`guix-daemon` expects `https_proxy` to be an HTTPS proxy and not a SOCKS5/SOCKS5H proxy. I'll look into Tor's HTTPTunnelPort.

On the foreign distro computer, adding an `HTTPTunnelPort 9080` to `/etc/tor/torrc` and then adding `http_proxy=https://127.0.0.1:9080https_proxy=https://127.0.0.1:9080` to `guix-daemon.service`, then restarting services, seems to work.

```
wine64-6.0 54.4MiB 579KiB/s 01:36 [##################] 100.0%

substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivation will be built:
/gnu/store/7mr17xka558smr0c76crf9g727ccj76g-profile.drv

3.2 MB will be downloaded
font-gnu-unifont-13.0.06 3.0MiB 515KiB/s 00:06 [##################] 100.0%

```

The above is significantly better than the previous runs where I get 11KiB/s, and matches the speeds I get from `torify wget`.

While it's a good ***workaround*** for my problem instead of me silently weeping at the ridiculous slowness of Guix substitutes, it doesn't solve my root problem:

* SOMETHING between my ISP and ci.guix.gnu.org is throttling access to the substitutes.
* Given that I have been using my ISP for a year without experiencing such spurious slowdowns, and I have been using ci.guix.gnu.org for the past few months only and have been hit with this slowness in the past month or so, I am more inclined to blame ci.guix.gnu.org, but please tell me how I can find out what is throttling the bandwidth here. The fact that Tor is ***FASTER*** is very suspicious.

Thanks
raid5atemyhomework
Z
Z
zimoun wrote on 5 Mar 2021 13:41
86a6rh7oyr.fsf@gmail.com
Hi,

On Fri, 05 Mar 2021 at 11:22, raid5atemyhomework via Bug reports for GNU Guix <bug-guix@gnu.org> wrote:

Toggle quote (2 lines)
> This can be very slow, including as slow as 4KiB/s at times.

[...]

Toggle quote (2 lines)
> The problem is not on my ISP, or at least not solely on my ISP. Doing a `wget` from `github.com`:

[...]

Toggle quote (2 lines)
> As can be seen above, I get 9.34MB/s elsewhere, which is better than the >60Mbit/s promised by my ISP.

Where are you located? I mean, ci.guix.gnu.org is in Berlin, so if you
are far, say in Alaska or in Puerto Williams[*], you get a poor
downloading rate. However, github.com probably uses CDN or something
like that, so obviously the rate becomes much better.


*Puerto Williams, the World Southest town with an Internet
connection. :-)


All the best,
simon
R
R
raid5atemyhomework wrote on 5 Mar 2021 14:00
(name . zimoun)(address . zimon.toutoune@gmail.com)(name . 46942@debbugs.gnu.org)(address . 46942@debbugs.gnu.org)
Tv5Og8T3E-ZDwSezIDZVRQrmcrz3u8TNXdJ25xZz-V7SOK6jpKVzFwKfwSQ3quUNMDqG4MhI5dGTNdxANnuDgujDCdB2CYBoLi3qKOLHQfA=@protonmail.com
Hi zimoun,

Toggle quote (22 lines)
> Hi,
>
> On Fri, 05 Mar 2021 at 11:22, raid5atemyhomework via Bug reports for GNU Guix bug-guix@gnu.org wrote:
>
> > This can be very slow, including as slow as 4KiB/s at times.
>
> [...]
>
> > The problem is not on my ISP, or at least not solely on my ISP. Doing a `wget` from `github.com`:
>
> [...]
>
> > As can be seen above, I get 9.34MB/s elsewhere, which is better than the >60Mbit/s promised by my ISP.
>
> Where are you located? I mean,ci.guix.gnu.org is in Berlin, so if you
> are far, say in Alaska or in Puerto Williams[*], you get a poor
> downloading rate. However, github.com probably uses CDN or something
> like that, so obviously the rate becomes much better.
>
> *Puerto Williams, the World Southest town with an Internet
> connection. :-)

I would rather not say, since I am a very private person (otherwise I wouldn't be using the "raid5atemyhomework" moniker).

What I **do** find strange is that ***Tor is faster***. Surely connecting directly from my ISP to ci.guix.gnu.org should be, in principle, faster than connecting to my ISP to a random node, which connects to another random node, which connects to another random node, which finally connects to Berlin, should be ***slower***?

Here are a few `ping`s to some places:

```
$ ping gnu.org
PING gnu.org (209.51.188.148) 56(84) bytes of data.
64 bytes from wildebeest.gnu.org (209.51.188.148): icmp_seq=1 ttl=50 time=301 ms
64 bytes from wildebeest.gnu.org (209.51.188.148): icmp_seq=2 ttl=50 time=306 ms
64 bytes from wildebeest.gnu.org (209.51.188.148): icmp_seq=3 ttl=50 time=335 ms
64 bytes from wildebeest.gnu.org (209.51.188.148): icmp_seq=4 ttl=50 time=257 ms
64 bytes from wildebeest.gnu.org (209.51.188.148): icmp_seq=5 ttl=50 time=234 ms
^C
--- gnu.org ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4226ms
rtt min/avg/max/mdev = 233.804/286.626/335.242/36.261 ms

$ ping www.vikings.net
PING www.vikings.net (168.119.169.112) 56(84) bytes of data.
64 bytes from v0.h.vkgs.io (168.119.169.112): icmp_seq=1 ttl=48 time=190 ms
64 bytes from v0.h.vkgs.io (168.119.169.112): icmp_seq=2 ttl=48 time=190 ms
64 bytes from v0.h.vkgs.io (168.119.169.112): icmp_seq=3 ttl=48 time=220 ms
64 bytes from v0.h.vkgs.io (168.119.169.112): icmp_seq=4 ttl=48 time=241 ms
64 bytes from v0.h.vkgs.io (168.119.169.112): icmp_seq=5 ttl=48 time=264 ms
^C
--- www.vikings.net ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4413ms
rtt min/avg/max/mdev = 189.767/220.941/264.326/29.054 ms

$ ping www.raptorcs.com
PING www.raptorcs.com (23.155.224.44) 56(84) bytes of data.
64 bytes from websvc.rptsys.com (23.155.224.44): icmp_seq=1 ttl=55 time=294 ms
64 bytes from websvc.rptsys.com (23.155.224.44): icmp_seq=2 ttl=55 time=307 ms
64 bytes from websvc.rptsys.com (23.155.224.44): icmp_seq=3 ttl=55 time=227 ms
^C
--- www.raptorcs.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2627ms
rtt min/avg/max/mdev = 226.868/275.933/306.748/35.071 ms
```

And here's a few select `wget`s:

```
Resolving store.vikings.net (store.vikings.net)... 185.199.141.17
Connecting to store.vikings.net (store.vikings.net)|185.199.141.17|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 246991 (241K) [image/jpeg]
Saving to: ‘kcmad8-1088x816.jpeg’

kcmad8-1088x816.jpeg 100%[=====================================================================================================>] 241.20K 19.2KB/s in 13s

2021-03-05 20:50:51 (19.2 KB/s) - ‘kcmad8-1088x816.jpeg’ saved [246991/246991]

Resolving store.vikings.net (store.vikings.net)... 185.199.141.17
Connecting to store.vikings.net (store.vikings.net)|185.199.141.17|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 246991 (241K) [image/jpeg]
Saving to: ‘kcmad8-1088x816.jpeg.1’

kcmad8-1088x816.jpeg.1 100%[=====================================================================================================>] 241.20K 293KB/s in 0.8s

2021-03-05 20:51:25 (293 KB/s) - ‘kcmad8-1088x816.jpeg.1’ saved [246991/246991]

Resolving static.fsf.org (static.fsf.org)... 209.51.188.233, 2001:470:142:5::233
Connecting to static.fsf.org (static.fsf.org)|209.51.188.233|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 40191351 (38M) [video/webm]
Saving to: ‘Fight-to-Repair-720p.webm’

Fight-to-Repair-720p.webm 2%[=> ] 856.00K 146KB/s eta 4m 32s ^C

Resolving static.fsf.org (static.fsf.org)... 209.51.188.233
Connecting to static.fsf.org (static.fsf.org)|209.51.188.233|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 40191351 (38M) [video/webm]
Saving to: ‘Fight-to-Repair-720p.webm.1’

Fight-to-Repair-720p.webm.1 6%[=====> ] 2.55M 498KB/s eta 92s ^C

Resolving static.raptorcs.com (static.raptorcs.com)... 23.155.224.44
Connecting to static.raptorcs.com (static.raptorcs.com)|23.155.224.44|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5398922 (5.1M) [image/png]
Saving to: ‘boardlarge.png’

boardlarge.png 100%[=====================================================================================================>] 5.15M 505KB/s in 9.4s

2021-03-05 20:57:30 (559 KB/s) - ‘boardlarge.png’ saved [5398922/5398922]

Resolving static.raptorcs.com (static.raptorcs.com)... 23.155.224.44
Connecting to static.raptorcs.com (static.raptorcs.com)|23.155.224.44|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5398922 (5.1M) [image/png]
Saving to: ‘boardlarge.png.1’

boardlarge.png.1 100%[=====================================================================================================>] 5.15M 534KB/s in 10s

2021-03-05 20:57:45 (515 KB/s) - ‘boardlarge.png.1’ saved [5398922/5398922]

```

I know Vikings is based in Germany, so it looks like my ISP does not like connecting to German sites --- again I am seeing this phenomenon where using Tor is faster than connecting directly, but it certainly now looks a whole lot more like a problem with my ISP.

On the other hand the `fsf.org` site has better speed when connected directly, but still the one via Tor is about 3x faster.

And finally RaptorCS is about the same speed both directly and over Tor. Again, this is still a surprise as I expect Tor to be slower than direct connection.

Lemme go poke at my ISP.

Thanks
raid5atemyhomework
R
R
raid5atemyhomework wrote on 5 Mar 2021 15:46
(name . zimoun)(address . zimon.toutoune@gmail.com)(name . 46942@debbugs.gnu.org)(address . 46942@debbugs.gnu.org)
AudwKJSxw0WHdt-_guxKICRlLck1NsyrFIYk00maDl9T7ayAkb2FVitR0VquGqbXHpY_1FmKokhoic0KWv0gsNfsgfJx1dyiGwvm7nZkYrw=@protonmail.com
Hi all,

In case it's useful, here's my `traceroute`, would this be helpful for Guix?

I erased some internal IP addresses and omit a few bytes off the source country. After hop 25 `traceroute` couldn't find anything anymore. I annotated the IP addresses to country map.

```
$ sudo `which traceroute` 141.80.181.40
traceroute to 141.80.181.40 (141.80.181.40), 64 hops max
1 <LAN> 3.140ms 0.947ms 0.981ms
2 <ISP INTERNAL> 2.527ms 1.571ms 1.416ms
3 <ISP INTERNAL> 4.318ms 4.705ms 3.680ms
4 <COUNTRY>.4.16 2.605ms 1.856ms 1.963ms
5 <COUNTRY>.1.205 2.177ms 2.087ms 1.876ms
6 <COUNTRY>.3.54 29.090ms 30.237ms 28.982ms
7 203.208.168.129 29.200ms 29.497ms 30.863ms # Singapore Singapore Telecommunications Pte Ltd
8 203.208.152.82 29.295ms 29.231ms 29.227ms # Singapore SingTel Internet Exchange
9 203.208.182.249 30.769ms 31.872ms 47.615ms # Singapore Singapore Telecommunications Pte Ltd
10 203.208.183.133 30.271ms 30.713ms 31.635ms # Singapore Singapore Telecommunications Pte Ltd
11 203.208.158.178 363.911ms 301.844ms 204.896ms # Singapore SingTel Internet Exchange
12 203.208.172.234 213.664ms 298.038ms 213.798ms # Singapore Singapore Telecommunications Pte Ltd
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 64.125.28.36 335.579ms 499.267ms 411.078ms # Netherlands Zayo Bandwidth
19 * * *
20 64.125.28.36 349.449ms 347.413ms 432.726ms # Netherlands Zayo Bandwidth
21 80.81.193.222 334.576ms 334.150ms 335.696ms # Germany DE-CIX Management GmbH
22 64.125.26.173 343.767ms 344.376ms 344.544ms # Germany Zayo Bandwidth
23 188.1.238.78 375.499ms 346.705ms 417.986ms # Germany Verein zur Foerderung eines Deutschen Forschungsnetzes e.V.
24 141.80.238.11 339.792ms 383.421ms 406.579ms # Germany Campus Berlin-Buch GmbH
25 188.1.146.210 410.225ms 334.844ms 335.196ms # Germany Verein zur Foerderung eines Deutschen Forschungsnetzes e.V.
```

The big increase in ping times is somewhere inside Singapore, which seems like a reasonable assumption that it's what is throttling bandwidth. A quick research shows Singapore Telecommunications Pte Ltd is the same as SingTel Internet Exchange.

Are there any other mirror substitute servers aside from `ci.guix.gnu.org`?

Thanks
raid5atemyhomework
Z
Z
zimoun wrote on 5 Mar 2021 19:39
(name . raid5atemyhomework)(address . raid5atemyhomework@protonmail.com)(name . 46942@debbugs.gnu.org)(address . 46942@debbugs.gnu.org)
86r1ktsaw3.fsf@gmail.com
Hi,

On Fri, 05 Mar 2021 at 14:46, raid5atemyhomework <raid5atemyhomework@protonmail.com> wrote:

Toggle quote (2 lines)
> Are there any other mirror substitute servers aside from `ci.guix.gnu.org`?

Well, only one in China I AFAIK.




All the best,
simon
L
L
Leo Famulari wrote on 5 Mar 2021 20:40
(name . raid5atemyhomework via Bug reports for GNU Guix)(address . bug-guix@gnu.org)(address . 46942@debbugs.gnu.org)
YEKJGlK1PyFulDx+@jasmine.lan
On Fri, Mar 05, 2021 at 11:22:17AM +0000, raid5atemyhomework via Bug reports for GNU Guix wrote:
Toggle quote (10 lines)
> Downloading substitutes from ci.guix.gnu.org is slow from my two Guix-using computers.
>
> One is a pure Guix System install without any channels, the other one is a foreign Guix install with the-channel-that-cannot-be-named.
>
>
> ```
> downloading from https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0 ...
> wine64-6.0 54.4MiB 10KiB/s 01:55 [ ] 2.1%
> ```

Sometimes we get reports like this, of strangely slow downloads from
ci.guix.gnu.org. They don't seem to depend on geography. They sometimes
happen to people who usually download quickly from ci.guix.gnu.org. So
far, we don't have any idea why they happen. But, it's not just you.

You are the first person to file a detailed bug report about it, if I
remember correctly (if!). If you'd like to help debug it, let us know,
but it would probably mean identifying your IP / location.
J
J
Julien Lepiller wrote on 5 Mar 2021 22:53
257938C2-93F2-41A9-A8E5-FF0F5A43DB5A@lepiller.eu
Actually, location usually has nothing to do with it (unless someone dug a hole right in the middle of optic fibers, but you'd notice). The most important info is AS (basically, which ISP you're using, but some have more than one AS). We don't need your IP, but ISP might be already too revealing for your taste.

Le 5 mars 2021 14:40:10 GMT-05:00, Leo Famulari <leo@famulari.name> a écrit :
Toggle quote (25 lines)
>On Fri, Mar 05, 2021 at 11:22:17AM +0000, raid5atemyhomework via Bug
>reports for GNU Guix wrote:
>> Downloading substitutes from ci.guix.gnu.org is slow from my two
>Guix-using computers.
>>
>> One is a pure Guix System install without any channels, the other one
>is a foreign Guix install with the-channel-that-cannot-be-named.
>>
>>
>> ```
>> downloading from
>https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
>...
>> wine64-6.0 54.4MiB
> 10KiB/s 01:55 [ ] 2.1%
>> ```
>
>Sometimes we get reports like this, of strangely slow downloads from
>ci.guix.gnu.org. They don't seem to depend on geography. They sometimes
>happen to people who usually download quickly from ci.guix.gnu.org. So
>far, we don't have any idea why they happen. But, it's not just you.
>
>You are the first person to file a detailed bug report about it, if I
>remember correctly (if!). If you'd like to help debug it, let us know,
>but it would probably mean identifying your IP / location.
Attachment: file
L
L
Leo Famulari wrote on 5 Mar 2021 23:12
(name . Julien Lepiller)(address . julien@lepiller.eu)(address . 46942@debbugs.gnu.org)
YEKs1Rd1vxNaPJij@jasmine.lan
Right, I meant that the required information would likely make it
possible to determine this person's location. Or at least make a good
guess.

On Fri, Mar 05, 2021 at 04:53:33PM -0500, Julien Lepiller wrote:
Toggle quote (28 lines)
> Actually, location usually has nothing to do with it (unless someone dug a hole right in the middle of optic fibers, but you'd notice). The most important info is AS (basically, which ISP you're using, but some have more than one AS). We don't need your IP, but ISP might be already too revealing for your taste.
>
> Le 5 mars 2021 14:40:10 GMT-05:00, Leo Famulari <leo@famulari.name> a �crit :
> >On Fri, Mar 05, 2021 at 11:22:17AM +0000, raid5atemyhomework via Bug
> >reports for GNU Guix wrote:
> >> Downloading substitutes from ci.guix.gnu.org is slow from my two
> >Guix-using computers.
> >>
> >> One is a pure Guix System install without any channels, the other one
> >is a foreign Guix install with the-channel-that-cannot-be-named.
> >>
> >>
> >> ```
> >> downloading from
> >https://ci.guix.gnu.org/nar/lzip/1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0
> >...
> >> wine64-6.0 54.4MiB
> > 10KiB/s 01:55 [ ] 2.1%
> >> ```
> >
> >Sometimes we get reports like this, of strangely slow downloads from
> >ci.guix.gnu.org. They don't seem to depend on geography. They sometimes
> >happen to people who usually download quickly from ci.guix.gnu.org. So
> >far, we don't have any idea why they happen. But, it's not just you.
> >
> >You are the first person to file a detailed bug report about it, if I
> >remember correctly (if!). If you'd like to help debug it, let us know,
> >but it would probably mean identifying your IP / location.
R
R
raid5atemyhomework wrote on 5 Mar 2021 23:57
(name . zimoun)(address . zimon.toutoune@gmail.com)
8bR3l7i5BPTQzCRF0BRGqsXMNe-DIM0QlFFj7qG-SJd7Z-KgPdvdlM5ZkhFlX8GVqQSocUgx9WCBthzbVrgZ5c2zMBegDmhCj-_1rjQ_oiY=@protonmail.com
Hi zimoun,

Toggle quote (13 lines)
> Hi,
>
> On Fri, 05 Mar 2021 at 14:46, raid5atemyhomework raid5atemyhomework@protonmail.com wrote:
>
> > Are there any other mirror substitute servers aside from `ci.guix.gnu.org`?
>
> Well, only one in China I AFAIK.
>
> https://mirrors.sjtug.sjtu.edu.cn/guix
>
> Seehttps://yhetil.org/guix/87czz24ilu.fsf@riseup.net for details.


Thank you very much, an experimental `wget` shows I get much higher speeds from this:

```
Resolving mirror.sjtu.edu.cn (mirror.sjtu.edu.cn)... 111.186.58.212, 2001:da8:8000:7100::322:a
Connecting to mirror.sjtu.edu.cn (mirror.sjtu.edu.cn)|111.186.58.212|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 57048561 (54M) [application/octet-stream]
Saving to: ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.3’

1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.3 100%[=====================================================================================================>] 54.41M 3.82MB/s in 17s

2021-03-06 06:45:31 (3.21 MB/s) - ‘1bdldr80p39g1mjnh76xw6hmwqrrb8lz-wine64-6.0.3’ saved [57048561/57048561]

```

I will try it out as the first server in `--substitute-urls`.

Would it not be appropriate to somehow put a list of such substitute servers (even if it is currently only a single mirror) somewhere official, like the Guix manual? I suppose if Peng Mei Yu thinks this mirror would be maintained long term?

Thanks
raid5atemyhomework
Z
Z
zimoun wrote on 6 Mar 2021 00:48
(name . raid5atemyhomework)(address . raid5atemyhomework@protonmail.com)
86ft19rwl4.fsf@gmail.com
Hi,

On Fri, 05 Mar 2021 at 22:57, raid5atemyhomework <raid5atemyhomework@protonmail.com> wrote:

Toggle quote (5 lines)
> Would it not be appropriate to somehow put a list of such substitute
> servers (even if it is currently only a single mirror) somewhere
> official, like the Guix manual? I suppose if Peng Mei Yu thinks this
> mirror would be maintained long term?

It was something discussed in the mentioned thread. But no one took the
time to send a patch for the manual or the cookbook. Feel free to do. :-)

Nice if it improves the situation for you.

Do you consider the problem (bug report) is solved? In order to close it.


Cheers,
simon
R
R
raid5atemyhomework wrote on 6 Mar 2021 01:23
(name . zimoun)(address . zimon.toutoune@gmail.com)
lGviRgmwsny7K6TMhFruJRyL-WG-ALbrPEIJ8X_qaEgTfC1fQDPUBxEvdVc-fATn7qRrX0BHMLDn8yCfQdJN4z7YjUgRinCdLzZxOSUxuwE=@protonmail.com
Hi zimoun,

Thanks, this is a definite improvement and I have set both systems to use it as the first item in `substitute-server`. I'll make a patch for the manual at least, then close this issue once that patch is accepted.

So I was thinking of modifying the installer so at least some page offers up options for mirrors. However, because of the way the installer works, it would have to be done by `(gnu installer services)`, which does not use `modify-services` on the base service list.

What the installer expects is that services will have their own `(service <type> <config>)` entry, without modifying the base service type.

What I *want* to do would be to have an extensible `guix-substitute-url-service-type`. Unfortunately the existing `guix-service-type` accepts a list of build directories to `chroot`.

So here's a sketch:

* Create a new `guix-daemon-service-type` and move most of the `guix-service-type` code into it.
* This is extensible; extensions provide a procedure which accepts a `<guix-configuration>` record and outputs a `<guix-configuration>` record.
* `(compose (cut apply compose <>))`
* `(extend (lambda (config modifier) (modifier config)))`
* Create a new `guix-build-chroot-service-type` which just extends `guix-daemon-service-type`:
* `(service-extension guix-daemon-service-type (lambda (build-chroots) (lambda (guix-config) (guix-configuration (inherit guix-config) (chroot-directories (append (guix-configuration-chroot-directories guix-config) build-chroots))))))`
* `(define-deprecated guix-service-type guix-build-chroot-service-type)`
* I mean seriously why does Guix assume only one configuration field of a base system service is usefully extensible, it seems to me that the general pattern should be that basic system service types should be extensible by a procedure that accepts an existing configuration record and returns a modified configuration record, then just define individual service types for each list-of-foo field of the configuration record to make a convenient way to extend such lists.
* Create a new `guix-substitute-url-service-type` similar to `guix-build-chroot-service-type`, which prepends a list of substitute URLs to the one in the configuration.

Then finally in the installer side:

* in `(gnu installer services)` add something like:
* `(system-service (name (G_ "https://mirrors.sjtug.sjtu.edu.cn/guix/(SJTUG, China)")) (type 'substitute-url) (snippet '((simple-service guix-substitute-url-service-type (list "https://mirrors.sjtug.sjtu.edu.cn/guix/")))))`
* `(system-service (name (G_ "https://ci.guix.gnu.org(Guix, Germany) - no mirror") (type 'substitute-url) (snippet '())))`
* In `(gnu installer newt services)`, add a page for substitute URL mirrors, probably a `run-listbox-selection-page`.
* `(G_ "You can select a mirror that is nearer to you for faster updating of packages. The main Guix substitute server, https://ci.guix.gnu.org/,will always be set as a fallback, so even if the mirror goes down, you can still upgrade from the main Guix substitute server.")`

Is that something that has any chance of getting accepted into Guix? I'd rather not do any coding unless this has any chance at all of getting merged in.

Thanks
raid5atemyhomework
Z
Z
zimoun wrote on 6 Mar 2021 01:42
(name . raid5atemyhomework)(address . raid5atemyhomework@protonmail.com)
864khpru3j.fsf@gmail.com
Hi,

On Sat, 06 Mar 2021 at 00:23, raid5atemyhomework <raid5atemyhomework@protonmail.com> wrote:
Toggle quote (7 lines)
> Hi zimoun,
>
> Thanks, this is a definite improvement and I have set both systems to
> use it as the first item in `substitute-server`. I'll make a patch for
> the manual at least, then close this issue once that patch is
> accepted.

Ok.


Toggle quote (6 lines)
> So I was thinking of modifying the installer so at least some page
> offers up options for mirrors. However, because of the way the
> installer works, it would have to be done by `(gnu installer
> services)`, which does not use `modify-services` on the base service
> list.

[...]

Toggle quote (4 lines)
> Is that something that has any chance of getting accepted into Guix?
> I'd rather not do any coding unless this has any chance at all of
> getting merged in.

From my point of view, I will get a better chance for an answer if you
ask on guix-devel with an appropriate subject, rather in (relatively)
long thread inside a bug report with a subject out this very
question. My 2 cent. ;-)


Cheers,
simon
R
R
raid5atemyhomework wrote on 15 Mar 2021 01:13
(name . zimoun)(address . zimon.toutoune@gmail.com)
U2_bWpJtXx-TvSWmCoXiEHqu4JSVRZdzKACg7Np2uK69o3TOgescc6iLx9d4RbHH77WgQBRqjZn0QibFUKfi-ew8YT35Ktbq9EzFtHO-QbU=@protonmail.com
Hello all,

Unfortunately, it seems that the SJTU server is somewhat unreliable, I sometimes get random failures in various parts talking to the substitute server, complaining of strange responses from the server:

```
Backtrace:
In guix/ui.scm:
2164:12 19 (run-guix-command _ . _)
In guix/scripts/substitute.scm:
652:2 18 (guix-substitute . _)
In unknown file:
17 (with-continuation-barrier #<procedure thunk ()>)
In ice-9/boot-9.scm:
1736:10 16 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
15 (apply-smob/0 #<thunk 7f5de06c9cc0>)
In ice-9/boot-9.scm:
1736:10 14 (with-exception-handler _ _ #:unwind? _ # _)
1736:10 13 (with-exception-handler _ _ #:unwind? _ # _)
1731:15 12 (with-exception-handler #<procedure 7f5dddeb8e70 at ic…> …)
In guix/scripts/substitute.scm:
701:17 11 (_)
410:7 10 (process-substitution _ "/gnu/store/yfzsz94qy92c7m9w0j…" …)
In ice-9/boot-9.scm:
1736:10 9 (with-exception-handler _ _ #:unwind? _ # _)
In guix/scripts/substitute.scm:
419:9 8 (_)
In ice-9/boot-9.scm:
1731:15 7 (with-exception-handler #<procedure 7f5dddeb8990 at ic…> …)
1669:16 6 (raise-exception _ #:continuable? _)
1667:16 5 (raise-exception _ #:continuable? _)
1669:16 4 (raise-exception _ #:continuable? _)
1764:13 3 (_ #<&compound-exception components: (#<&error> #<&irri…>)
1669:16 2 (raise-exception _ #:continuable? _)
1667:16 1 (raise-exception _ #:continuable? _)
1669:16 0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1669:16: In procedure raise-exception:
Throw to key `bad-response' with args `("Bad Response-Line: ~s" (""))'.
Backtrace:
In ice-9/boot-9.scm:
1736:10 4 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
3 (apply-smob/0 #<thunk 7f5de074e520>)
In ice-9/boot-9.scm:
718:2 2 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
619:8 1 (_ #(#(#<directory (guile-user) 7f5de0751c80>)))
In guix/ui.scm:
2164:12 0 (run-guix-command _ . _)

guix/ui.scm:2164:12: In procedure run-guix-command:
Throw to key `bad-response' with args `("Bad Response-Line: ~s" (""))'.
substitution of /gnu/store/yfzsz94qy92c7m9w0jbll7slc2pcap45-guix-packages-base failed
guix pull: error: corrupt input while restoring archive from #<closed: file 7f0dc2367bd0>
```

I configured `guix-daemon` to have two substitute URLs: SJTUG, and the official Cuirass in Berlin. My expectation is that if the SJTUG server fails, it should fall back on using the Cuirass server.

So my problems are two-fold:

* Apparently the SJTUG mirror sometimes acts in ways that aren't quite how Guix expects it. Either SJTUG or Guix has to be fixed so they talk in compatible manners.
* Multiple Substitute URLs are not useful for the mirroring configuration where one server is temporarily unable to properly serve; further servers are not checked.

I'm still not satisfied with this solution overall. Yes it's fast but it's imperfect.

I recently had to rebuild an OS (because I was dumb; the Guix language for shepherd services can easily lead you deadlocking shepherd itself) and had supreme difficulty reinstalling, because the Berlin server was too slow (would have taken ~10hours downloading everything) while the SJTUG server was unreliable and would consistently break the automated install. I had to `guix system build --fallback` separately and *then* `guix system init` manually.

Thanks
raid5atemyhomework
M
M
Maxime Devos wrote on 15 Mar 2021 08:21
d12e130d104378118e42e00906d1b7453b7615cf.camel@telenet.be
On Mon, 2021-03-15 at 00:13 +0000, raid5atemyhomework via Bug reports for GNU Guix wrote:
Toggle quote (7 lines)
> Hello all,
>
> [...]
> I recently had to rebuild an OS (because I was dumb; the Guix language
> for shepherd services can easily lead you deadlocking shepherd itself)
> and had supreme difficulty reinstalling, [...]

Reinstalling after a messed up configuration file shouldn't be necessary.
At least when using GRUB as bootloader, guix keeps some old (& presumably
not broken) system generations around, that can be selected when booting
from the bootloader. (I don't recall exactly how the menu is named,
maybe ‘Old system generations of $HOSTNAMES?)

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYIADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYE8K/BccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7rLyAQCjY1+ynWZu4dtcoemrhTrxbT5k
oK7tbX1vAjM4pTQfrAEAridUBgcbzQWDozD56gc5m04O6FGy2jL8nmkOAAYKdgw=
=fMem
-----END PGP SIGNATURE-----


R
R
raid5atemyhomework wrote on 15 Mar 2021 11:01
(name . Maxime Devos)(address . maximedevos@telenet.be)
Bzc3UqqEf-Z40v5LkTwMLOPNf5TuKM4PGiGoK67rUgzyn5wccw_BnbfBs4RZ1VUHdLyprDWXcfqifMO_zgoLZg8hmhL-XqkPWdmVCluLjT8=@protonmail.com
Hi Maxime,

Toggle quote (14 lines)
> On Mon, 2021-03-15 at 00:13 +0000, raid5atemyhomework via Bug reports for GNU Guix wrote:
>
> > Hello all,
> > [...]
> > I recently had to rebuild an OS (because I was dumb; the Guix language
> > for shepherd services can easily lead you deadlocking shepherd itself)
> > and had supreme difficulty reinstalling, [...]
>
> Reinstalling after a messed up configuration file shouldn't be necessary.
> At least when using GRUB as bootloader, guix keeps some old (& presumably
> not broken) system generations around, that can be selected when booting
> from the bootloader. (I don't recall exactly how the menu is named,
> maybe ‘Old system generations of $HOSTNAMES?)

Unfortunately I had a long-standing latent bug in my configuration file that triggered on a (persistent on-disk) edge case which would cause the shepherd process to enter an infinite loop (because the shepherd configuration language is Turing-complete enough to allow infinite loops in the first place). All the remaining generations (since I didn't like keeping more than a dozen, and had recently been excessively tweaking the configuration file) had this bug, so I had no way of reverting to an even older generation that predated the bug.

Thanks
raid5atemyhomework
R
R
raid5atemyhomework wrote on 15 Mar 2021 11:14
(name . Maxime Devos)(address . maximedevos@telenet.be)
z6nOXvNtoKQBgCIH0RxAjdfgS_C2_YkwftJ0puBjlrdolBkZ1ZgklW4qwg-LyMiHHLdUBgSdme5Z8WIA1qa7cQxJNI7FFTml37BI14FEklM=@protonmail.com
Toggle quote (19 lines)
> Hi Maxime,
>
> > On Mon, 2021-03-15 at 00:13 +0000, raid5atemyhomework via Bug reports for GNU Guix wrote:
> >
> > > Hello all,
> > > [...]
> > > I recently had to rebuild an OS (because I was dumb; the Guix language
> > > for shepherd services can easily lead you deadlocking shepherd itself)
> > > and had supreme difficulty reinstalling, [...]
> >
> > Reinstalling after a messed up configuration file shouldn't be necessary.
> > At least when using GRUB as bootloader, guix keeps some old (& presumably
> > not broken) system generations around, that can be selected when booting
> > from the bootloader. (I don't recall exactly how the menu is named,
> > maybe ‘Old system generations of $HOSTNAMES?)
>
> Unfortunately I had a long-standing latent bug in my configuration file that triggered on a (persistent on-disk) edge case which would cause the shepherd process to enter an infinite loop (because the shepherd configuration language is Turing-complete enough to allow infinite loops in the first place). All the remaining generations (since I didn't like keeping more than a dozen, and had recently been excessively tweaking the configuration file) had this bug, so I had no way of reverting to an even older generation that predated the bug.


And regardless, this kind of problem shouldn't occur in the first place.

* Instead of running the `start` code in the same process 1 (which is special enough that no amount of `kill -s SIGKILL 1` will work even if you manage to log into a console), `shepherd` should really run it in a separate process and monitor it if it's taking too long and possibly allow the operator to break out of it. Principle of least power and all that...
* If you want details: there is a shepherd service A that is a requirement of shepherd service B, however the daemon launched by A needed to reach a particular point in its initialization before B can start talking to it. B itself will fail to start if A has not reached that point in initialization. The extra code I added to the `start` of shepherd service A was to wait for that point of initialization before A was considered "started". It turned out it was buggy in that if the point was not reached in 1 second it would inadvertently enter an incorrect looping logic (ironically, the logic was supposed to exit it after 60 seconds, but I got increment/decrement crossed, meaning it would always loop as long as you never reached -60 seconds, which was impossible....) that ended up being an infinite loop and preventing process 1 from advancing. And this point was getting delayed when the process launched by A had to do a lot of (important) data on-disk that it needed to process at startup, so it was persistent on-disk data that would need > 1 second to process, thus ensuring that the buggy code would be entered.
* If this was a new computer it would also be just as screwed during installation anyway, you should consider this a fortuitous discovery of a latent bug.
* New users trying out Guix System that happen to get hit by this bug might very well decide that Guix is not stable enough for them to commit to using.

Thanks
raid5atemyhomework
R
R
raid5atemyhomework wrote on 16 Mar 2021 13:38
(name . Maxime Devos)(address . maximedevos@telenet.be)
RT2tVgwCg0_fz7MfLI4JCLLxvl2BYvKfyTyqbDXirTDwWEPqb1wEtsIokwbnFwwi_cGEQz4hkvxvVHhWvwXMapLdj0eXvxp4RDICArbs4DE=@protonmail.com
Here's another error!

For *this* instance notice the very slow download speed; other downloads got up to 2MiB/s. If my understanding is correct the SJTUG server is effectively a caching proxy, meaning that the low download speed here probably means that the SJTUG server itself is downloading from the Berlin Cuirass. As noted before, often an issue with using ci.guix.gnu.org is that for large multi-megabyte substitutes sometimes the Berlin server will just disconnect and EOF early, so I suspect something similar is happening to the SJTUG server.

```
ffmpeg-4.3.2 8.8MiB 13KiB/s 05:05 [####### ] 44.2%
Backtrace:
In guix/store/deduplication.scm:
227:2 19 (dump-file/deduplicate "/gnu/store/6fa3h0n957hr2nbd05y…" …)
In ice-9/ports.scm:
463:17 18 (call-with-output-file _ _ #:binary _ #:encoding _)
In guix/store/deduplication.scm:
232:10 17 (_ _)
In guix/serialization.scm:
261:6 16 (dump _)
247:20 15 (dump #<input: string 7fb2d8c83cb0> #<output: string 7…> …)
In unknown file:
14 (get-bytevector-n! #<input: string 7fb2d8c83cb0> # 0 #)
In guix/store/deduplication.scm:
203:22 13 (read! #vu8(85 110 107 110 111 119 110 32 112 105 99 …) …)
In unknown file:
12 (get-bytevector-n! #<input: string 7fb2d8ed4d90> # 0 #)
In gcrypt/hash.scm:
223:13 11 (read! #vu8(85 110 107 110 111 119 110 32 112 105 99 …) …)
In unknown file:
10 (get-bytevector-n! #<input: string 7fb2d8ed4e70> # 0 #)
In lzlib.scm:
501:4 9 (lzread! #<lz-decoder 7fb2d8c6bbf0> #<input: string 7f…> …)
In unknown file:
8 (get-bytevector-n #<input: string 7fb2d8ed4ee0> 65537)
In guix/progress.scm:
358:30 7 (read! _ _ _)
In unknown file:
6 (get-bytevector-n! #<input: string 7fb2d8c832a0> # 0 #)
In web/response.scm:
95:2 5 (read! _ _ _)
In ice-9/boot-9.scm:
1669:16 4 (raise-exception _ #:continuable? _)
1669:16 3 (raise-exception _ #:continuable? _)
1764:13 2 (_ #<&compound-exception components: (#<&error> #<&irri…>)
1669:16 1 (raise-exception _ #:continuable? _)
1669:16 0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1669:16: In procedure raise-exception:
Throw to key `bad-response' with args `("EOF while reading response body: ~a bytes of ~a" (4112173 9196598))'.
Backtrace:
In ice-9/boot-9.scm:
1736:10 4 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
3 (apply-smob/0 #<thunk 7fb2dec76520>)
In ice-9/boot-9.scm:
718:2 2 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
619:8 1 (_ #(#(#<directory (guile-user) 7fb2dec79c80>)))
In guix/ui.scm:
2164:12 0 (run-guix-command _ . _)

guix/ui.scm:2164:12: In procedure run-guix-command:
Throw to key `bad-response' with args `("EOF while reading response body: ~a bytes of ~a" (4112173 9196598))'.
substitution of /gnu/store/6fa3h0n957hr2nbd05ygjnk4idjq122q-ffmpeg-4.3.2 failed
guix upgrade: error: some substitutes for the outputs of derivation `/gnu/store/9xx9kbbjals2y7llhak65fnnyfh9rkyq-gnome-tweaks-3.34.1.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
```

Thanks
raid5atemyhomework
?