Maxime Devos wrote 2 years ago
(address . bug-guix@gnu.org)
18d9e40b-15b0-c1c0-71c4-4e829f9df887@telenet.be
According to the manual (6.3.6 Substitution Failure):
> Even when a substitute for a derivation is available, sometimes the
> substitution attempt will fail.
Inconvenient, but expected.
> This can happen for a variety of reasons: the substitute server
> might be offline, the substitute may recently have been deleted,
So, according to this paragraph, it is possible that simultaneously
(1) a substitute is available
(2) the substitution attempt fails
(3) and the substitute server is offline
but (1) and (3) contradict each other -- if the substitute server is
offline, then from the client's perspective, the substitute might as
well not exist, so then substitutes are unavailable, and likewise, if
the substitute is deleted, it doesn't exist anymore, so it isn't available.
This inconsistency is the specification of Guix substitution is the
first bug I'd like to point out.
(The paragraph also mentions ‘the connection might have been
interrupted, etc.’, which I do not take any issue with.)
This is not only a documentation issue, because it impacts --fallback:
> When substitutes are enabled and a substitute for a derivation is
> available, but the substitution attempt fails, Guix will attempt to
> build the derivation locally depending on whether or not ‘--fallback’
> was given (*note common build option ‘--fallback’: fallback-option.
I think its reasonable that if a substitute is unavailable (*), then
Guix will built it locally. Often, this is already the case (e.g. think
of all the times you locally updated a package and prepared a patch),
yet this is apparently not always the case when the substitute server is
offline or when the substitute previously existed but was deleted.
This ‘Yet this is apparently not always the case when [...]’ is the
second bug.
(*) Unavailability caused by interruption is its own can of worms, which
will be covered separately.
Now, consider the following output of "guix home build [...]":
Toggle quote (1 lines)
> substitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het bijwerkesubstitute: lijst van substituten van ‘https://ci.guix.gnu.org’ aan het bijwerken... 100.0%
Third bug: unless you're being informal and putting the NL in nl_NL,
it's "bijwerken", not "bijwerke". Also, newlines are missing.
Looking at po/guix/nl.po, the translation does, in fact, write
"bijwerken" instead of "bijwerke" and it has the same format stuff at
the end as the English translation, so there appears to be a bug in the
code (instead of the translation).
Toggle quote (1 lines)
> 21,2 MB will be downloaded:
Fourth bug: inconsistent units -- in some substitute output, Guix uses
binary prefixes (see, e.g., #63784) and in other places it used metric
prefixes.
Toggle quote (29 lines)
> /gnu/store/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0
> /gnu/store/y1918l95a3wjij79456n9qdhlk8b8nja-python-qtconsole-5.3.0
> /gnu/store/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0
> /gnu/store/c536wjwfxdx3z7s9bwwhj1vngpq50aq6-python-ipyparallel-bootstrap-8.2.1
> /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0
> /gnu/store/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0
> /gnu/store/ifk6bsddap0smgq7x6lgkw3174hy29gz-python-ipython-8.5.0
> /gnu/store/dsd9dnf54571qggry3g5qwihx2f3v23m-python-jupyter-client-7.2.2
> /gnu/store/55zn484d18xdy0iarwcfxc5lxzsrjszf-qtconnectivity-5.15.8
> /gnu/store/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8
> /gnu/store/kzg742h88nd03yagry7ddhpk5plqhnlj-qtmultimedia-5.15.8
> /gnu/store/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8
> /gnu/store/0dxzxs9yza70ypmw2hw07s9dqghmlda3-qtlocation-5.15.8
> /gnu/store/nb5wjfx56bsw0m4bvgjh2aas4g3zzq2w-qtwebchannel-5.15.8
> /gnu/store/9vpx7hb9dg9r9410sg2r2lmw6yx82ixq-qtsensors-5.15.8
> /gnu/store/m1nw1w8qhkmnj4vmn97bwy6dvhi1awli-qttools-5.15.8
> /gnu/store/9rmh4afafhc77q16gbf1080ffdc7cz1l-qtxmlpatterns-5.15.8
> /gnu/store/fwkxw2ylbx0bd5vx1rnw1zkk40mfx0yn-qtwebsockets-5.15.8
> /gnu/store/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8
> /gnu/store/c9nq0sbciv9jcw02b61fp4xf82dnn53a-python-pyqt-5.15.9
> /gnu/store/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0
> /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 vervangen...
> /gnu/store/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0 vervangen...
> /gnu/store/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0 vervangen...
> /gnu/store/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0 vervangen...
> /gnu/store/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0 vervangen...
> /gnu/store/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8 vervangen...
> /gnu/store/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8 vervangen...
> /gnu/store/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8 vervangen...
(For context, these are the ‘substituting FOO’ messages)
Toggle quote (2 lines)
> guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0:de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is
(For context, this is the ‘server is somewhat slow’ message.)
Toggle quote (27 lines)
> retrying download of '/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0' with other substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0: de server is een beetje traag
> guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0: de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is
> retrying download of '/gnu/store/viqz0fvfy3bdspzl31s6qv27sma8cbhs-python-numpydoc-1.5.0' with other substitute URLs...
> retrying download of '/gnu/store/a9r3n52w9f224mlj7mgrw6avl45s3kxn-python-seaborn-0.12.0' with other substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8: de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is
> retrying download of '/gnu/store/1l3xi0xcsf09i2ilsbzbrfvpla0pgya8-qtdeclarative-5.15.8' with other substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8: de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is
> retrying download of '/gnu/store/j3i78xw93xa5fwsg7l84x6p7abfl0xh3-qtserialport-5.15.8' with other substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0: de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is
> retrying download of '/gnu/store/6x5rzyhqi2pchfb8w4nc7rs0jnxi5mil-python-ipykernel-6.13.0' with other substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8: de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is
> retrying download of '/gnu/store/zsxcgdxfr10k1f8jg32axk60wzi8fyny-qtsvg-5.15.8' with other substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0: de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is
> retrying download of '/gnu/store/gh1yr17j1amkjf653k8l31wqyzliqnmm-python-pytest-qt-3.3.0' with other substitute URLs...
> guix substitute: waarschuwing: tijdens het binnenhalen van https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0: de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het probleem hardnekkig is
> guix substitute: fout: failed to find alternative substitute for '/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0'
> vervanging van /gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0 mislukt
> guix home: fout: some substitutes for the outputs of derivation `/gnu/store/nvynln5vrrd8983slpais1c4dqn397j4-python-jupyter-client-7.2.2.drv' failed (usually happens due to networking issues); try `--fallback' to build derivation from source
OK, substituting ipykernel-bootstrap failed. Let's look what happened to
ipytkernel-bootstrap:
> 21,2 MB will be downloaded:
> [...]
>
/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0
> [...]
> [...]
>
/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0
vervangen...
> guix substitute: waarschuwing: tijdens het binnenhalen van
https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0:
de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het
probleem hardnekkig is
> retrying download of
'/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0'
with other substitute URLs...
> [...]
> guix substitute: waarschuwing: tijdens het binnenhalen van
https://ci.guix.gnu.org/nar/lzip/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0:
de server is een beetje traag
> guix substitute: waarschuwing: probeer ‘--no-substitutes’ als het
probleem hardnekkig is
> guix substitute: fout: failed to find alternative substitute for
'/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0'
> vervanging van
/gnu/store/y0qskmhmy4kzh02rlj2r238r7p39svwf-python-ipykernel-bootstrap-6.13.0
mislukt
> guix home: fout: some substitutes for the outputs of derivation
`/gnu/store/nvynln5vrrd8983slpais1c4dqn397j4-python-jupyter-client-7.2.2.drv'
failed (usually happens due to networking issues); try `--fallback' to
build derivation from source
I.e.:
1. Let's substitute python-pykernel-bootstrap-6.13.0
2. I'm downloading from ci.guix.gnu.org, but it's going a bit slow,
patience!
3. For some reason I'm not disclosing, let's download from
somewhere else instead.
4. Again, I'm downloading from ci.guix.gnu.org, but again it's
going a bit slow.
(Also, I lied, instead of downloading from somewhere else,
I'm downloading from the same location!)
5. I couldn't find any substitute at another location.
There are multiple bugs here:
5. The warning message is badly constructed:
‘during [...]: the server is a bit slow’ is technically English
I suppose, but there exist much better ways to phrase this
(e.g.: ‘Warning: the download of https://[...] is proceeding
slowly.’)
6. When substituting from ci.guix.gnu.org failed, it then used
the same location again as ‘other substitution location’ --
that message is a lie, and it also isn't going to accomplish
anything (except, maybe, by brute force).
7. It doesn't give information on why it (tries to) switch to
another substitution server: did the first server remove the
substitute that Guix expected? Is the TLS certificate out-of-date?
A time-out? A prematurely closed connection? A hash mismatch?
Depending on the cause, the user should do nothing or report
an issue to whoever manages the server.
My best guess is that Guix decided that the old server is too
slow, but it would be better if Guix actually said that,
e.g. ‘warning: the download of https://[...] is proceeding too
slowly; download aborted.’
8. It looks like no server had a substitute available
(or it was technically available, only very slowly, but for
practical purposes that counts as unavailable), so Guix
should have built it from source without having to add
--fallback.
About --fallback, I think that too much unexpected network behaviour is
only handled by --fallback, and that more --fallback behaviour should be
done unconditionally. I think that, by default, Guix should always
build from source when a substitute is unavailable, except when the
cause of the unavailability is a symptom of some kind of wrongness.
The option --fallback would then also build from source even when the
cause of unavailability is a symptom of wrongness.
More concretely:
* Being offline is inconvenient, but not wrong.
* A substitute being deleted isn't wrong either.
-- it might a symptom of TTL not being properly
implemented, but it could also just be the server
administrator going ‘OOps, I need more space, I'll
need to delete some substitutes earlier than expected.’
(That's kind of a wrongness, but it's a wrongness
that the client doesn't have to report.)
(Improperly implemented TTL is wrong IMO, but it's not
really problematic and Guix has bigger problems to handle.)
* Slowness can be caused by peering problems, but that's
not really anything that the client or server can do
much about, so it doesn't count as wrongness.
* I don't know about interrupted connections, because
I don't know what it usually is a symptom of. (There
can exist multiple causes, but I don't have a clue
about their frequencies.)
* Likewise, I don't know about timeouts.
* Hash mismatches count as wrongness (likely symptom
of FS corruption!)
* TLS errors: wrongness, likely indicates misconfiguration
or MITM attempt.
Best regards,
Maxime.