Evaluation comparison on cuirass

  • Open
  • quality assurance status badge
Details
4 participants
  • Andreas Enge
  • Josselin Poiret
  • Ludovic Courtès
  • Christopher Baines
Owner
unassigned
Submitted by
Andreas Enge
Severity
normal
A
A
Andreas Enge wrote on 10 May 2023 12:31
(address . bug-guix@gnu.org)
ZFtycPWNqiRCXGYX@jurong
When working on a branch and deciding whether to merge it, we need a way
of comparing its status with that of the master branch. As far as I can see,
there is currently no way in cuirass to compare arbitrary evaluations and
get a list (or a dashboard) of builds that fail in one, but not the other.

Andreas
J
J
Josselin Poiret wrote on 10 May 2023 20:27
87cz38doue.fsf@jpoiret.xyz
Hi Andreas,

Andreas Enge <andreas@enge.fr> writes:

Toggle quote (7 lines)
> When working on a branch and deciding whether to merge it, we need a way
> of comparing its status with that of the master branch. As far as I can see,
> there is currently no way in cuirass to compare arbitrary evaluations and
> get a list (or a dashboard) of builds that fail in one, but not the other.
>
> Andreas

I guess that this is one of the features that the Build Coordinator was
built for (and it is pretty damn good at this). Maybe we could start
considering whether it makes sense to duplicate effort on Cuirass and
the Build Coordinator? I don't know how "production-ready" the build
coordinator is, compared to Cuirass? Maybe we could target getting the
Build Coordinator up to feature parity with Cuirass so that it may be
used on a wider scale? If this is something we want to focus on, we
could create a team around it and set clear goals, which would probably
lessen the burden that's on Chris currently.

I understand that Cuirass is general enough to support much more than
Guix, but the coordinator is a wonderful piece of software and our
workflows might be outgrowing it.

WDYT?

--
Josselin Poiret
-----BEGIN PGP SIGNATURE-----

iQHEBAEBCgAuFiEEOSSM2EHGPMM23K8vUF5AuRYXGooFAmRb4fkQHGRldkBqcG9p
cmV0Lnh5egAKCRBQXkC5FhcaiiwuC/9eadFgdn45bz0YavoK/e5zmdWqySAcx56R
P3wwWVs1NVcRTCLExFFkDVKPrOfFMdvXM7He6u5h/JFCMCeyqd5DC4bDVMrMH7Tk
Frx1Zl85PyL0dNr8H242MyNSwPtDbWtZP3QGpHzzQV+FIJEx73h8m4v+WKK/B6ps
ae0qKcy3JiSjYSDf6FybGBlE52+ebH7CJ1BxCzjDJ2ip8Miwxt55SxNpBXpgXzCz
2Y+t3tVOs5GntlRJ8urR5iaqguZG5pOX5oM3OndAsVhCNqrevn88BLYLy2Fst8WU
srJzX1qtA2inUhYmtsDjx8W+IuaLKW0KGC0CLRGpELiRoHb6Uh8L9QOIA801Ek5M
laFzfaC9d8pIj3zlVc8bI8tSof3SBo4tkjB2QkWV/i1nCp59TTPZYBBNYuAuLIV3
v8FDmKI7LTp/YjrCg+5pYV8hM1rtSWiO2TnqS0WC2BUDTjC3DgZ83ccXLGDWEBQo
yxy60RbiODMVrXQrHe/1dLCXkpdTp+o=
=KIjm
-----END PGP SIGNATURE-----

C
C
Christopher Baines wrote on 11 May 2023 17:15
(name . Josselin Poiret)(address . dev@jpoiret.xyz)
878rduzxvr.fsf@cbaines.net
Josselin Poiret via Bug reports for GNU Guix <bug-guix@gnu.org> writes:

Toggle quote (25 lines)
> Hi Andreas,
>
> Andreas Enge <andreas@enge.fr> writes:
>
>> When working on a branch and deciding whether to merge it, we need a way
>> of comparing its status with that of the master branch. As far as I can see,
>> there is currently no way in cuirass to compare arbitrary evaluations and
>> get a list (or a dashboard) of builds that fail in one, but not the other.
>>
>> Andreas
>
> I guess that this is one of the features that the Build Coordinator was
> built for (and it is pretty damn good at this). Maybe we could start
> considering whether it makes sense to duplicate effort on Cuirass and
> the Build Coordinator? I don't know how "production-ready" the build
> coordinator is, compared to Cuirass? Maybe we could target getting the
> Build Coordinator up to feature parity with Cuirass so that it may be
> used on a wider scale? If this is something we want to focus on, we
> could create a team around it and set clear goals, which would probably
> lessen the burden that's on Chris currently.
>
> I understand that Cuirass is general enough to support much more than
> Guix, but the coordinator is a wonderful piece of software and our
> workflows might be outgrowing it.

There's some pedantic bits here to bring up. The build coordinator
doesn't have anything to do with comparing revisions (it doesn't even
really know what builds correspond to which revisions), it's just for
performing builds potentially across many machines, and doing something
useful with the results.

The data service however is meant for comparing revisions. There's a
circular relationship between the two as well, since the data service
can provide substitutes for derivations, which enables the build
coordinator to easily build them, and then report the results of those
builds back to the data service. This information about builds is
important since that can then factor in to comparisons between
revisions.

On the bit about "feature parity with Cuirass" though, this is a bit
misleading as the build coordinator exists because I wanted something
with very different design decisions to Cuirass. In terms of core
features, the build coordinator was complete back in late 2020
[1]. There's obviously lots that Cuirass does that the build coordinator
does not, but adding features without looking at the bigger picture can
be detrimental in the long term.


This is not to say there aren't things to work on in the build
coordinator. There are some ideas in the README and I'm more than happy
to try and help people get more involved.
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmRdCrhfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XdO9g/9GAAOMMuw2cm64zVQ0jin0a9+eT5g9149
kXVuX3JFLBcunIbJvEsgpknMyR7g4F8++qaFckSiX7ZwAhrP/MbawEs7GBrW6e3y
TYXc2rRv3ZsM3hGRIJKSKyJsSwknXeWJk0v9su+MphBM3BxTM2HdIL+LC8HrwvFf
Ml9zq0s/hLa8krfUDXpgb0oVvxaItjxdP+lMBzFsbWYlvmQAbMGYE+e6n2e2WC8+
AIl4pLbX1qS0MG/5o/FIcb+CwNETEFXkgDT/o8DvJ1APvF1MhzdfvkpSsSPNTt0D
CpcjS11BObLUKStYS6icacJ2x0Jz597psF0tgTq61elVYz4NyaRiyP0R7z4FwsF+
/DLX+TRw6xY0w7Mg3RMLi7mSLTxSCfT4s1P4+RvDOxZTxVRvfah7pkhyHdkDefvx
/rm+A3YbkLhhdKeLuFLLJwvS86I3tCM79cvOd/u8IUtynFgya5ugO+AykXUujUNF
SVCanh2gewp/WQYFQ7qpWzk456MXakq0/SmS7kBkXC8UEo6mBWLfja1nqcyVSPkY
3hhQUQAmrEFQ/f9j2FAenEMwEMIrAbtuiQUFSGr9xnLs9vtXyVpyU22NVAjoOaA+
jGnVl/kJgsQaQuv143AlZLg69mZK//4Zgl+CqQWPdR73esqd77uar5rkNr87jyKo
mQ2oRhDJ4jY=
=ImSd
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 29 Oct 2023 17:51
(name . Andreas Enge)(address . andreas@enge.fr)
87y1fl74nr.fsf@gnu.org
Hello,

Andreas Enge <andreas@enge.fr> skribis:

Toggle quote (5 lines)
> When working on a branch and deciding whether to merge it, we need a way
> of comparing its status with that of the master branch. As far as I can see,
> there is currently no way in cuirass to compare arbitrary evaluations and
> get a list (or a dashboard) of builds that fail in one, but not the other.

Going back to this, I agree with Josselin that the Data Service does an
excellent job at comparing the status of different revisions; I think we
should leverage that rather than try to implement something similar in
Cuirass.

Perhaps one thing we can improve though is the information flow from
Cuirass to the Data Service. ISTR that Christopher mentioned that the
Data Service couldn’t easily get information about substitute
availability from ci.guix, or something like that.

Is there still a problem here, Chris? If we need a new HTTP endpoint in
Cuirass to address that, I’m happy to give a hand.

Thanks,
Ludo’.
C
C
Christopher Baines wrote on 30 Oct 2023 08:32
(name . Ludovic Courtès)(address . ludo@gnu.org)
87cywwzh8l.fsf@cbaines.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (19 lines)
> Hello,
>
> Andreas Enge <andreas@enge.fr> skribis:
>
>> When working on a branch and deciding whether to merge it, we need a way
>> of comparing its status with that of the master branch. As far as I can see,
>> there is currently no way in cuirass to compare arbitrary evaluations and
>> get a list (or a dashboard) of builds that fail in one, but not the other.
>
> Going back to this, I agree with Josselin that the Data Service does an
> excellent job at comparing the status of different revisions; I think we
> should leverage that rather than try to implement something similar in
> Cuirass.
>
> Perhaps one thing we can improve though is the information flow from
> Cuirass to the Data Service. ISTR that Christopher mentioned that the
> Data Service couldn’t easily get information about substitute
> availability from ci.guix, or something like that.

Substitute availability is easy to get, but there's some difficulties
around build information.

Toggle quote (3 lines)
> Is there still a problem here, Chris? If we need a new HTTP endpoint in
> Cuirass to address that, I’m happy to give a hand.

The data service used to poll ci.guix.gnu.org for builds and build
status information, but I stopped that quite a while ago after the data
got messy when the Cuirass database was deleted and recreated. Since the
data service stores and uses the build IDs from Cuirass, it's confusing
when they're reused.

Anyway, even ignoring that, I'm unsure if polling worked well. That's
why I started to look at pushing data from Cuirass to the data
serivce. I did implement that, but the code on the Cuirass side was
never used. It's that same endpoint though that the build coordinator
uses to send build information to the data service (both instances).

The other blocker to making use of Cuirass data in the data service is
making sure it's high quality, in particular that if it says a build has
failed, I at least want to know it's started to build that
derivation. We don't want things showing up on QA as problems when it's
just Cuirass being unable to start builds.

Thanks,

Chris
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmU/XvpfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XfiHg//W82VccgptjuwiVAWk89dFf/myKvM0fX4
4V6ExW+B6G3Jp5a4Pq5jOoLmh8rOlc0rflKypfVqY1vCC3jddXEGmzaJnQ54Oa0s
ihEjjwz/2L2L5xhnla1joW2O0ESpPgxAuJyNBpDBV4YFgg3Z6e+5gI4WmRHPiECv
rMCiad5JBZBsknCjwTjqJ3ceUpdVOVeCVN/ArTRzeWHXeot4IL4YM1ky9EJL6vDz
0AR5te5YybUAtKLmgJXgywDKYJQDGgW/HCdtCI46goSP7k9BDtEsSuHyMiYDqox9
yzIr8dWWgwnMIiTzVgn4Mjy97PAuBcPxmurcxlpeXrrHMdVY0+MWtLE2q0BuxL2O
iWry73N1HR7sbgaJaGQSUZxG8ALPl57jzLIVSjbFkkaojIWdoJ+V8wNhR/NwpuP0
pFFIJyQ5vNHuE9HsVMJ8evCJxPCO1idvNHwixywlR4X71INWbG8rXyfG9fcVFmwo
qefmdexc4v+FhTU25mCsLn0ExS6Zsq2/4kuDiEO8RbwVMVb8k44T7BvfW3McBPKq
FK3ylPnZygheytK6AQq2uMdDoy/5sxiLowUO28y3Wf513JO7S1/DVO1xswRjYNH8
JTGxAMgDbqS6NyOCvoVWQDnJutAMWQnUUjx9XI4v2d0VHkAx2B6JcHT6FboRmkRu
YBZZYuq6MrA=
=prSg
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 5 Nov 2023 15:38
(name . Christopher Baines)(address . mail@cbaines.net)
87sf5k1czo.fsf@gnu.org
Hi,

Christopher Baines <mail@cbaines.net> skribis:

Toggle quote (6 lines)
> The data service used to poll ci.guix.gnu.org for builds and build
> status information, but I stopped that quite a while ago after the data
> got messy when the Cuirass database was deleted and recreated. Since the
> data service stores and uses the build IDs from Cuirass, it's confusing
> when they're reused.

Ah yes, that’s a problem. Maybe it should have UUIDs or something in
addition to those database IDs; or maybe the Data Service can use, say,
derivation + ID as a unique ID for Cuirass builds?

[...]

Toggle quote (6 lines)
> The other blocker to making use of Cuirass data in the data service is
> making sure it's high quality, in particular that if it says a build has
> failed, I at least want to know it's started to build that
> derivation. We don't want things showing up on QA as problems when it's
> just Cuirass being unable to start builds.

Indeed. :-) Well, I do hope that status = failed really means build
failure; seems we’re not completely done with the infamous “missing
.drv” bug though, and that’s erroneously reported as “failed”.

Ludo’.
?