Prioritize providing substitutes for security-critical packages with potentially long build times

  • Open
  • quality assurance status badge
Details
7 participants
  • Dr. Arne Babenhauserheide
  • Bengt Richter
  • chaosmonk
  • Leo Famulari
  • Ludovic Courtès
  • Ricardo Wurmus
  • zimoun
Owner
unassigned
Submitted by
chaosmonk
Severity
normal
C
C
chaosmonk wrote on 27 Aug 2020 22:50
(address . bug-guix@gnu.org)
2WPQFQ.3JQYOGZG7WXZ@riseup.net
ungoogled-chromium receives frequent security updates, so it is
important for users to keep it up-to-date. However, binary substitutes
for the latest version are usually not available, and it can take a
very long time to build from source, possibly multiple days on low-end
hardware. This might tempt or force some users to put off upgrading
the package and run an older, vulnerable version until a binary
substitute is available or they have a chance to set aside the uptime
needed to build from source.

I don't know what Guix's CI system looks like or how packages are
queued for building, but if there is a way to prioritize builds for
certain packages, I propose that substitutes for packages like
ungoogled-chromium should be built as soon as possible once there is a
new version. Other security-critical packages with potentially long
build times that come to mind are icecat and linux-libre.
L
L
Ludovic Courtès wrote on 10 Sep 2020 10:00
(name . chaosmonk)(address . chaosmonk@riseup.net)(address . 43075@debbugs.gnu.org)
87bliejc3j.fsf@gnu.org
Hi,

chaosmonk <chaosmonk@riseup.net> skribis:

Toggle quote (16 lines)
> ungoogled-chromium receives frequent security updates, so it is
> important for users to keep it up-to-date. However, binary
> substitutes for the latest version are usually not available, and it
> can take a very long time to build from source, possibly multiple
> days on low-end hardware. This might tempt or force some users to put
> off upgrading the package and run an older, vulnerable version until a
> binary substitute is available or they have a chance to set aside the
> uptime needed to build from source.
>
> I don't know what Guix's CI system looks like or how packages are
> queued for building, but if there is a way to prioritize builds for
> certain packages, I propose that substitutes for packages like
> ungoogled-chromium should be built as soon as possible once there is a
> new version. Other security-critical packages with potentially long
> build times that come to mind are icecat and linux-libre.

Thanks for your feedback. Our build farm has often been lagging behind
lately and that’s something we’ve been working on. The
ungoogled-chromium package is even more problematic because it takes
more than ~80 CPU-hours to build, and that often times out with our
current build farm settings (where we don’t allow builds to take more
than 6h, IIRC).

Right now we’re trying to improve build throughput in general but your
proposal makes sense, of course.

Thanks,
Ludo’.
Z
Z
zimoun wrote on 10 Sep 2020 11:19
(name . Ludovic Courtès)(address . ludo@gnu.org)
CAJ3okZ3_u1TWLRjM_di62W0AfvsNifWpvKyRyaWgh727Qk4HBg@mail.gmail.com
Hi,

On Thu, 10 Sep 2020 at 10:01, Ludovic Courtès <ludo@gnu.org> wrote:
Toggle quote (12 lines)
> chaosmonk <chaosmonk@riseup.net> skribis:

> > I don't know what Guix's CI system looks like or how packages are
> > queued for building, but if there is a way to prioritize builds for
> > certain packages, I propose that substitutes for packages like
> > ungoogled-chromium should be built as soon as possible once there is a
> > new version. Other security-critical packages with potentially long
> > build times that come to mind are icecat and linux-libre.

> Right now we’re trying to improve build throughput in general but your
> proposal makes sense, of course.

The recent updates of ungoogled-chromium do not mention [security
updates]. Well, I do not know if they are. So the question would be:
what triggers the special security build?

Well, the work-in-progress [1] about some metrics of Cuirass (Guix's
CI) would provide interesting answers on the concrete feasibility and
future improvements.



All the best,
simon
B
B
Bengt Richter wrote on 11 Sep 2020 02:47
(name . zimoun)(address . zimon.toutoune@gmail.com)
20200911004727.GA2910@LionPure
Hi,

On +2020-09-10 11:19:11 +0200, zimoun wrote:
Toggle quote (31 lines)
> Hi,
>
> On Thu, 10 Sep 2020 at 10:01, Ludovic Courtès <ludo@gnu.org> wrote:
> > chaosmonk <chaosmonk@riseup.net> skribis:
>
> > > I don't know what Guix's CI system looks like or how packages are
> > > queued for building, but if there is a way to prioritize builds for
> > > certain packages, I propose that substitutes for packages like
> > > ungoogled-chromium should be built as soon as possible once there is a
> > > new version. Other security-critical packages with potentially long
> > > build times that come to mind are icecat and linux-libre.
>
> > Right now we’re trying to improve build throughput in general but your
> > proposal makes sense, of course.
>
> The recent updates of ungoogled-chromium do not mention [security
> updates]. Well, I do not know if they are. So the question would be:
> what triggers the special security build?
>
> Well, the work-in-progress [1] about some metrics of Cuirass (Guix's
> CI) would provide interesting answers on the concrete feasibility and
> future improvements.
>
> [1] http://issues.guix.gnu.org/32548#1
>
>
> All the best,
> simon
>
>
>
Given


I would guess that any publicly visible coding meant to trigger special prioritized
security builds would feed the process described in [1].

Maybe that's insignificant compared to scraping commit notes and patches etc, idk.

HTH :)

--
Regards,
Bengt Richter
M
M
Mason Hock wrote on 11 Sep 2020 03:06
(address . 43075@debbugs.gnu.org)
C5K4Y29Y8DMA.12JKZ0JOKGQWE@libricia-thinkcentre
On Thu Sep 10, 2020 at 2:19 AM PDT, zimoun wrote:
Toggle quote (18 lines)
> Hi,
>
> On Thu, 10 Sep 2020 at 10:01, Ludovic Courtès <ludo@gnu.org> wrote:
> > chaosmonk <chaosmonk@riseup.net> skribis:
>
> > > I don't know what Guix's CI system looks like or how packages are
> > > queued for building, but if there is a way to prioritize builds for
> > > certain packages, I propose that substitutes for packages like
> > > ungoogled-chromium should be built as soon as possible once there is a
> > > new version. Other security-critical packages with potentially long
> > > build times that come to mind are icecat and linux-libre.
>
> > Right now we’re trying to improve build throughput in general but your
> > proposal makes sense, of course.
>
> The recent updates of ungoogled-chromium do not mention [security
> updates].

Security fixes are generally provided upstream by the Chromium devs, so
the place to look for them is not ungoogled-chromium's changelog, but
Chrome/Chromium's changelog.[1]

Toggle quote (3 lines)
> Well, I do not know if they are. So the question would be:
> what triggers the special security build?

For ungoogled-chromium, it is safe to assume that every new Chromium
release will contain security fixes. I'm not sure about a general
solution that would work for other packages. If Guix is tracking a
package's upstream VCS and upstream has a consistent commit message
format indicating security fixes, perhaps releases containing such
commits could trigger a security build. Otherwise I'm not sure.


Toggle quote (9 lines)
> Well, the work-in-progress [1] about some metrics of Cuirass (Guix's
> CI) would provide interesting answers on the concrete feasibility and
> future improvements.
>
> [1] http://issues.guix.gnu.org/32548#1
>
>
> All the best,
> simon
M
M
Mason Hock wrote on 11 Sep 2020 03:14
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43075@debbugs.gnu.org)
C5K545LNO64W.2PDZIYAKMIVOJ@libricia-thinkcentre
On Thu Sep 10, 2020 at 1:00 AM PDT, Ludovic Courtès wrote:
Toggle quote (27 lines)
> Hi,
>
> chaosmonk <chaosmonk@riseup.net> skribis:
>
> > ungoogled-chromium receives frequent security updates, so it is
> > important for users to keep it up-to-date. However, binary
> > substitutes for the latest version are usually not available, and it
> > can take a very long time to build from source, possibly multiple
> > days on low-end hardware. This might tempt or force some users to put
> > off upgrading the package and run an older, vulnerable version until a
> > binary substitute is available or they have a chance to set aside the
> > uptime needed to build from source.
> >
> > I don't know what Guix's CI system looks like or how packages are
> > queued for building, but if there is a way to prioritize builds for
> > certain packages, I propose that substitutes for packages like
> > ungoogled-chromium should be built as soon as possible once there is a
> > new version. Other security-critical packages with potentially long
> > build times that come to mind are icecat and linux-libre.
>
> Thanks for your feedback. Our build farm has often been lagging behind
> lately and that’s something we’ve been working on. The
> ungoogled-chromium package is even more problematic because it takes
> more than ~80 CPU-hours to build, and that often times out with our
> current build farm settings (where we don’t allow builds to take more
> than 6h, IIRC).

Yes, Chromium's build time is obscene. However, not providing
substitutes for it duplicates that problem to the machines of every Guix
user who uses ungoogled-chromium. In the time that it would take Guix's
build farm to build u-c it could probably build many other packages, but
users are in the exact same situation, so a substitute for u-c is likely
more valuable to them than substitutes for those other packages. If it
is possible to override the 6h timeout value for individual packages, I
think that it would be worth doing so for u-c, and perhaps for Icecat
and Linux-libre as well.

Toggle quote (5 lines)
> Right now we’re trying to improve build throughput in general but your
> proposal makes sense, of course.
>
> Thanks,
> Ludo’.
L
L
Ludovic Courtès wrote on 11 Sep 2020 08:53
(name . Mason Hock)(address . chaosmonk@riseup.net)(address . 43075@debbugs.gnu.org)
87d02s7qiu.fsf@gnu.org
Hi,

"Mason Hock" <chaosmonk@riseup.net> skribis:

Toggle quote (38 lines)
> On Thu Sep 10, 2020 at 1:00 AM PDT, Ludovic Courtès wrote:
>> Hi,
>>
>> chaosmonk <chaosmonk@riseup.net> skribis:
>>
>> > ungoogled-chromium receives frequent security updates, so it is
>> > important for users to keep it up-to-date. However, binary
>> > substitutes for the latest version are usually not available, and it
>> > can take a very long time to build from source, possibly multiple
>> > days on low-end hardware. This might tempt or force some users to put
>> > off upgrading the package and run an older, vulnerable version until a
>> > binary substitute is available or they have a chance to set aside the
>> > uptime needed to build from source.
>> >
>> > I don't know what Guix's CI system looks like or how packages are
>> > queued for building, but if there is a way to prioritize builds for
>> > certain packages, I propose that substitutes for packages like
>> > ungoogled-chromium should be built as soon as possible once there is a
>> > new version. Other security-critical packages with potentially long
>> > build times that come to mind are icecat and linux-libre.
>>
>> Thanks for your feedback. Our build farm has often been lagging behind
>> lately and that’s something we’ve been working on. The
>> ungoogled-chromium package is even more problematic because it takes
>> more than ~80 CPU-hours to build, and that often times out with our
>> current build farm settings (where we don’t allow builds to take more
>> than 6h, IIRC).
>
> Yes, Chromium's build time is obscene. However, not providing
> substitutes for it duplicates that problem to the machines of every Guix
> user who uses ungoogled-chromium. In the time that it would take Guix's
> build farm to build u-c it could probably build many other packages, but
> users are in the exact same situation, so a substitute for u-c is likely
> more valuable to them than substitutes for those other packages. If it
> is possible to override the 6h timeout value for individual packages, I
> think that it would be worth doing so for u-c, and perhaps for Icecat
> and Linux-libre as well.

Definitely, yes. I just meant to explain why the build farm often lacks
u-c substitutes currently, but I agree it must be addressed.

Ludo’.
L
L
Ludovic Courtès wrote on 11 Sep 2020 08:56
(name . zimoun)(address . zimon.toutoune@gmail.com)
878sdg7qej.fsf@gnu.org
Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (17 lines)
> On Thu, 10 Sep 2020 at 10:01, Ludovic Courtès <ludo@gnu.org> wrote:
>> chaosmonk <chaosmonk@riseup.net> skribis:
>
>> > I don't know what Guix's CI system looks like or how packages are
>> > queued for building, but if there is a way to prioritize builds for
>> > certain packages, I propose that substitutes for packages like
>> > ungoogled-chromium should be built as soon as possible once there is a
>> > new version. Other security-critical packages with potentially long
>> > build times that come to mind are icecat and linux-libre.
>
>> Right now we’re trying to improve build throughput in general but your
>> proposal makes sense, of course.
>
> The recent updates of ungoogled-chromium do not mention [security
> updates]. Well, I do not know if they are. So the question would be:
> what triggers the special security build?

To me the proposal is more about introducing scheduling priorities. For
these packages, it’s indeed safe to assume that every new release brings
security fixes.

Thanks,
Ludo’.
Z
Z
zimoun wrote on 11 Sep 2020 09:37
(name . Ludovic Courtès)(address . ludo@gnu.org)
CAJ3okZ30O-Y-1QyKymxwGh6WCcxKiqKizogiT=A5Sy76Ef3gVA@mail.gmail.com
Hi,

On Fri, 11 Sep 2020 at 08:56, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (8 lines)
> > The recent updates of ungoogled-chromium do not mention [security
> > updates]. Well, I do not know if they are. So the question would be:
> > what triggers the special security build?
>
> To me the proposal is more about introducing scheduling priorities. For
> these packages, it’s indeed safe to assume that every new release brings
> security fixes.

Why would some packages be prioritized on the build farm than others?
Based on what? Which criteria?
Popularity? But we do not measure (yet?) how many times a substitute
is downloaded.
For example, I do not use ungoogled-chromium so I would prefer that
the resources of the build farm would be spent on these X packages.
Bob and Alice, they would prefer these Y packages. How do we reach a
consensus?
And security is one criteria. But how to detect it is a security fix?

(Aside the issue of ungoogled-chromium about the time limit you
described; which should be fixed, obviously. :-))


I understand the annoyance and the frustration of the substitutes
availability but I am not convinced that some packages have higher
priority on the substitute delivery than others.

All the best,
simon
R
R
Ricardo Wurmus wrote on 11 Sep 2020 10:23
(name . zimoun)(address . zimon.toutoune@gmail.com)
87h7s43enr.fsf@elephly.net
zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (4 lines)
> I understand the annoyance and the frustration of the substitutes
> availability but I am not convinced that some packages have higher
> priority on the substitute delivery than others.

Hard to say. I think this discussion is a little premature given our
historic underutilization of the build farm hardware, which is very
often idle. Perhaps once the instrumentation of Cuirass has yielded
actionable paths to improving this we can reconsider if priorization is
still necessary or even feasible.

--
Ricardo
L
L
Leo Famulari wrote on 11 Sep 2020 15:39
(name . zimoun)(address . zimon.toutoune@gmail.com)
20200911133928.GB32741@jasmine.lan
On Fri, Sep 11, 2020 at 09:37:59AM +0200, zimoun wrote:
Toggle quote (4 lines)
> I understand the annoyance and the frustration of the substitutes
> availability but I am not convinced that some packages have higher
> priority on the substitute delivery than others.

In general, I agree with you, especially since most packages do not
really take very long to build if there are no substitutes available, or
they do not change very often.

However, I noticed today that we do not have substitutes for linux-libre
5.4.64, which was released upstream and added to Guix two days ago. In
my experience, we have rarely required users to build linux-libre, and I
think the current situation is a serious regression for our build farm.

We should prioritize building the kernel.
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAl9bfgoACgkQJkb6MLrK
fwjxFw//b5Dgxnzf07qAqDAJ6ZrXxNFBIoG/bDJ1PJQAv/aAXG6dqlbzQHXga7+d
cEtrXQWF1DyY47RCnNiu6Ap3/uCa+KeDPQM/gRzSQ2J0jIC2sbL7c9xLO/xB4+83
iVw92MYLamDqVcZnhiQovdvBZ4GrQ1WUR2ctM0fokcjbwVi/9GkMSVMfu/AF17Y3
1p1FyBd6pLPrbSVnE9VvyzCDXaQnCClOXnd3GLmyAy0DUcVFMujd0uY4poi8dQjT
G63uCvEtgZO7N0e6XRIGRSpdIhYS1qS54MRX2opu2lgOqHQh3gilxtuEf1MuAyKu
NmvCy5dJccqrzuxujWeyANw2wmFWqNKDCc7H6JWkaJEjZnP8JMp66D37iYQvXFfn
CTi6pUG9RAoC+V/212fqura/niBizrIRLpQVVaAYV1u9SE9cHy/4LvRoJ6lXY0a2
BQgexnD8j0YBSO8I3X2+l2PMzRgMPoSVW2jSjsyeYETCyBbhwbLsV3vilThEj1SM
DtlV7c2t7jct9jeJd0gGk2VKRLkbD9DXk4ZsFSh5iQ+b3AvSuKgPg2CatnwjjytU
ZEPQs3X/26qIdag68H/Sqa1UVaZ7D/9x2SV3Fzled1O9nMucrtW8s08pOV43tHnm
F/yfmCKntZaJakEkJtFsc6FKIpxQ/i/bVFpqwf9Y/4Qkt4yjRfc=
=ADWQ
-----END PGP SIGNATURE-----


D
D
Dr. Arne Babenhauserheide wrote on 11 Sep 2020 16:33
(name . zimoun)(address . zimon.toutoune@gmail.com)
87a6xwjsdf.fsf@web.de
zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (8 lines)
> On Fri, 11 Sep 2020 at 08:56, Ludovic Courtès <ludo@gnu.org> wrote:
>> To me the proposal is more about introducing scheduling priorities. For
>> these packages, it’s indeed safe to assume that every new release brings
>> security fixes.
>
> Why would some packages be prioritized on the build farm than others?
> Based on what? Which criteria?

There are two aspects that make ungoogled-chromium, icecat and
linux-libre special:

- long build time
- security critical

If a user cannot run the newest ungoogled-chromium, icecat, or
linux-libre due to too high build times (so it can for example only be
built on a weekend, but not on a weekday when the computer is only
active for a few hours), then this user is prone to be hit by zero-day
vulnerabilities.

So the minimal criterion would be: Protect users from zero-days.

For ungoogled-chromium, icecat, and linux-libre, two factors match:

- the chance is very high that an update fixes a vulnerability, and
- they take so long to build that many users won’t be able to do it
right away.

I certainly can’t: I cannot update ungoogled-chromium during work-time
because the compile is so heavy on resources, that it considerably slows
down my work.

Best wishes,
Arne
--
Unpolitisch sein
heißt politisch sein
ohne es zu merken
-----BEGIN PGP SIGNATURE-----

iQJEBAEBCAAuFiEE801qEjXQSQPNItXAE++NRSQDw+sFAl9bip8QHGFybmVfYmFi
QHdlYi5kZQAKCRAT741FJAPD6/rmD/4kag8/3+DZv59TTILgjjNV/0RnAKzEMmRJ
ZZV2fziszfjY9z67Jk2cRSbR8PL4UQsz93rj0lKJ+aPsY0bxaBQ3rqB+oC+aGhCB
6qnD5J9/2AQKfvrEQy075XkDvnm+sHicsNOLehr+DQffYGtWshv6kpAqqutL1Yvf
szSpumHaXv53iwhZg213yAoFCptv8yp+6nGU9KvVbGhgT+Tl4jHco0Er3UxcG6k/
dI2qN9yjFH89FgRfkUTJi+RXZLpis6TdgPizg+mmn6BzeSYTWEjR/np6eAub9WiX
hP3cDrJpPau0U5h1CytDwW4WNbe1V7IB+zkMVEzyTptLk0FuHEvxBxKYup4D+GAq
DYa02xCnssooas9CKai301u917IitmpZbitKk0Mp3aDCdZLgr3zFdrrFrL2nBpIA
xsRPwRFmJDuUdCb8bc4/dFnBApAxAvm217VCuDV0hIWBPq1CDvMe3ez4Lvwg7eG6
06Zt2fhrYOoifsxQx7BHO113aoY8qAu5bkTAZZsByXroRq8i11edi3JLJFrKNVm2
qVfdf/JEjCqxzADJPUu0quGLJUyJIGDVJwZ3f5ifRP786ihQc6Q46O7gLNf36DgF
4zmGFe7+q1R//w8OV1G+D+zuIfYenu9lvGJ18JuLwhfzlSNpXWBr3/JbEfvrzBDE
Q/WQGlN0YojEBAEBCAAuFiEE3Si95tmHXKvOSosd3M8NswvBBUgFAl9biqEQHGFy
bmVfYmFiQHdlYi5kZQAKCRDczw2zC8EFSEWMA/97UouOyqFm1RdVhZbediNf+UNn
4NCrRx1NwGQN6I3BADwj2EKmS7wDpQQK+qtiSYjw5ehYjJlgJ2L3UpHk7aPxW4pT
3CcE0tepj3l7kBmV/Somaou4aeBucUCaNyRXbDGdo6ZbgBP6fIVicccveLjj3WKv
Ig/y5EPKwe5oCdU08A==
=GM9R
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 11 Sep 2020 16:45
(name . zimoun)(address . zimon.toutoune@gmail.com)
87tuw4tlsh.fsf@gnu.org
Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (23 lines)
> On Fri, 11 Sep 2020 at 08:56, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> > The recent updates of ungoogled-chromium do not mention [security
>> > updates]. Well, I do not know if they are. So the question would be:
>> > what triggers the special security build?
>>
>> To me the proposal is more about introducing scheduling priorities. For
>> these packages, it’s indeed safe to assume that every new release brings
>> security fixes.
>
> Why would some packages be prioritized on the build farm than others?
> Based on what? Which criteria?
> Popularity? But we do not measure (yet?) how many times a substitute
> is downloaded.
> For example, I do not use ungoogled-chromium so I would prefer that
> the resources of the build farm would be spent on these X packages.
> Bob and Alice, they would prefer these Y packages. How do we reach a
> consensus?
> And security is one criteria. But how to detect it is a security fix?
>
> (Aside the issue of ungoogled-chromium about the time limit you
> described; which should be fixed, obviously. :-))

All we’re saying is that for some packages, we should always assume that
new releases bring security fixes. These are key packages like
Linux-libre, IceCat, ungoogled-chromium, etc.

Furthermore, ungoogled-chromium is practically not buildable on one’s
laptop, and thus it’s even more important to provide substitutes.

For now, the focus should be on improving overall build throughput since
there’s a lot of room for improvement.

Ludo’.
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 43075@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 43075
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch