Prioritize providing substitutes for security-critical packages with potentially long build times

Open

Details

7 participants

Dr. Arne Babenhauserheide
Bengt Richter
chaosmonk
Leo Famulari
Ludovic Courtès
Ricardo Wurmus
zimoun

Owner: unassigned

Submitted by: chaosmonk

Severity: normal

chaosmonk wrote on 27 Aug 2020 22:50

Recipients:(address . bug-guix@gnu.org)

Message-ID:2WPQFQ.3JQYOGZG7WXZ@riseup.net

ungoogled-chromium receives frequent security updates, so it is 
important for users to keep it up-to-date.  However, binary substitutes 
for the latest version are usually not available, and it can take a  
very long time to build from source, possibly multiple days on low-end 
hardware.  This might tempt or force some users to put off upgrading 
the package and run an older, vulnerable version until a binary 
substitute is available or they have a chance to set aside the uptime 
needed to build from source.

I don't know what Guix's CI system looks like or how packages are 
queued for building, but if there is a way to prioritize builds for 
certain packages, I propose that substitutes for packages like 
ungoogled-chromium should be built as soon as possible once there is a 
new version.  Other security-critical packages with potentially long 
build times that come to mind are icecat and linux-libre.

Ludovic Courtès wrote on 10 Sep 2020 10:00

Recipients:(name . chaosmonk)(address . chaosmonk@riseup.net)(address . 43075@debbugs.gnu.org)

Message-ID:87bliejc3j.fsf@gnu.org

Hi,

chaosmonk <chaosmonk@riseup.net> skribis:

Toggle quote (16 lines)> ungoogled-chromium receives frequent security updates, so it is
> important for users to keep it up-to-date.  However, binary
> substitutes for the latest version are usually not available, and it
> can take a  very long time to build from source, possibly multiple
> days on low-end hardware.  This might tempt or force some users to put
> off upgrading the package and run an older, vulnerable version until a
> binary substitute is available or they have a chance to set aside the
> uptime needed to build from source.
>
> I don't know what Guix's CI system looks like or how packages are
> queued for building, but if there is a way to prioritize builds for
> certain packages, I propose that substitutes for packages like
> ungoogled-chromium should be built as soon as possible once there is a
> new version.  Other security-critical packages with potentially long
> build times that come to mind are icecat and linux-libre.

Thanks for your feedback.  Our build farm has often been lagging behind
lately and that’s something we’ve been working on.  The
ungoogled-chromium package is even more problematic because it takes
more than ~80 CPU-hours to build, and that often times out with our
current build farm settings (where we don’t allow builds to take more
than 6h, IIRC).

Right now we’re trying to improve build throughput in general but your
proposal makes sense, of course.

Thanks,
Ludo’.

zimoun wrote on 10 Sep 2020 11:19

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:CAJ3okZ3_u1TWLRjM_di62W0AfvsNifWpvKyRyaWgh727Qk4HBg@mail.gmail.com

Hi,

On Thu, 10 Sep 2020 at 10:01, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (12 lines)

> chaosmonk <chaosmonk@riseup.net> skribis:

> > I don't know what Guix's CI system looks like or how packages are

> > queued for building, but if there is a way to prioritize builds for

> > certain packages, I propose that substitutes for packages like

> > ungoogled-chromium should be built as soon as possible once there is a

> > new version. Other security-critical packages with potentially long

> > build times that come to mind are icecat and linux-libre.

> Right now we’re trying to improve build throughput in general but your

> proposal makes sense, of course.

The recent updates of ungoogled-chromium do not mention [security

updates]. Well, I do not know if they are. So the question would be:

what triggers the special security build?

Well, the work-in-progress [1] about some metrics of Cuirass (Guix's

CI) would provide interesting answers on the concrete feasibility and

future improvements.

[1] http://issues.guix.gnu.org/32548#1

All the best,

simon

Bengt Richter wrote on 11 Sep 2020 02:47

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:20200911004727.GA2910@LionPure

Hi,

On +2020-09-10 11:19:11 +0200, zimoun wrote:

Toggle quote (31 lines)> Hi,
> 
> On Thu, 10 Sep 2020 at 10:01, Ludovic Courtès <ludo@gnu.org> wrote:
> > chaosmonk <chaosmonk@riseup.net> skribis:
> 
> > > I don't know what Guix's CI system looks like or how packages are
> > > queued for building, but if there is a way to prioritize builds for
> > > certain packages, I propose that substitutes for packages like
> > > ungoogled-chromium should be built as soon as possible once there is a
> > > new version.  Other security-critical packages with potentially long
> > > build times that come to mind are icecat and linux-libre.
> 
> > Right now we’re trying to improve build throughput in general but your
> > proposal makes sense, of course.
> 
> The recent updates of ungoogled-chromium do not mention [security
> updates].  Well, I do not know if they are.  So the question would be:
> what triggers the special security build?
> 
> Well, the work-in-progress [1] about some metrics of Cuirass (Guix's
> CI) would provide interesting answers on the concrete feasibility and
> future improvements.
> 
> [1] http://issues.guix.gnu.org/32548#1
> 
> 
> All the best,
> simon
> 
> 
>

Given

[1] https://www.theregister.com/2020/09/04/linux_kernel_flaw_detection/

I would guess that any publicly visible coding meant to trigger special prioritized

security builds would feed the process described in [1].

Maybe that's insignificant compared to scraping commit notes and patches etc, idk.

HTH :)

Regards,

Bengt Richter

Mason Hock wrote on 11 Sep 2020 03:06

Recipients:(address . 43075@debbugs.gnu.org)

Message-ID:C5K4Y29Y8DMA.12JKZ0JOKGQWE@libricia-thinkcentre

On Thu Sep 10, 2020 at 2:19 AM PDT, zimoun wrote:

Toggle quote (18 lines)> Hi,
>
> On Thu, 10 Sep 2020 at 10:01, Ludovic Courtès <ludo@gnu.org> wrote:
> > chaosmonk <chaosmonk@riseup.net> skribis:
>
> > > I don't know what Guix's CI system looks like or how packages are
> > > queued for building, but if there is a way to prioritize builds for
> > > certain packages, I propose that substitutes for packages like
> > > ungoogled-chromium should be built as soon as possible once there is a
> > > new version.  Other security-critical packages with potentially long
> > > build times that come to mind are icecat and linux-libre.
>
> > Right now we’re trying to improve build throughput in general but your
> > proposal makes sense, of course.
>
> The recent updates of ungoogled-chromium do not mention [security
> updates].

Security fixes are generally provided upstream by the Chromium devs, so

the place to look for them is not ungoogled-chromium's changelog, but

Chrome/Chromium's changelog.[1]

Toggle quote (3 lines)

> Well, I do not know if they are. So the question would be:

> what triggers the special security build?

For ungoogled-chromium, it is safe to assume that every new Chromium

release will contain security fixes. I'm not sure about a general

solution that would work for other packages. If Guix is tracking a

package's upstream VCS and upstream has a consistent commit message

format indicating security fixes, perhaps releases containing such

commits could trigger a security build. Otherwise I'm not sure.

[1] https://chromereleases.googleblog.com/2020/08/stable-channel-update-for-desktop.html

Toggle quote (9 lines)> Well, the work-in-progress [1] about some metrics of Cuirass (Guix's
> CI) would provide interesting answers on the concrete feasibility and
> future improvements.
>
> [1] http://issues.guix.gnu.org/32548#1
>
>
> All the best,
> simon

Mason Hock wrote on 11 Sep 2020 03:14

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 43075@debbugs.gnu.org)

Message-ID:C5K545LNO64W.2PDZIYAKMIVOJ@libricia-thinkcentre

On Thu Sep 10, 2020 at 1:00 AM PDT, Ludovic Courtès wrote:

Toggle quote (27 lines)> Hi,
>
> chaosmonk <chaosmonk@riseup.net> skribis:
>
> > ungoogled-chromium receives frequent security updates, so it is
> > important for users to keep it up-to-date.  However, binary
> > substitutes for the latest version are usually not available, and it
> > can take a  very long time to build from source, possibly multiple
> > days on low-end hardware.  This might tempt or force some users to put
> > off upgrading the package and run an older, vulnerable version until a
> > binary substitute is available or they have a chance to set aside the
> > uptime needed to build from source.
> >
> > I don't know what Guix's CI system looks like or how packages are
> > queued for building, but if there is a way to prioritize builds for
> > certain packages, I propose that substitutes for packages like
> > ungoogled-chromium should be built as soon as possible once there is a
> > new version.  Other security-critical packages with potentially long
> > build times that come to mind are icecat and linux-libre.
>
> Thanks for your feedback. Our build farm has often been lagging behind
> lately and that’s something we’ve been working on. The
> ungoogled-chromium package is even more problematic because it takes
> more than ~80 CPU-hours to build, and that often times out with our
> current build farm settings (where we don’t allow builds to take more
> than 6h, IIRC).

Yes, Chromium's build time is obscene.  However, not providing
substitutes for it duplicates that problem to the machines of every Guix
user who uses ungoogled-chromium.  In the time that it would take Guix's
build farm to build u-c it could probably build many other packages, but
users are in the exact same situation, so a substitute for u-c is likely
more valuable to them than substitutes for those other packages.  If it
is possible to override the 6h timeout value for individual packages, I
think that it would be worth doing so for u-c, and perhaps for Icecat
and Linux-libre as well.

Toggle quote (5 lines)> Right now we’re trying to improve build throughput in general but your
> proposal makes sense, of course.
>
> Thanks,
> Ludo’.

Ludovic Courtès wrote on 11 Sep 2020 08:53

Recipients:(name . Mason Hock)(address . chaosmonk@riseup.net)(address . 43075@debbugs.gnu.org)

Message-ID:87d02s7qiu.fsf@gnu.org

Hi,

"Mason Hock" <chaosmonk@riseup.net> skribis:

Toggle quote (38 lines)> On Thu Sep 10, 2020 at 1:00 AM PDT, Ludovic Courtès wrote:
>> Hi,
>>
>> chaosmonk <chaosmonk@riseup.net> skribis:
>>
>> > ungoogled-chromium receives frequent security updates, so it is
>> > important for users to keep it up-to-date.  However, binary
>> > substitutes for the latest version are usually not available, and it
>> > can take a  very long time to build from source, possibly multiple
>> > days on low-end hardware.  This might tempt or force some users to put
>> > off upgrading the package and run an older, vulnerable version until a
>> > binary substitute is available or they have a chance to set aside the
>> > uptime needed to build from source.
>> >
>> > I don't know what Guix's CI system looks like or how packages are
>> > queued for building, but if there is a way to prioritize builds for
>> > certain packages, I propose that substitutes for packages like
>> > ungoogled-chromium should be built as soon as possible once there is a
>> > new version.  Other security-critical packages with potentially long
>> > build times that come to mind are icecat and linux-libre.
>>
>> Thanks for your feedback. Our build farm has often been lagging behind
>> lately and that’s something we’ve been working on. The
>> ungoogled-chromium package is even more problematic because it takes
>> more than ~80 CPU-hours to build, and that often times out with our
>> current build farm settings (where we don’t allow builds to take more
>> than 6h, IIRC).
>
> Yes, Chromium's build time is obscene.  However, not providing
> substitutes for it duplicates that problem to the machines of every Guix
> user who uses ungoogled-chromium.  In the time that it would take Guix's
> build farm to build u-c it could probably build many other packages, but
> users are in the exact same situation, so a substitute for u-c is likely
> more valuable to them than substitutes for those other packages.  If it
> is possible to override the 6h timeout value for individual packages, I
> think that it would be worth doing so for u-c, and perhaps for Icecat
> and Linux-libre as well.

Definitely, yes. I just meant to explain why the build farm often lacks

u-c substitutes currently, but I agree it must be addressed.

Ludo’.

Ludovic Courtès wrote on 11 Sep 2020 08:56

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:878sdg7qej.fsf@gnu.org

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (17 lines)> On Thu, 10 Sep 2020 at 10:01, Ludovic Courtès <ludo@gnu.org> wrote:
>> chaosmonk <chaosmonk@riseup.net> skribis:
>
>> > I don't know what Guix's CI system looks like or how packages are
>> > queued for building, but if there is a way to prioritize builds for
>> > certain packages, I propose that substitutes for packages like
>> > ungoogled-chromium should be built as soon as possible once there is a
>> > new version.  Other security-critical packages with potentially long
>> > build times that come to mind are icecat and linux-libre.
>
>> Right now we’re trying to improve build throughput in general but your
>> proposal makes sense, of course.
>
> The recent updates of ungoogled-chromium do not mention [security
> updates].  Well, I do not know if they are.  So the question would be:
> what triggers the special security build?

To me the proposal is more about introducing scheduling priorities.  For
these packages, it’s indeed safe to assume that every new release brings
security fixes.

Thanks,
Ludo’.

zimoun wrote on 11 Sep 2020 09:37

Recipients:(name . Ludovic Courtès)(address . ludo@gnu.org)

Message-ID:CAJ3okZ30O-Y-1QyKymxwGh6WCcxKiqKizogiT=A5Sy76Ef3gVA@mail.gmail.com

Hi,

On Fri, 11 Sep 2020 at 08:56, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (8 lines)

> > The recent updates of ungoogled-chromium do not mention [security

> > updates]. Well, I do not know if they are. So the question would be:

> > what triggers the special security build?

> To me the proposal is more about introducing scheduling priorities. For

> these packages, it’s indeed safe to assume that every new release brings

> security fixes.

Why would some packages be prioritized on the build farm than others?

Based on what? Which criteria?

Popularity? But we do not measure (yet?) how many times a substitute

is downloaded.

For example, I do not use ungoogled-chromium so I would prefer that

the resources of the build farm would be spent on these X packages.

Bob and Alice, they would prefer these Y packages. How do we reach a

consensus?

And security is one criteria. But how to detect it is a security fix?

(Aside the issue of ungoogled-chromium about the time limit you

described; which should be fixed, obviously. :-))

I understand the annoyance and the frustration of the substitutes

availability but I am not convinced that some packages have higher

priority on the substitute delivery than others.

All the best,

simon

Ricardo Wurmus wrote on 11 Sep 2020 10:23

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87h7s43enr.fsf@elephly.net

zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (4 lines)

> I understand the annoyance and the frustration of the substitutes

> availability but I am not convinced that some packages have higher

> priority on the substitute delivery than others.

Hard to say.  I think this discussion is a little premature given our
historic underutilization of the build farm hardware, which is very
often idle.  Perhaps once the instrumentation of Cuirass has yielded
actionable paths to improving this we can reconsider if priorization is
still necessary or even feasible.

-- 
Ricardo

Leo Famulari wrote on 11 Sep 2020 15:39

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:20200911133928.GB32741@jasmine.lan

On Fri, Sep 11, 2020 at 09:37:59AM +0200, zimoun wrote:

Toggle quote (4 lines)

> I understand the annoyance and the frustration of the substitutes

> availability but I am not convinced that some packages have higher

> priority on the substitute delivery than others.

In general, I agree with you, especially since most packages do not

really take very long to build if there are no substitutes available, or

they do not change very often.

However, I noticed today that we do not have substitutes for linux-libre

5.4.64, which was released upstream and added to Guix two days ago. In

my experience, we have rarely required users to build linux-libre, and I

think the current situation is a serious regression for our build farm.

We should prioritize building the kernel.

Dr. Arne Babenhauserheide wrote on 11 Sep 2020 16:33

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87a6xwjsdf.fsf@web.de

zimoun <zimon.toutoune@gmail.com> writes:

Toggle quote (8 lines)

> On Fri, 11 Sep 2020 at 08:56, Ludovic Courtès <ludo@gnu.org> wrote:

>> To me the proposal is more about introducing scheduling priorities. For

>> these packages, it’s indeed safe to assume that every new release brings

>> security fixes.

> Why would some packages be prioritized on the build farm than others?

> Based on what? Which criteria?

There are two aspects that make ungoogled-chromium, icecat and

linux-libre special:

- long build time

- security critical

If a user cannot run the newest ungoogled-chromium, icecat, or

linux-libre due to too high build times (so it can for example only be

built on a weekend, but not on a weekday when the computer is only

active for a few hours), then this user is prone to be hit by zero-day

vulnerabilities.

So the minimal criterion would be: Protect users from zero-days.

For ungoogled-chromium, icecat, and linux-libre, two factors match:

- the chance is very high that an update fixes a vulnerability, and

- they take so long to build that many users won’t be able to do it

right away.

I certainly can’t: I cannot update ungoogled-chromium during work-time

because the compile is so heavy on resources, that it considerably slows

down my work.

Best wishes,

Arne

Unpolitisch sein

heißt politisch sein

ohne es zu merken

Ludovic Courtès wrote on 11 Sep 2020 16:45

Recipients:(name . zimoun)(address . zimon.toutoune@gmail.com)

Message-ID:87tuw4tlsh.fsf@gnu.org

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

Toggle quote (23 lines)> On Fri, 11 Sep 2020 at 08:56, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> > The recent updates of ungoogled-chromium do not mention [security
>> > updates].  Well, I do not know if they are.  So the question would be:
>> > what triggers the special security build?
>>
>> To me the proposal is more about introducing scheduling priorities.  For
>> these packages, it’s indeed safe to assume that every new release brings
>> security fixes.
>
> Why would some packages be prioritized on the build farm than others?
> Based on what?   Which criteria?
> Popularity?  But we do not measure (yet?) how many times a substitute
> is downloaded.
> For example, I do not use ungoogled-chromium so I would prefer that
> the resources of the build farm would be spent on these X packages.
> Bob and Alice, they would prefer these Y packages.  How do we reach a
> consensus?
> And security is one criteria.  But how to detect it is a security fix?
>
> (Aside the issue of ungoogled-chromium about the time limit you
> described; which should be fixed, obviously. :-))

All we’re saying is that for some packages, we should always assume that

new releases bring security fixes. These are key packages like

Linux-libre, IceCat, ungoogled-chromium, etc.

Furthermore, ungoogled-chromium is practically not buildable on one’s

laptop, and thus it’s even more important to provide substitutes.

For now, the focus should be on improving overall build throughput since

there’s a lot of room for improvement.

Ludo’.

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 43075@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date