[PATCH] doc: Add "Contributing to Guix's infrastructure".

  • Open
  • quality assurance status badge
Details
4 participants
  • Greg Hogan
  • Ekaitz Zarraga
  • Ludovic Courtès
  • Nicolas Graves
Owner
unassigned
Submitted by
Ekaitz Zarraga
Severity
normal
E
E
Ekaitz Zarraga wrote on 27 Oct 13:45 +0100
(address . guix-patches@gnu.org)
881dbbde93521a4f6957cd795e6942c43103f688.1730033134.git.ekaitz@elenq.tech
Use the "Call for contribution to the Guix infrastructure" by Ludovic
Courtès to create a section in the documentation that describes how to
contribute to the infrastructure.


* doc/contributing.texi (Contributing to Guix's infrastructure): New
section.

Change-Id: I3f3a99ad884110cc8323789e8c14bec1f7327e97
---
doc/contributing.texi | 187 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 187 insertions(+)

Toggle diff (206 lines)
diff --git a/doc/contributing.texi b/doc/contributing.texi
index acdc303be6..1510e07ddc 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -37,6 +37,7 @@ Contributing
* Deprecation Policy:: Commitments and tools for deprecation.
* Writing Documentation:: Improving documentation in GNU Guix.
* Translating Guix:: Make Guix speak your native language.
+* Contributing to Guix's infrastructure:: Make Guix ecosystem work better.
@end menu
@node Requirements
@@ -3594,3 +3595,189 @@ Translating Guix
be updated accordingly (see @file{website/i18n-howto.txt} for more
information on the process).
@end itemize
+
+
+@cindex infrastructure
+@node Contributing to Guix's infrastructure
+@section Contributing to Guix's infrastructure
+
+Since its inception, the Guix project has always valued its autonomy, and that
+reflects in its infrastructure: our servers run Guix System and exclusively
+free software, none of them is hosted by one of these transnational companies,
+and they're administered by volunteers.
+
+Of course this comes at a cost and this is why we're sending this call for
+contributions. Our hope is to make infrastructure-related activity more
+legible so that maybe you can picture yourself helping in one of these areas.
+
+
+@menu
+* Coding::
+* System administration::
+* Day-to-day system administration::
+* On-site intervention::
+* Hosting::
+* Administrative tasks::
+@end menu
+
+@node Coding
+@subsection Coding
+
+Guix runs many Guix-specific services; this is all lovely Scheme code but it
+tends to receive less attention than Guix itself:
+
+@itemize
+@item Build Farm Front-End: @url{https://git.cbaines.net/guix/bffe}
+@item Cuirass: @url{https://guix.gnu.org/cuirass/}
+@item Goggles (IRC logger):
+@url{https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/goggles.scm}
+@item Guix Build Coordinator:
+@url{https://git.savannah.gnu.org/cgit/guix/build-coordinator.git/}
+@item Guix Data Service:
+@url{https://git.savannah.gnu.org/git/guix/data-service.git/}
+@item Guix Packages Website:
+@url{https://codeberg.org/luis-felipe/guix-packages-website.git}
+@item mumi: @url{https://git.savannah.gnu.org/cgit/guix/mumi.git/}
+@item nar-herder: @url{https://git.savannah.gnu.org/cgit/guix/nar-herder.git/}
+@item QA Frontpage: @url{https://git.savannah.gnu.org/git/guix/qa-frontpage.git}
+@end itemize
+
+There is no time constraint on this coding activity: any improvement is
+welcome, whenever it comes. Most of these code bases are relatively small,
+which should make it easier to get started.
+
+Prerequisites: Familiarity with Guile, HTTP, and databases.
+
+If you wish to get started, check out the README of the project of your choice
+and get in touch with guix-devel and the primary developer(s) of the tool as
+per @code{git shortlog -s | sort -k1 -n}.
+
+@node System administration
+@subsection System administration
+
+Guix System configuration for all our systems is held in this repository:
+
+@url{https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/}
+
+The two front-ends are @file{berlin.scm} (the machine behind ci.guix.gnu.org)
+and @file{bayfront.scm} (the machine behind bordeaux.guix.gnu.org,
+guix.gnu.org, hpc.guix.info, qa.guix.gnu.org, and more). Both connect to a
+number of build machines and helpers.
+
+Without even having SSH access to the machine, you can help by posting patches
+to improve the configuration (you can test it with @code{guix system vm}).
+Here are ways you can help:
+
+@itemize
+@item
+Improve infra monitoring: set up a dashboard to monitor all the infrastructure,
+and an out-of-band channel to communicate about downtime.
+
+@item
+Implement web site redundancy: guix.gnu.org should be backed by several
+machines on different sites. Get in touch with us and/or send a patch!
+
+@item
+Implement substitute redundancy: likewise, bordeaux.guix.gnu.org and
+ci.guix.gnu.org should be backed by several head nodes.
+
+@item
+Improve backup: there's currently ad-hoc backup of selected pieces over rsync
+between the two head nodes; we can improve on that, for example with a
+dedicated backup site and proper testing of recoverability.
+
+@item
+Support mirroring: We'd like to make it easy for others to mirror substitutes
+from ci.guix and bordeaux.guix, perhaps by offering public rsync access.
+
+@item
+Optimize our web services: Monitor the performance of our services and tweak
+nginx config or whatever it takes to improve it.
+
+There is no time constraint on this activity: any improvement is welcome,
+whenever you can work on it.
+
+Prerequisite: Familiarity with Guix System administration and ideally with the
+infrastructure handbook:
+
+@url{https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org}
+
+@end itemize
+
+@node Day-to-day system administration
+@subsection Day-to-day system administration
+
+We're also looking for people who'd be willing to have SSH access to some of
+the infrastructure to help with day-to-day maintenance: restarting a build,
+restarting the occasional service that has gone wild (that can happen),
+reconfiguring/upgrading a machine, rebooting, etc.
+
+This day-to-day activity requires you to be available some of the time (during
+office hours or not, during the week-end or not), whenever is convenient for
+you, so you can react to issues reported on IRC, on the mailing list, or
+elsewhere, and synchronize with other sysadmins.
+
+Prerequisite: Being a “known” member of the community, familiarity with Guix
+System administration, with some of the services/web sites being run, and with
+the infrastructure handbook:
+
+@url{https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org}
+
+@node On-site intervention
+@subsection On-site intervention
+
+The first front-end is currently generously hosted by the Max Delbrück Center
+(MDC), a research institute in Berlin, Germany. Only authorized personnel can
+physically access it.
+
+The second one, bordeaux.guix.gnu.org, is hosted in Bordeaux, France, in a
+professional data center shared with non-profit ISP Aquilenet. If you live in
+the region of Bordeaux and would like to help out when we need to go on-site,
+please make yourself known by emailing @email{guix-sysadmin@@gnu.org}.
+
+On-site interventions are rare, but they're usually in response to an
+emergency.
+
+@node Hosting
+@subsection Hosting
+
+We're looking for people who can host machines and help out whenever
+physical access is needed. More specifically:
+
+@itemize
+@item
+We need hosting of “small” machines such as single-board computers (AArch64,
+RISC-V) for use as build machines.
+
+@item
+We need hosting for front-ends and x86_64 build machines in a data center where
+they can be racked and where, ideally, several local Guix sysadmins can
+physically access them.
+@end itemize
+
+The machines should be accessible over Wireguard VPN most of the
+time, so longer power or network interruptions should be the
+exception.
+
+Prerequisites: Familiarity with installing and remotely administering Guix
+System.
+
+@node Administrative tasks
+@subsection Administrative tasks
+
+The infra remains up and running thanks to crucial administrative tasks, which
+includes:
+
+@itemize
+@item
+Selecting and purchasing hardware, for example build machines.
+@item
+Renewing domain names.
+
+@item
+Securing funding, in particular via the Guix Foundation:
+@url{https://foundation.guix.info}
+@end itemize
+
+Prerequisites: Familiarity with hardware, and/or DNS registrars,
+and/or sponsorship, and/or crowdfunding.

base-commit: 091131af64fd4e4e925fff829fa19097cfcdfcc5
--
2.46.0
G
G
Greg Hogan wrote on 28 Oct 15:50 +0100
(name . Ekaitz Zarraga)(address . ekaitz@elenq.tech)
CA+3U0Zn+q7f7XhMhJ=_VB4Q8MkTk6WpTQKOL4CJhFzu4UP1Wcw@mail.gmail.com
On Sun, Oct 27, 2024 at 8:46?AM Ekaitz Zarraga <ekaitz@elenq.tech> wrote:
Toggle quote (6 lines)
>
> +Since its inception, the Guix project has always valued its autonomy, and that
> +reflects in its infrastructure: our servers run Guix System and exclusively
> +free software, none of them is hosted by one of these transnational companies,
> +and they're administered by volunteers.

Would it be better to remove the negative reference to "transnational
companies" and replace it with a positive description of the current
and desired hosting sites? Or remove that snippet altogether? I think
this is saying we don't like "clouds", and if the reason is trusted
hardware then we can simply state that and note the requisite free
software bootloader.

Greg
N
N
Nicolas Graves wrote on 3 Nov 19:51 +0100
[PATCH] doc: Add "Contributing to Guix's infrastructure".
(address . 74046@debbugs.gnu.org)
87msigp2v3.fsf@ngraves.fr
I have no particular knowledge about distributed computing other than a
past use of BOINC, but I wonder if some contribution in that form would
be useful in a near future in Guix. I have a beefy machine and would be
happy to lend ~10 cores for 10h/day to building Guix binaries. Not that
much, but I like the idea, and I think it also helps to create another
contribution option and might participate to create a stronger
community. (By the way, same thing for peer-sharing build results, with
the same problematics).

Has there already been talks about that? I guess the hard part is to
guarantee security / untouched contributed binaries but I guess guys
from distributed computing also have the same issues, so maybe there's a
way to guarantee that?

--
Best regards,
Nicolas Graves
E
E
Ekaitz Zarraga wrote on 7 Nov 10:03 +0100
[PATCH v2] doc: Add "Contributing to Guix's infrastructure".
(address . 74046@debbugs.gnu.org)
e661d69afb3a40516403cc722ecbbe73639d3892.1730970238.git.ekaitz@elenq.tech
Use the "Call for contribution to the Guix infrastructure" by Ludovic
Courtès to create a section in the documentation that describes how to
contribute to the infrastructure.


* doc/contributing.texi (Contributing to Guix's infrastructure): New
section.

Change-Id: I3f3a99ad884110cc8323789e8c14bec1f7327e97
---
doc/contributing.texi | 186 ++++++++++++++++++++++++++++++++++++++++++
1 file changed, 186 insertions(+)

Toggle diff (205 lines)
diff --git a/doc/contributing.texi b/doc/contributing.texi
index acdc303be6..1fb6ce2e1e 100644
--- a/doc/contributing.texi
+++ b/doc/contributing.texi
@@ -37,6 +37,7 @@ Contributing
* Deprecation Policy:: Commitments and tools for deprecation.
* Writing Documentation:: Improving documentation in GNU Guix.
* Translating Guix:: Make Guix speak your native language.
+* Contributing to Guix's infrastructure:: Make Guix ecosystem work better.
@end menu
@node Requirements
@@ -3594,3 +3595,188 @@ Translating Guix
be updated accordingly (see @file{website/i18n-howto.txt} for more
information on the process).
@end itemize
+
+
+@cindex infrastructure
+@node Contributing to Guix's infrastructure
+@section Contributing to Guix's infrastructure
+
+Since its inception, the Guix project has always valued its autonomy, and that
+reflects in its infrastructure: our servers run Guix System and exclusively
+free software, and are administered by volunteers.
+
+Of course this comes at a cost and this is why we need contributions. Our hope
+is to make infrastructure-related activity more legible so that maybe you can
+picture yourself helping in one of these areas.
+
+
+@menu
+* Coding::
+* System administration::
+* Day-to-day system administration::
+* On-site intervention::
+* Hosting::
+* Administrative tasks::
+@end menu
+
+@node Coding
+@subsection Coding
+
+Guix runs many Guix-specific services; this is all lovely Scheme code but it
+tends to receive less attention than Guix itself:
+
+@itemize
+@item Build Farm Front-End: @url{https://git.cbaines.net/guix/bffe}
+@item Cuirass: @url{https://guix.gnu.org/cuirass/}
+@item Goggles (IRC logger):
+@url{https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/goggles.scm}
+@item Guix Build Coordinator:
+@url{https://git.savannah.gnu.org/cgit/guix/build-coordinator.git/}
+@item Guix Data Service:
+@url{https://git.savannah.gnu.org/git/guix/data-service.git/}
+@item Guix Packages Website:
+@url{https://codeberg.org/luis-felipe/guix-packages-website.git}
+@item mumi: @url{https://git.savannah.gnu.org/cgit/guix/mumi.git/}
+@item nar-herder: @url{https://git.savannah.gnu.org/cgit/guix/nar-herder.git/}
+@item QA Frontpage: @url{https://git.savannah.gnu.org/git/guix/qa-frontpage.git}
+@end itemize
+
+There is no time constraint on this coding activity: any improvement is
+welcome, whenever it comes. Most of these code bases are relatively small,
+which should make it easier to get started.
+
+Prerequisites: Familiarity with Guile, HTTP, and databases.
+
+If you wish to get started, check out the README of the project of your choice
+and get in touch with guix-devel and the primary developer(s) of the tool as
+per @code{git shortlog -s | sort -k1 -n}.
+
+@node System administration
+@subsection System administration
+
+Guix System configuration for all our systems is held in this repository:
+
+@url{https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/}
+
+The two front-ends are @file{berlin.scm} (the machine behind ci.guix.gnu.org)
+and @file{bayfront.scm} (the machine behind bordeaux.guix.gnu.org,
+guix.gnu.org, hpc.guix.info, qa.guix.gnu.org, and more). Both connect to a
+number of build machines and helpers.
+
+Without even having SSH access to the machine, you can help by posting patches
+to improve the configuration (you can test it with @code{guix system vm}).
+Here are ways you can help:
+
+@itemize
+@item
+Improve infra monitoring: set up a dashboard to monitor all the infrastructure,
+and an out-of-band channel to communicate about downtime.
+
+@item
+Implement web site redundancy: guix.gnu.org should be backed by several
+machines on different sites. Get in touch with us and/or send a patch!
+
+@item
+Implement substitute redundancy: likewise, bordeaux.guix.gnu.org and
+ci.guix.gnu.org should be backed by several head nodes.
+
+@item
+Improve backup: there's currently ad-hoc backup of selected pieces over rsync
+between the two head nodes; we can improve on that, for example with a
+dedicated backup site and proper testing of recoverability.
+
+@item
+Support mirroring: We'd like to make it easy for others to mirror substitutes
+from ci.guix and bordeaux.guix, perhaps by offering public rsync access.
+
+@item
+Optimize our web services: Monitor the performance of our services and tweak
+nginx config or whatever it takes to improve it.
+
+There is no time constraint on this activity: any improvement is welcome,
+whenever you can work on it.
+
+Prerequisite: Familiarity with Guix System administration and ideally with the
+infrastructure handbook:
+
+@url{https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org}
+
+@end itemize
+
+@node Day-to-day system administration
+@subsection Day-to-day system administration
+
+We're also looking for people who'd be willing to have SSH access to some of
+the infrastructure to help with day-to-day maintenance: restarting a build,
+restarting the occasional service that has gone wild (that can happen),
+reconfiguring/upgrading a machine, rebooting, etc.
+
+This day-to-day activity requires you to be available some of the time (during
+office hours or not, during the week-end or not), whenever is convenient for
+you, so you can react to issues reported on IRC, on the mailing list, or
+elsewhere, and synchronize with other sysadmins.
+
+Prerequisite: Being a “known” member of the community, familiarity with Guix
+System administration, with some of the services/web sites being run, and with
+the infrastructure handbook:
+
+@url{https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org}
+
+@node On-site intervention
+@subsection On-site intervention
+
+The first front-end is currently generously hosted by the Max Delbrück Center
+(MDC), a research institute in Berlin, Germany. Only authorized personnel can
+physically access it.
+
+The second one, bordeaux.guix.gnu.org, is hosted in Bordeaux, France, in a
+professional data center shared with non-profit ISP Aquilenet. If you live in
+the region of Bordeaux and would like to help out when we need to go on-site,
+please make yourself known by emailing @email{guix-sysadmin@@gnu.org}.
+
+On-site interventions are rare, but they're usually in response to an
+emergency.
+
+@node Hosting
+@subsection Hosting
+
+We're looking for people who can host machines and help out whenever
+physical access is needed. More specifically:
+
+@itemize
+@item
+We need hosting of “small” machines such as single-board computers (AArch64,
+RISC-V) for use as build machines.
+
+@item
+We need hosting for front-ends and x86_64 build machines in a data center where
+they can be racked and where, ideally, several local Guix sysadmins can
+physically access them.
+@end itemize
+
+The machines should be accessible over Wireguard VPN most of the
+time, so longer power or network interruptions should be the
+exception.
+
+Prerequisites: Familiarity with installing and remotely administering Guix
+System.
+
+@node Administrative tasks
+@subsection Administrative tasks
+
+The infra remains up and running thanks to crucial administrative tasks, which
+includes:
+
+@itemize
+@item
+Selecting and purchasing hardware, for example build machines.
+@item
+Renewing domain names.
+
+@item
+Securing funding, in particular via the Guix Foundation:
+@url{https://foundation.guix.info}
+@end itemize
+
+Prerequisites: Familiarity with hardware, and/or DNS registrars,
+and/or sponsorship, and/or crowdfunding.

base-commit: 091131af64fd4e4e925fff829fa19097cfcdfcc5
--
2.46.0
L
L
Ludovic Courtès wrote 4 days ago
Re: [bug#74046] [PATCH] doc: Add "Contributing to Guix's infrastructure".
(name . Nicolas Graves)(address . ngraves@ngraves.fr)(address . 74046@debbugs.gnu.org)
87bjy9v9e8.fsf@gnu.org
Hi,

Nicolas Graves <ngraves@ngraves.fr> skribis:

Toggle quote (9 lines)
> I have no particular knowledge about distributed computing other than a
> past use of BOINC, but I wonder if some contribution in that form would
> be useful in a near future in Guix. I have a beefy machine and would be
> happy to lend ~10 cores for 10h/day to building Guix binaries. Not that
> much, but I like the idea, and I think it also helps to create another
> contribution option and might participate to create a stronger
> community. (By the way, same thing for peer-sharing build results, with
> the same problematics).

That’s not really an option, from a security viewpoint: since the only
way to check that the binaries someone provides really corresponds to
the source is to build them, the binary provider has to be trusted by
its users. A BOINC-style model doesn’t seem practical.

Ludo’.
L
L
Ludovic Courtès wrote 4 days ago
Re: [bug#74046] [PATCH v2] doc: Add "Contributing to Guix's infrastructure".
(name . Ekaitz Zarraga)(address . ekaitz@elenq.tech)
875xohv8u9.fsf@gnu.org
Hi!

Ekaitz Zarraga <ekaitz@elenq.tech> skribis:

Toggle quote (11 lines)
> Use the "Call for contribution to the Guix infrastructure" by Ludovic
> Courtès to create a section in the documentation that describes how to
> contribute to the infrastructure.
>
> https://lists.gnu.org/archive/html/guix-devel/2024-05/msg00183.html
>
> * doc/contributing.texi (Contributing to Guix's infrastructure): New
> section.
>
> Change-Id: I3f3a99ad884110cc8323789e8c14bec1f7327e97

I think it’s a good idea. I have some reservations about the style, as
Greg already noted :-), and the fact that some of the info is quite
detailed and might become outdated rather quickly. But I think the
result is still a big improvement, and that outweighs the “risks”.

Toggle quote (3 lines)
> +@node Contributing to Guix's infrastructure
> +@section Contributing to Guix's infrastructure

Nitpick: Could you use title case for all section/node titles?

Toggle quote (4 lines)
> +@itemize
> +@item Build Farm Front-End: @url{https://git.cbaines.net/guix/bffe}
> +@item Cuirass: @url{https://guix.gnu.org/cuirass/}

To improve rendering, especially in HTML, could you use two-argument
@url as shown below, for all the URLs?


Toggle quote (6 lines)
> +Prerequisite: Being a “known” member of the community, familiarity with Guix
> +System administration, with some of the services/web sites being run, and with
> +the infrastructure handbook:
> +
> +@url{https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org}

Toggle quote (2 lines)
> +We need hosting of “small” machines such as single-board computers (AArch64,

Quotes should be written ``like this'' to make sure nothing goes wrong
with the various Texinfo backends.

Could you send an updated version?

Thanks!

Ludo’.

PS: At some point we should prolly consider make the “Contributing”
chapter to a separate document altogether.
E
E
Ekaitz Zarraga wrote 4 days ago
(name . Ludovic Courtès)(address . ludo@gnu.org)
daf7321f-f50b-4874-9e59-9d5d820a0ced@elenq.tech
Hi,

Toggle quote (5 lines)
> I think it’s a good idea. I have some reservations about the style, as
> Greg already noted :-), and the fact that some of the info is quite
> detailed and might become outdated rather quickly. But I think the
> result is still a big improvement, and that outweighs the “risks”.

I just copied what you wrote :)

If it becomes outdated we could update... that's the good part of a
living document. We have thousands of software packages in the same
repository, that become outdated way faster than the documentation. I
think we should encourage people to update documentation as we do with
the packages. Still, I agree with what you say, it can be hard to keep
up with translations.

We could evolve it to become more generic so it doesn't need that many
updates, but still keep the essence that people should pay attention to
the fact that we need help with the infrastructure.

Toggle quote (5 lines)
>> +@node Contributing to Guix's infrastructure
>> +@section Contributing to Guix's infrastructure
>
> Nitpick: Could you use title case for all section/node titles?

Ok

Toggle quote (9 lines)
>> +@itemize
>> +@item Build Farm Front-End: @url{https://git.cbaines.net/guix/bffe}
>> +@item Cuirass: @url{https://guix.gnu.org/cuirass/}
>
> To improve rendering, especially in HTML, could you use two-argument
> @url as shown below, for all the URLs?
>
> @url{https://guix.gnu.org/cuirass/, Cuirass}

Sure.

Toggle quote (15 lines)
>> +Prerequisite: Being a “known” member of the community, familiarity with Guix
>> +System administration, with some of the services/web sites being run, and with
>> +the infrastructure handbook:
>> +
>> +@url{https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org}
>
> the
> @uref{https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/doc/infra-handbook.org,
> infrastructure handbook}.
>
>> +We need hosting of “small” machines such as single-board computers (AArch64,
>
> Quotes should be written ``like this'' to make sure nothing goes wrong
> with the various Texinfo backends.

Ok

Toggle quote (2 lines)
> Could you send an updated version?

Find it attached.

Toggle quote (4 lines)
> Thanks!
>
> Ludo’.

Thank you!


Toggle quote (3 lines)
> PS: At some point we should prolly consider make the “Contributing”
> chapter to a separate document altogether.

It depends on if we consider contributing is really part of how to use
Guix or not. I'm ok with keeping it here but also with moving it
somewhere else.
Attachment: file
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 74046@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 74046
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch