Local privilege escalation via guix-daemon and ‘--keep-failed’

  • Done
  • quality assurance status badge
Details
4 participants
  • Leo Famulari
  • Léo Le Bouter
  • Ludovic Courtès
  • Nathan Nye
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
serious
L
L
Ludovic Courtès wrote on 18 Mar 2021 12:17
(address . bug-guix@gnu.org)
87lfaksock.fsf@gnu.org
A security vulnerability that can lead to local privilege escalation has
been found in ’guix-daemon’. It affects multi-user setups in which
’guix-daemon’ runs locally.

It does not affect multi-user setups where ‘guix-daemon’ runs on a
separate machine and is accessed over the network, via
‘GUIX_DAEMON_SOCKET’, as is customary on cluster setups. Machines where
the Linux “protected hardlink”[*] feature is enabled, which is common,
are also unaffected—this is the case when the contents of
/proc/sys/fs/protected_hardlinks are 1.



Vulnerability
~~~~~~~~~~~~~

The attack consists in having an unprivileged user spawn a build
process, for instance with ‘guix build’, that makes its build directory
world-writable. The user then creates a hardlink within the build
directory to a root-owned file from outside of the build directory, such
as ‘/etc/shadow’. If the user passed the ‘--keep-failed’ option and the
build eventually fails, the daemon changes ownership of the whole build
tree, including the hardlink, to the user. At that point, the user has
write access to the target file.


Fix
~~~

The fix (patch attached) consists in adding a root-owned “wrapper”
directory in which the build directory itself is located. If the user
passed the ‘--keep-failed’ option and the build fails, the ‘guix-daemon’
first changes ownership of the build directory, and then, in two stages,
moves the build directory into the location where users expect to find
failed builds, roughly like this:

1. chown -R USER /tmp/guix-build-foo.drv-0/top
2. mv /tmp/guix-build-foo.drv-0{,.pivot}
3. mv /tmp/guix-build-foo.drv-0.pivot/top /tmp/guix-build-foo.drv-0

In step #1, /tmp/guix-build-foo.drv-0 remains root-owned, with
permissions of #o700. Thus, only root can change directory into it or
into ‘top’. Likewise in step #2.

The build tree becomes accessible to the user once step #3 has
succeeded, not before. These steps are performed after the package
build scripts have stopped running.


Additionally, the patch at https://issues.guix.gnu.org/47013 enables
protected hardlinks and symlinks by default on Guix System, which will
protect against this class of vulnerability from now on.


Credit
~~~~~~

We are grateful to Nathan Nye of WhiteBeam Security for reporting this
bug and discussing fixes with us!


Timeline
~~~~~~~~

We learned about this bug on the private guix-security@gnu.org list on
February 7th, and discussed and prepared fixes in the interim.

Ludo’ & Leo Famulari.
Toggle diff (73 lines)
diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index 20d83fea4a..4f486f0822 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -1621,6 +1621,24 @@ void DerivationGoal::startBuilder()
auto drvName = storePathToName(drvPath);
tmpDir = createTempDir("", "guix-build-" + drvName, false, false, 0700);
+ if (useChroot) {
+ /* Make the build directory seen by the build process a sub-directory.
+ That way, "/tmp/guix-build-foo.drv-0" is root-owned, and thus its
+ permissions cannot be changed by the build process, while
+ "/tmp/guix-build-foo.drv-0/top" is owned by the build user. This
+ cannot be done when !useChroot because then $NIX_BUILD_TOP would
+ be inaccessible to the build user by its full file name.
+
+ If the build user could make the build directory world-writable,
+ then an attacker could create in it a hardlink to a root-owned file
+ such as /etc/shadow. If 'keepFailed' is true, the daemon would
+ then chown that hardlink to the user, giving them write access to
+ that file. */
+ tmpDir += "/top";
+ if (mkdir(tmpDir.c_str(), 0700) == 1)
+ throw SysError("creating top-level build directory");
+ }
+
/* In a sandbox, for determinism, always use the same temporary
directory. */
tmpDirInSandbox = useChroot ? canonPath("/tmp", true) + "/guix-build-" + drvName + "-0" : tmpDir;
@@ -2626,20 +2644,41 @@ static void _chown(const Path & path, uid_t uid, gid_t gid)
void DerivationGoal::deleteTmpDir(bool force)
{
if (tmpDir != "") {
+ // When useChroot is true, tmpDir looks like
+ // "/tmp/guix-build-foo.drv-0/top". Its parent is root-owned.
+ string top;
+ if (useChroot) {
+ if (baseNameOf(tmpDir) != "top") abort();
+ top = dirOf(tmpDir);
+ } else top = tmpDir;
+
if (settings.keepFailed && !force) {
printMsg(lvlError,
format("note: keeping build directory `%2%'")
- % drvPath % tmpDir);
+ % drvPath % top);
chmod(tmpDir.c_str(), 0755);
+
// Change the ownership if clientUid is set. Never change the
// ownership or the group to "root" for security reasons.
if (settings.clientUid != (uid_t) -1 && settings.clientUid != 0) {
_chown(tmpDir, settings.clientUid,
settings.clientGid != 0 ? settings.clientGid : -1);
+
+ if (top != tmpDir) {
+ // Rename tmpDir to its parent, with an intermediate step.
+ string pivot = top + ".pivot";
+ if (rename(top.c_str(), pivot.c_str()) == -1)
+ throw SysError("pivoting failed build tree");
+ if (rename((pivot + "/top").c_str(), top.c_str()) == -1)
+ throw SysError("renaming failed build tree");
+ rmdir(pivot.c_str());
+ }
}
}
- else
+ else {
deletePath(tmpDir);
+ if (top != tmpDir) rmdir(dirOf(tmpDir).c_str());
+ }
tmpDir = "";
}
}
L
L
Ludovic Courtès wrote on 18 Mar 2021 12:18
control message for bug #47229
(address . control@debbugs.gnu.org)
87k0q4soah.fsf@gnu.org
tags 47229 + security
quit
L
L
Ludovic Courtès wrote on 18 Mar 2021 12:18
(address . control@debbugs.gnu.org)
87im5osoab.fsf@gnu.org
severity 47229 serious
quit
L
L
Ludovic Courtès wrote on 18 Mar 2021 12:45
Re: bug#47229: Local privilege escalation via guix-daemon and ‘--keep-failed’
(address . 47229@debbugs.gnu.org)(name . Leo Famulari)(address . leo@famulari.name)
878s6ksn1b.fsf@gnu.org
Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (3 lines)
> The fix (patch attached) consists in adding a root-owned “wrapper”
> directory in which the build directory itself is located.

The fix has now been pushed:


Followed by an update of the ‘guix’ package to make the fix available:


We recommend upgrading the daemon (using commit 94f03125 or later).
On Guix System, you achieve that by running something along these lines:

guix pull
sudo guix system reconfigure /run/current-system/configuration.scm
sudo herd restart guix-daemon

On other distros, assuming services are managed by systemd:

sudo --login guix pull
sudo systemctl restart guix-daemon.service


Ludo’.
L
L
Léo Le Bouter wrote on 18 Mar 2021 12:53
9cd72f29cb33019439d2d71d6a313f930b3e7941.camel@zaclys.net
Thanks a lot to the reporter and for working on this!
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAmBTP08ACgkQRaix6GvN
EKZemw/+Lzhig+JBobZMvUq58WOC1Am98EFd8FmNzlTan7T1CHYevgJ1bi79dOho
exqyQ6Vd9klEQGSImOdHivvB7xd5Ds/VQqPGrT+wmNI+29hL3yx4Eh0wdwbsLUbU
rMwpuEA/gNNQgmJplRkmhzVRwdfxVcYOzmTfOj4yUmaCPUjLxsy9a4HuFmuSMzL9
qyt6EjCJvXx9GWp2iOt4+3ON8diIYgl/k8d+9an907mSBmJbJoD5+pb7mwqb+SCV
9nGz8eGsHBLXK8YVo55CfkHQJ3lf6E96WM+PgF3cfIVQbOBfSmHsX4DGv9spnbV6
SOP9ucT1Miuvqs1XqzIHRM5NAmkF2Pg07mfWUpSDd5DXFLtwYpWBuX2F4t5stIyF
7Yut+mLrWF8ImV4Mk4Ut1VDtjtpANkwXYr4WbSrL06DQlYeF0wW62UrEHElyCMVV
J+B0Xk9Z3Z5OFEUXeOltwMzEYOYT/LxbUm8W5iQVnHMXY6ufTg3RNod1JqOP3Rhx
SJkmSj9hpNlGzJLWdYtpJEldaZLlU4Ufcm/+YGYhVgDHV+cOQIPY1H5zv/PTehJs
abT2iCQzDy4ZuSybW8Qxt7Up2dUS+lzrjrB3psooIuOBf4zXmz0Zeozm2hVxx1uX
cHVJcAOdntAb30+P/YjLNEwTTDLpJ8+D9pw6ios4hc4OT3yYUC8=
=25iD
-----END PGP SIGNATURE-----


L
L
L
Ludovic Courtès wrote on 18 Mar 2021 14:26
control message for bug #47229
(address . control@debbugs.gnu.org)
877dm4r3rw.fsf@gnu.org
tags 47229 fixed
close 47229
quit
L
L
Leo Famulari wrote on 18 Mar 2021 22:10
Re: bug#47229: Local privilege e scalation via guix-daemon and ‘--keep-failed’
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47229@debbugs.gnu.org)
YFPB2RTVkRxx8sfk@jasmine.lan
On Thu, Mar 18, 2021 at 12:17:15PM +0100, Ludovic Courtès wrote:
Toggle quote (7 lines)
> It does not affect multi-user setups where ‘guix-daemon’ runs on a
> separate machine and is accessed over the network, via
> ‘GUIX_DAEMON_SOCKET’, as is customary on cluster setups. Machines where
> the Linux “protected hardlink”[*] feature is enabled, which is common,
> are also unaffected—this is the case when the contents of
> /proc/sys/fs/protected_hardlinks are 1.

After publishing the advisory, we received a clarification about the
impact of "protected hardlinks".

When using a guix-daemon that does not include the fix [0] for the bug
reported here, it is still possible for rogue build scripts to escape
the build environment, even when protected hardlinks are enabled.

Protected hardlinks do make exploitation significantly more difficult,
but not impossible.

For this reason, we continue to recommend that all Guix users upgrade
their guix-daemons, as described in the original advisory.

[0]
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAmBTwdkACgkQJkb6MLrK
fwhRjhAA7Q1QD4rjWsQv3r83DUZs2lGrH7lh8nJQTevij6xmFBNda4g+aFicdmq9
mHOeQLqnZKw/KOdVAcND1IXghKrjq0fiLA8cwxUG0XcrVAQjwCv58KLQMfjYbYfs
L99rbFWLUbw6T9PlarWsiNOZSKfW0i8rycNGaWoYpNqhWczR4cdSWOcAjkt6u6Ps
Wqk3PZmALPnT3gSMP4b6j8Ra/H8jgpo4RT4DmleMtt6aiVrA9r+ssRN8z2UwAVMg
UZ2afiHyaQWN1flUzwCM0mVgaGhMUWAUriIWLTykRBZnI0hoboNrBvHrLa0lge81
oZBQ5cFepFMshTRLHXjP44A7KGRAH5WJlUGXCNlTr6s2ATxyfD6ZkcsU/a2HPxOj
BJVvgZuSa78yNo3uHNzwkGsU6Ghi0muYFiet/gqUytH/BCHR44PyosKKRrChgfPa
p/hnaA752w5bwYpUs10KcGjRGFsTLBlLl+cFqVGBa+oRT1Aq7DbdPFh1bBSREDfg
piMX+L+4tweC0isOy7SaM2dnj1BNfg05hYwjDY4lY1uCTZ98dGRXaYElEev1h35c
LHdunj8xM5HOJ18uNJ8pPxtyheR5WwSE4bb5IVOHRzfi7MXJja8ofrx7CBfb9IiO
PWmmwrXWsSaLg+OjI1NXIfGgCFH+sH3uzmNaT4hs63IysIYKEo0=
=QbGk
-----END PGP SIGNATURE-----


N
N
Nathan Nye wrote on 23 Mar 2021 19:18
Hardlink mitigation limits
(address . 47229@debbugs.gnu.org)
8f95179a-5574-98bd-c44e-f5ee74638dc3@whitebeamsec.com
Hello,

I'm sharing here for future reference why protected hardlinks alone did
not mitigate the recent LPE security advisory, pre-patch:

"The reasons why are lines 2633 and 2637 of nix/libstore/build.cc:


When a package fails to build and the keep failed flag is set
(-K/--keep-failed), it runs a recursive chown on the build directory
(which is writable following guixbuilder01 changing the permissions to
777). It starts at the top level and chowns downwards.

The first important thing to notice here is that at any point (even
pre-chown) the build user has been compromised. The build user can write
a SUID /bin/sh to the build path, and because a normal user can traverse
into the directory before and during the chown, they can run a SUID
shell (allowing them to become guixbuilder01 even after the build user
processes are terminated). Becoming the build user allows multiple paths
to privilege escalation, but in this scenario we have faster ways of
becoming root.

Moving on to getting root, we're choosing not to use a hardlink to show
why it isn't necessary. Instead, we create a directory under the build
directory with thousands of sequentially named files, the final entry
being "passwd" or "shadow". Then we terminate the build and watch for
the first entry to be chowned to our user ID (possibly with the inotify
API). This way, we have opened a lengthy window of time where it is
enumerating over a list of file paths in our chosen directory and
chowning each of them. Now we can execute our TOCTOU race condition
vulnerability.

At the time of check (TOC), the guix-daemon has a list of file paths to
chown under what it assumes is a regular directory (because it ran
S_ISDIR on the directory). But we can swap out the directory from under
it with a symlink to /etc (most efficiently with renameat2() and using
the RENAME_EXCHANGE flag to atomically exchange the paths). At the time
of use (TOU) lchown() only checks if the file /itself/ that is being
chowned is a symlink, not if the path components are, as can be
demonstrated with Python:

$ mkdir td;touch td/tf;python3 -c 'import os;os.lchown("/home/example/td/tf", 1000, 4)';ls -lahtrd td td/tf
-rw-rw-r-- 1 example adm       0 Mar 19 19:20 td/tf
drwxrwxr-x 2 example example 4.0K Mar 19 19:20 td
$ rm -rf td
$ mkdir td; ln -s td td2;touch td2/tf;python3 -c 'import os;os.lchown("/home/example/td2/tf", 1000, 4)';ls -lahtrd td2 td2/tf
lrwxrwxrwx 1 example example 2 Mar 19 19:21 td2 -> td
-rw-rw-r-- 1 example adm    0 Mar 19 19:21 td2/tf

So lchown can blindly chown /etc/passwd to our user by following the
directory symlink and subsequently verifying that passwd itself is not a
symlink. I hope this explains the TOCTOU race condition and why
protected hardlinks help (forcing an attacker to get root using this
race condition), but they are not a solution to the problem (alone)."

- Nathan
Attachment: file
L
L
Ludovic Courtès wrote on 29 Mar 2021 17:22
(name . Nathan Nye)(address . nnye@whitebeamsec.com)(address . 47229@debbugs.gnu.org)
87h7kunfwp.fsf@gnu.org
Hi Nathan,

Nathan Nye <nnye@whitebeamsec.com> skribis:

Toggle quote (3 lines)
> I'm sharing here for future reference why protected hardlinks alone
> did not mitigate the recent LPE security advisory, pre-patch:

Thanks a lot for this clarification!

Ludo’.
L
L
Leo Famulari wrote on 10 Apr 2021 19:56
Re: bug#47229: Local privilege e scalation via guix-daemon and ‘--keep-failed’
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47229@debbugs.gnu.org)
YHHmy4+z9UOunSyB@jasmine.lan
On Thu, Mar 18, 2021 at 12:17:15PM +0100, Ludovic Courtès wrote:
Toggle quote (12 lines)
> Vulnerability
> ~~~~~~~~~~~~~
>
> The attack consists in having an unprivileged user spawn a build
> process, for instance with ‘guix build’, that makes its build directory
> world-writable. The user then creates a hardlink within the build
> directory to a root-owned file from outside of the build directory, such
> as ‘/etc/shadow’. If the user passed the ‘--keep-failed’ option and the
> build eventually fails, the daemon changes ownership of the whole build
> tree, including the hardlink, to the user. At that point, the user has
> write access to the target file.

This has been assigned CVE-2021-27851.

Soon, it should be available in the CVE database at
-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAmBx5sgACgkQJkb6MLrK
fwhIwBAAwbuP3cK68tXgVV4ho8HoCPa9Aww5EynTNrZcmPkElJgCgdg3Phb3i+Mo
QY8S2jOXHlSg0cMgtYjePkPeL7bSeEmRGtUSiTq/HGXucJlfDT3D0wHNgtqBcIYN
Mt6Z9ROHjb9GPkD29Si0H3UovwBOKDS5QlUWXGuaeC9Y5Itl7MYhhnJpsDRBhLGk
+vCmWjxbgKh/qm2KYkKH/b3KR2BMSmPQhZ7MT37E9CTGo2eiodtbfUYhWh5OfpM+
iLIIcyMgULjF887KqSf6RGnt5ajolSh9zOZUXDwSqoEiMyH5g7g9rJyua7DvXYfO
xrpXVtu5tRxuN6ME4g9m8eoakV28fSZ6VVR9X6mODcT1fD2amB8q32/kuOt6SORX
NWvDwU0qcW+RAaN97KyrdQ95EQfYa0PTEdze+VUoar6LV124zCpvbqa8FkPJ1fF7
sKJw4+LvhVMWf+zhFfTeKCv+JgxZi06WFcaEMQSwvfI0wBhfsOE9P8M364t9BQ/l
5f05HxoySSEPpi9POoQGe38rKLlwkhIQvyiidSpEDn5azo6LLe577PCzdlQT7xqg
lqvcfJIstD9jss43TdAQffZmaTrdaJe/AaY1SPh2HCzPe6dN3BGOP1FaZ94ltyZp
q1/86spDa8eKW+LcLZng3TxUufjvQn5nIx8FUU+pHR110krvJ1s=
=bt5R
-----END PGP SIGNATURE-----


?
Your comment

This issue is archived.

To comment on this conversation send an email to 47229@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 47229
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch