Local privilege escalation via guix-daemon and ‘--keep-failed’

  • Done
  • quality assurance status badge
Details
3 participants
  • Léo Le Bouter
  • Ludovic Courtès
  • Nathan Nye
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
serious
L
L
Ludovic Courtès wrote on 18 Mar 2021 12:17
(address . bug-guix@gnu.org)
87lfaksock.fsf@gnu.org
A security vulnerability that can lead to local privilege escalation has
been found in ’guix-daemon’. It affects multi-user setups in which
’guix-daemon’ runs locally.

It does not affect multi-user setups where ‘guix-daemon’ runs on a
separate machine and is accessed over the network, via
‘GUIX_DAEMON_SOCKET’, as is customary on cluster setups. Machines where
the Linux “protected hardlink”[*] feature is enabled, which is common,
are also unaffected—this is the case when the contents of
/proc/sys/fs/protected_hardlinks are 1.



Vulnerability
~~~~~~~~~~~~~

The attack consists in having an unprivileged user spawn a build
process, for instance with ‘guix build’, that makes its build directory
world-writable. The user then creates a hardlink within the build
directory to a root-owned file from outside of the build directory, such
as ‘/etc/shadow’. If the user passed the ‘--keep-failed’ option and the
build eventually fails, the daemon changes ownership of the whole build
tree, including the hardlink, to the user. At that point, the user has
write access to the target file.


Fix
~~~

The fix (patch attached) consists in adding a root-owned “wrapper”
directory in which the build directory itself is located. If the user
passed the ‘--keep-failed’ option and the build fails, the ‘guix-daemon’
first changes ownership of the build directory, and then, in two stages,
moves the build directory into the location where users expect to find
failed builds, roughly like this:

1. chown -R USER /tmp/guix-build-foo.drv-0/top
2. mv /tmp/guix-build-foo.drv-0{,.pivot}
3. mv /tmp/guix-build-foo.drv-0.pivot/top /tmp/guix-build-foo.drv-0

In step #1, /tmp/guix-build-foo.drv-0 remains root-owned, with
permissions of #o700. Thus, only root can change directory into it or
into ‘top’. Likewise in step #2.

The build tree becomes accessible to the user once step #3 has
succeeded, not before. These steps are performed after the package
build scripts have stopped running.


Additionally, the patch at https://issues.guix.gnu.org/47013 enables
protected hardlinks and symlinks by default on Guix System, which will
protect against this class of vulnerability from now on.


Credit
~~~~~~

We are grateful to Nathan Nye of WhiteBeam Security for reporting this
bug and discussing fixes with us!


Timeline
~~~~~~~~

We learned about this bug on the private guix-security@gnu.org list on
February 7th, and discussed and prepared fixes in the interim.

Ludo’ & Leo Famulari.
Toggle diff (73 lines)
diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index 20d83fea4a..4f486f0822 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -1621,6 +1621,24 @@ void DerivationGoal::startBuilder()
auto drvName = storePathToName(drvPath);
tmpDir = createTempDir("", "guix-build-" + drvName, false, false, 0700);
+ if (useChroot) {
+ /* Make the build directory seen by the build process a sub-directory.
+ That way, "/tmp/guix-build-foo.drv-0" is root-owned, and thus its
+ permissions cannot be changed by the build process, while
+ "/tmp/guix-build-foo.drv-0/top" is owned by the build user. This
+ cannot be done when !useChroot because then $NIX_BUILD_TOP would
+ be inaccessible to the build user by its full file name.
+
+ If the build user could make the build directory world-writable,
+ then an attacker could create in it a hardlink to a root-owned file
+ such as /etc/shadow. If 'keepFailed' is true, the daemon would
+ then chown that hardlink to the user, giving them write access to
+ that file. */
+ tmpDir += "/top";
+ if (mkdir(tmpDir.c_str(), 0700) == 1)
+ throw SysError("creating top-level build directory");
+ }
+
/* In a sandbox, for determinism, always use the same temporary
directory. */
tmpDirInSandbox = useChroot ? canonPath("/tmp", true) + "/guix-build-" + drvName + "-0" : tmpDir;
@@ -2626,20 +2644,41 @@ static void _chown(const Path & path, uid_t uid, gid_t gid)
void DerivationGoal::deleteTmpDir(bool force)
{
if (tmpDir != "") {
+ // When useChroot is true, tmpDir looks like
+ // "/tmp/guix-build-foo.drv-0/top". Its parent is root-owned.
+ string top;
+ if (useChroot) {
+ if (baseNameOf(tmpDir) != "top") abort();
+ top = dirOf(tmpDir);
+ } else top = tmpDir;
+
if (settings.keepFailed && !force) {
printMsg(lvlError,
format("note: keeping build directory `%2%'")
- % drvPath % tmpDir);
+ % drvPath % top);
chmod(tmpDir.c_str(), 0755);
+
// Change the ownership if clientUid is set. Never change the
// ownership or the group to "root" for security reasons.
if (settings.clientUid != (uid_t) -1 && settings.clientUid != 0) {
_chown(tmpDir, settings.clientUid,
settings.clientGid != 0 ? settings.clientGid : -1);
+
+ if (top != tmpDir) {
+ // Rename tmpDir to its parent, with an intermediate step.
+ string pivot = top + ".pivot";
+ if (rename(top.c_str(), pivot.c_str()) == -1)
+ throw SysError("pivoting failed build tree");
+ if (rename((pivot + "/top").c_str(), top.c_str()) == -1)
+ throw SysError("renaming failed build tree");
+ rmdir(pivot.c_str());
+ }
}
}
- else
+ else {
deletePath(tmpDir);
+ if (top != tmpDir) rmdir(dirOf(tmpDir).c_str());
+ }
tmpDir = "";
}
}
L
L
Ludovic Courtès wrote on 18 Mar 2021 12:18
control message for bug #47229
(address . control@debbugs.gnu.org)
87k0q4soah.fsf@gnu.org
tags 47229 + security
quit
L
L
Ludovic Courtès wrote on 18 Mar 2021 12:18
(address . control@debbugs.gnu.org)
87im5osoab.fsf@gnu.org
severity 47229 serious
quit
L
L
Ludovic Courtès wrote on 18 Mar 2021 12:45
Re: bug#47229: Local privilege escalation via guix-daemon and ‘--keep-failed’
(address . 47229@debbugs.gnu.org)(name . Leo Famulari)(address . leo@famulari.name)
878s6ksn1b.fsf@gnu.org
Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (3 lines)
> The fix (patch attached) consists in adding a root-owned “wrapper”
> directory in which the build directory itself is located.

The fix has now been pushed:


Followed by an update of the ‘guix’ package to make the fix available:


We recommend upgrading the daemon (using commit 94f03125 or later).
On Guix System, you achieve that by running something along these lines:

guix pull
sudo guix system reconfigure /run/current-system/configuration.scm
sudo herd restart guix-daemon

On other distros, assuming services are managed by systemd:

sudo --login guix pull
sudo systemctl restart guix-daemon.service


Ludo’.
L
L
Léo Le Bouter wrote on 18 Mar 2021 12:53
9cd72f29cb33019439d2d71d6a313f930b3e7941.camel@zaclys.net
Thanks a lot to the reporter and for working on this!
-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAmBTP08ACgkQRaix6GvN
EKZemw/+Lzhig+JBobZMvUq58WOC1Am98EFd8FmNzlTan7T1CHYevgJ1bi79dOho
exqyQ6Vd9klEQGSImOdHivvB7xd5Ds/VQqPGrT+wmNI+29hL3yx4Eh0wdwbsLUbU
rMwpuEA/gNNQgmJplRkmhzVRwdfxVcYOzmTfOj4yUmaCPUjLxsy9a4HuFmuSMzL9
qyt6EjCJvXx9GWp2iOt4+3ON8diIYgl/k8d+9an907mSBmJbJoD5+pb7mwqb+SCV
9nGz8eGsHBLXK8YVo55CfkHQJ3lf6E96WM+PgF3cfIVQbOBfSmHsX4DGv9spnbV6
SOP9ucT1Miuvqs1XqzIHRM5NAmkF2Pg07mfWUpSDd5DXFLtwYpWBuX2F4t5stIyF
7Yut+mLrWF8ImV4Mk4Ut1VDtjtpANkwXYr4WbSrL06DQlYeF0wW62UrEHElyCMVV
J+B0Xk9Z3Z5OFEUXeOltwMzEYOYT/LxbUm8W5iQVnHMXY6ufTg3RNod1JqOP3Rhx
SJkmSj9hpNlGzJLWdYtpJEldaZLlU4Ufcm/+YGYhVgDHV+cOQIPY1H5zv/PTehJs
abT2iCQzDy4ZuSybW8Qxt7Up2dUS+lzrjrB3psooIuOBf4zXmz0Zeozm2hVxx1uX
cHVJcAOdntAb30+P/YjLNEwTTDLpJ8+D9pw6ios4hc4OT3yYUC8=
=25iD
-----END PGP SIGNATURE-----


L
L
L
Ludovic Courtès wrote on 18 Mar 2021 14:26
control message for bug #47229
(address . control@debbugs.gnu.org)
877dm4r3rw.fsf@gnu.org
tags 47229 fixed
close 47229
quit
N
N
Nathan Nye wrote on 23 Mar 2021 19:18
Hardlink mitigation limits
(address . 47229@debbugs.gnu.org)
8f95179a-5574-98bd-c44e-f5ee74638dc3@whitebeamsec.com
Hello,

I'm sharing here for future reference why protected hardlinks alone did
not mitigate the recent LPE security advisory, pre-patch:

"The reasons why are lines 2633 and 2637 of nix/libstore/build.cc:


When a package fails to build and the keep failed flag is set
(-K/--keep-failed), it runs a recursive chown on the build directory
(which is writable following guixbuilder01 changing the permissions to
777). It starts at the top level and chowns downwards.

The first important thing to notice here is that at any point (even
pre-chown) the build user has been compromised. The build user can write
a SUID /bin/sh to the build path, and because a normal user can traverse
into the directory before and during the chown, they can run a SUID
shell (allowing them to become guixbuilder01 even after the build user
processes are terminated). Becoming the build user allows multiple paths
to privilege escalation, but in this scenario we have faster ways of
becoming root.

Moving on to getting root, we're choosing not to use a hardlink to show
why it isn't necessary. Instead, we create a directory under the build
directory with thousands of sequentially named files, the final entry
being "passwd" or "shadow". Then we terminate the build and watch for
the first entry to be chowned to our user ID (possibly with the inotify
API). This way, we have opened a lengthy window of time where it is
enumerating over a list of file paths in our chosen directory and
chowning each of them. Now we can execute our TOCTOU race condition
vulnerability.

At the time of check (TOC), the guix-daemon has a list of file paths to
chown under what it assumes is a regular directory (because it ran
S_ISDIR on the directory). But we can swap out the directory from under
it with a symlink to /etc (most efficiently with renameat2() and using
the RENAME_EXCHANGE flag to atomically exchange the paths). At the time
of use (TOU) lchown() only checks if the file /itself/ that is being
chowned is a symlink, not if the path components are, as can be
demonstrated with Python:

$ mkdir td;touch td/tf;python3 -c 'import os;os.lchown("/home/example/td/tf", 1000, 4)';ls -lahtrd td td/tf
-rw-rw-r-- 1 example adm       0 Mar 19 19:20 td/tf
drwxrwxr-x 2 example example 4.0K Mar 19 19:20 td
$ rm -rf td
$ mkdir td; ln -s td td2;touch td2/tf;python3 -c 'import os;os.lchown("/home/example/td2/tf", 1000, 4)';ls -lahtrd td2 td2/tf
lrwxrwxrwx 1 example example 2 Mar 19 19:21 td2 -> td
-rw-rw-r-- 1 example adm    0 Mar 19 19:21 td2/tf

So lchown can blindly chown /etc/passwd to our user by following the
directory symlink and subsequently verifying that passwd itself is not a
symlink. I hope this explains the TOCTOU race condition and why
protected hardlinks help (forcing an attacker to get root using this
race condition), but they are not a solution to the problem (alone)."

- Nathan
Attachment: file
L
L
Ludovic Courtès wrote on 29 Mar 2021 17:22
(name . Nathan Nye)(address . nnye@whitebeamsec.com)(address . 47229@debbugs.gnu.org)
87h7kunfwp.fsf@gnu.org
Hi Nathan,

Nathan Nye <nnye@whitebeamsec.com> skribis:

Toggle quote (3 lines)
> I'm sharing here for future reference why protected hardlinks alone
> did not mitigate the recent LPE security advisory, pre-patch:

Thanks a lot for this clarification!

Ludo’.
?
Your comment

This issue is archived.

To comment on this conversation send an email to 47229@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 47229
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch