Local privilege escalation via guix-daemon and ‘--keep-failed’

DoneSubmitted by Ludovic Courtès.
Details
4 participants
  • Leo Famulari
  • Léo Le Bouter
  • Ludovic Courtès
  • Nathan Nye
Owner
unassigned
Severity
serious
L
L
Ludovic Courtès wrote on 18 Mar 12:17 +0100
(address . bug-guix@gnu.org)
87lfaksock.fsf@gnu.org
A security vulnerability that can lead to local privilege escalation hasbeen found in ’guix-daemon’. It affects multi-user setups in which’guix-daemon’ runs locally.
It does not affect multi-user setups where ‘guix-daemon’ runs on aseparate machine and is accessed over the network, via‘GUIX_DAEMON_SOCKET’, as is customary on cluster setups. Machines wherethe Linux “protected hardlink”[*] feature is enabled, which is common,are also unaffected—this is the case when the contents of/proc/sys/fs/protected_hardlinks are 1.
[*] https://www.kernel.org/doc/Documentation/sysctl/fs.txt

Vulnerability~~~~~~~~~~~~~
The attack consists in having an unprivileged user spawn a buildprocess, for instance with ‘guix build’, that makes its build directoryworld-writable. The user then creates a hardlink within the builddirectory to a root-owned file from outside of the build directory, suchas ‘/etc/shadow’. If the user passed the ‘--keep-failed’ option and thebuild eventually fails, the daemon changes ownership of the whole buildtree, including the hardlink, to the user. At that point, the user haswrite access to the target file.

Fix~~~
The fix (patch attached) consists in adding a root-owned “wrapper”directory in which the build directory itself is located. If the userpassed the ‘--keep-failed’ option and the build fails, the ‘guix-daemon’first changes ownership of the build directory, and then, in two stages,moves the build directory into the location where users expect to findfailed builds, roughly like this:
1. chown -R USER /tmp/guix-build-foo.drv-0/top 2. mv /tmp/guix-build-foo.drv-0{,.pivot} 3. mv /tmp/guix-build-foo.drv-0.pivot/top /tmp/guix-build-foo.drv-0
In step #1, /tmp/guix-build-foo.drv-0 remains root-owned, withpermissions of #o700. Thus, only root can change directory into it orinto ‘top’. Likewise in step #2.
The build tree becomes accessible to the user once step #3 hassucceeded, not before. These steps are performed after the packagebuild scripts have stopped running.

Additionally, the patch at https://issues.guix.gnu.org/47013 enablesprotected hardlinks and symlinks by default on Guix System, which willprotect against this class of vulnerability from now on.

Credit~~~~~~
We are grateful to Nathan Nye of WhiteBeam Security for reporting thisbug and discussing fixes with us!

Timeline~~~~~~~~
We learned about this bug on the private guix-security@gnu.org list onFebruary 7th, and discussed and prepared fixes in the interim.
Ludo’ & Leo Famulari.
Toggle diff (73 lines)diff --git a/nix/libstore/build.cc b/nix/libstore/build.ccindex 20d83fea4a..4f486f0822 100644--- a/nix/libstore/build.cc+++ b/nix/libstore/build.cc@@ -1621,6 +1621,24 @@ void DerivationGoal::startBuilder() auto drvName = storePathToName(drvPath); tmpDir = createTempDir("", "guix-build-" + drvName, false, false, 0700); + if (useChroot) {+ /* Make the build directory seen by the build process a sub-directory.+ That way, "/tmp/guix-build-foo.drv-0" is root-owned, and thus its+ permissions cannot be changed by the build process, while+ "/tmp/guix-build-foo.drv-0/top" is owned by the build user. This+ cannot be done when !useChroot because then $NIX_BUILD_TOP would+ be inaccessible to the build user by its full file name.++ If the build user could make the build directory world-writable,+ then an attacker could create in it a hardlink to a root-owned file+ such as /etc/shadow. If 'keepFailed' is true, the daemon would+ then chown that hardlink to the user, giving them write access to+ that file. */+ tmpDir += "/top";+ if (mkdir(tmpDir.c_str(), 0700) == 1)+ throw SysError("creating top-level build directory");+ }+ /* In a sandbox, for determinism, always use the same temporary directory. */ tmpDirInSandbox = useChroot ? canonPath("/tmp", true) + "/guix-build-" + drvName + "-0" : tmpDir;@@ -2626,20 +2644,41 @@ static void _chown(const Path & path, uid_t uid, gid_t gid) void DerivationGoal::deleteTmpDir(bool force) { if (tmpDir != "") {+ // When useChroot is true, tmpDir looks like+ // "/tmp/guix-build-foo.drv-0/top". Its parent is root-owned.+ string top;+ if (useChroot) {+ if (baseNameOf(tmpDir) != "top") abort();+ top = dirOf(tmpDir);+ } else top = tmpDir;+ if (settings.keepFailed && !force) { printMsg(lvlError, format("note: keeping build directory `%2%'")- % drvPath % tmpDir);+ % drvPath % top); chmod(tmpDir.c_str(), 0755);+ // Change the ownership if clientUid is set. Never change the // ownership or the group to "root" for security reasons. if (settings.clientUid != (uid_t) -1 && settings.clientUid != 0) { _chown(tmpDir, settings.clientUid, settings.clientGid != 0 ? settings.clientGid : -1);++ if (top != tmpDir) {+ // Rename tmpDir to its parent, with an intermediate step.+ string pivot = top + ".pivot";+ if (rename(top.c_str(), pivot.c_str()) == -1)+ throw SysError("pivoting failed build tree");+ if (rename((pivot + "/top").c_str(), top.c_str()) == -1)+ throw SysError("renaming failed build tree");+ rmdir(pivot.c_str());+ } } }- else+ else { deletePath(tmpDir);+ if (top != tmpDir) rmdir(dirOf(tmpDir).c_str());+ } tmpDir = ""; } }
L
L
Ludovic Courtès wrote on 18 Mar 12:18 +0100
control message for bug #47229
(address . control@debbugs.gnu.org)
87k0q4soah.fsf@gnu.org
tags 47229 + securityquit
L
L
Ludovic Courtès wrote on 18 Mar 12:18 +0100
(address . control@debbugs.gnu.org)
87im5osoab.fsf@gnu.org
severity 47229 seriousquit
L
L
Ludovic Courtès wrote on 18 Mar 12:45 +0100
Re: bug#47229: Local privilege escalation via guix-daemon and ‘--keep-failed’
(address . 47229@debbugs.gnu.org)(name . Leo Famulari)(address . leo@famulari.name)
878s6ksn1b.fsf@gnu.org
Ludovic Courtès <ludo@gnu.org> skribis:
Toggle quote (3 lines)> The fix (patch attached) consists in adding a root-owned “wrapper”> directory in which the build directory itself is located.
The fix has now been pushed:
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=ec7fb669945bfb47c5e1fdf7de3a5d07f7002ccf
Followed by an update of the ‘guix’ package to make the fix available:
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=94f03125463ee0dba2f7916fcd43fd19d4b6c892
We recommend upgrading the daemon (using commit 94f03125 or later).On Guix System, you achieve that by running something along these lines:
guix pull sudo guix system reconfigure /run/current-system/configuration.scm sudo herd restart guix-daemon
On other distros, assuming services are managed by systemd:
sudo --login guix pull sudo systemctl restart guix-daemon.service
(See https://guix.gnu.org/manual/en/html_node/Upgrading-Guix.html.)
Ludo’.
L
L
Léo Le Bouter wrote on 18 Mar 12:53 +0100
9cd72f29cb33019439d2d71d6a313f930b3e7941.camel@zaclys.net
Thanks a lot to the reporter and for working on this!
-----BEGIN PGP SIGNATURE-----
iQIzBAABCgAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAmBTP08ACgkQRaix6GvNEKZemw/+Lzhig+JBobZMvUq58WOC1Am98EFd8FmNzlTan7T1CHYevgJ1bi79dOhoexqyQ6Vd9klEQGSImOdHivvB7xd5Ds/VQqPGrT+wmNI+29hL3yx4Eh0wdwbsLUbUrMwpuEA/gNNQgmJplRkmhzVRwdfxVcYOzmTfOj4yUmaCPUjLxsy9a4HuFmuSMzL9qyt6EjCJvXx9GWp2iOt4+3ON8diIYgl/k8d+9an907mSBmJbJoD5+pb7mwqb+SCV9nGz8eGsHBLXK8YVo55CfkHQJ3lf6E96WM+PgF3cfIVQbOBfSmHsX4DGv9spnbV6SOP9ucT1Miuvqs1XqzIHRM5NAmkF2Pg07mfWUpSDd5DXFLtwYpWBuX2F4t5stIyF7Yut+mLrWF8ImV4Mk4Ut1VDtjtpANkwXYr4WbSrL06DQlYeF0wW62UrEHElyCMVVJ+B0Xk9Z3Z5OFEUXeOltwMzEYOYT/LxbUm8W5iQVnHMXY6ufTg3RNod1JqOP3RhxSJkmSj9hpNlGzJLWdYtpJEldaZLlU4Ufcm/+YGYhVgDHV+cOQIPY1H5zv/PTehJsabT2iCQzDy4ZuSybW8Qxt7Up2dUS+lzrjrB3psooIuOBf4zXmz0Zeozm2hVxx1uXcHVJcAOdntAb30+P/YjLNEwTTDLpJ8+D9pw6ios4hc4OT3yYUC8==25iD-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 18 Mar 14:14 +0100
(address . 47229@debbugs.gnu.org)(name . Leo Famulari)(address . leo@famulari.name)
87tup8r4cr.fsf@gnu.org
An additional data point: guix-daemon chowns build trees to the callerupon failure (a very handy feature) since this 2016 commit:
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=2608e40988ba8cf51723fe0d21bdedf6b3997c9c
The Nix build daemon, which guix-daemon is based on, did not have thisfeature.
L
L
Ludovic Courtès wrote on 18 Mar 14:26 +0100
control message for bug #47229
(address . control@debbugs.gnu.org)
877dm4r3rw.fsf@gnu.org
tags 47229 fixedclose 47229 quit
L
L
Leo Famulari wrote on 18 Mar 22:10 +0100
Re: bug#47229: Local privilege e scalation via guix-daemon and ‘--keep-failed’
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47229@debbugs.gnu.org)
YFPB2RTVkRxx8sfk@jasmine.lan
On Thu, Mar 18, 2021 at 12:17:15PM +0100, Ludovic Courtès wrote:
Toggle quote (7 lines)> It does not affect multi-user setups where ‘guix-daemon’ runs on a> separate machine and is accessed over the network, via> ‘GUIX_DAEMON_SOCKET’, as is customary on cluster setups. Machines where> the Linux “protected hardlink”[*] feature is enabled, which is common,> are also unaffected—this is the case when the contents of> /proc/sys/fs/protected_hardlinks are 1.
After publishing the advisory, we received a clarification about theimpact of "protected hardlinks".
When using a guix-daemon that does not include the fix [0] for the bugreported here, it is still possible for rogue build scripts to escapethe build environment, even when protected hardlinks are enabled.
Protected hardlinks do make exploitation significantly more difficult,but not impossible.
For this reason, we continue to recommend that all Guix users upgradetheir guix-daemons, as described in the original advisory.
[0]https://git.savannah.gnu.org/cgit/guix.git/commit/?id=ec7fb669945bfb47c5e1fdf7de3a5d07f7002ccf
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAmBTwdkACgkQJkb6MLrKfwhRjhAA7Q1QD4rjWsQv3r83DUZs2lGrH7lh8nJQTevij6xmFBNda4g+aFicdmq9mHOeQLqnZKw/KOdVAcND1IXghKrjq0fiLA8cwxUG0XcrVAQjwCv58KLQMfjYbYfsL99rbFWLUbw6T9PlarWsiNOZSKfW0i8rycNGaWoYpNqhWczR4cdSWOcAjkt6u6PsWqk3PZmALPnT3gSMP4b6j8Ra/H8jgpo4RT4DmleMtt6aiVrA9r+ssRN8z2UwAVMgUZ2afiHyaQWN1flUzwCM0mVgaGhMUWAUriIWLTykRBZnI0hoboNrBvHrLa0lge81oZBQ5cFepFMshTRLHXjP44A7KGRAH5WJlUGXCNlTr6s2ATxyfD6ZkcsU/a2HPxOjBJVvgZuSa78yNo3uHNzwkGsU6Ghi0muYFiet/gqUytH/BCHR44PyosKKRrChgfPap/hnaA752w5bwYpUs10KcGjRGFsTLBlLl+cFqVGBa+oRT1Aq7DbdPFh1bBSREDfgpiMX+L+4tweC0isOy7SaM2dnj1BNfg05hYwjDY4lY1uCTZ98dGRXaYElEev1h35cLHdunj8xM5HOJ18uNJ8pPxtyheR5WwSE4bb5IVOHRzfi7MXJja8ofrx7CBfb9IiOPWmmwrXWsSaLg+OjI1NXIfGgCFH+sH3uzmNaT4hs63IysIYKEo0==QbGk-----END PGP SIGNATURE-----

N
N
Nathan Nye wrote on 23 Mar 19:18 +0100
Hardlink mitigation limits
(address . 47229@debbugs.gnu.org)
8f95179a-5574-98bd-c44e-f5ee74638dc3@whitebeamsec.com
Hello,
I'm sharing here for future reference why protected hardlinks alone did not mitigate the recent LPE security advisory, pre-patch:
"The reasons why are lines 2633 and 2637 of nix/libstore/build.cc:
* https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc#n2633 * https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc#n2637
When a package fails to build and the keep failed flag is set (-K/--keep-failed), it runs a recursive chown on the build directory (which is writable following guixbuilder01 changing the permissions to 777). It starts at the top level and chowns downwards.
The first important thing to notice here is that at any point (even pre-chown) the build user has been compromised. The build user can write a SUID /bin/sh to the build path, and because a normal user can traverse into the directory before and during the chown, they can run a SUID shell (allowing them to become guixbuilder01 even after the build user processes are terminated). Becoming the build user allows multiple paths to privilege escalation, but in this scenario we have faster ways of becoming root.
Moving on to getting root, we're choosing not to use a hardlink to show why it isn't necessary. Instead, we create a directory under the build directory with thousands of sequentially named files, the final entry being "passwd" or "shadow". Then we terminate the build and watch for the first entry to be chowned to our user ID (possibly with the inotify API). This way, we have opened a lengthy window of time where it is enumerating over a list of file paths in our chosen directory and chowning each of them. Now we can execute our TOCTOU race condition vulnerability.
At the time of check (TOC), the guix-daemon has a list of file paths to chown under what it assumes is a regular directory (because it ran S_ISDIR on the directory). But we can swap out the directory from under it with a symlink to /etc (most efficiently with renameat2() and using the RENAME_EXCHANGE flag to atomically exchange the paths). At the time of use (TOU) lchown() only checks if the file /itself/ that is being chowned is a symlink, not if the path components are, as can be demonstrated with Python:
$ mkdir td;touch td/tf;python3 -c 'import os;os.lchown("/home/example/td/tf", 1000, 4)';ls -lahtrd td td/tf-rw-rw-r-- 1 example adm       0 Mar 19 19:20 td/tfdrwxrwxr-x 2 example example 4.0K Mar 19 19:20 td$ rm -rf td$ mkdir td; ln -s td td2;touch td2/tf;python3 -c 'import os;os.lchown("/home/example/td2/tf", 1000, 4)';ls -lahtrd td2 td2/tflrwxrwxrwx 1 example example 2 Mar 19 19:21 td2 -> td-rw-rw-r-- 1 example adm    0 Mar 19 19:21 td2/tf
So lchown can blindly chown /etc/passwd to our user by following the directory symlink and subsequently verifying that passwd itself is not a symlink. I hope this explains the TOCTOU race condition and why protected hardlinks help (forcing an attacker to get root using this race condition), but they are not a solution to the problem (alone)."
- Nathan
Attachment: file
L
L
Ludovic Courtès wrote on 29 Mar 17:22 +0200
(name . Nathan Nye)(address . nnye@whitebeamsec.com)(address . 47229@debbugs.gnu.org)
87h7kunfwp.fsf@gnu.org
Hi Nathan,
Nathan Nye <nnye@whitebeamsec.com> skribis:
Toggle quote (3 lines)> I'm sharing here for future reference why protected hardlinks alone> did not mitigate the recent LPE security advisory, pre-patch:
Thanks a lot for this clarification!
Ludo’.
L
L
Leo Famulari wrote on 10 Apr 19:56 +0200
Re: bug#47229: Local privilege e scalation via guix-daemon and ‘--keep-failed’
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 47229@debbugs.gnu.org)
YHHmy4+z9UOunSyB@jasmine.lan
On Thu, Mar 18, 2021 at 12:17:15PM +0100, Ludovic Courtès wrote:
Toggle quote (12 lines)> Vulnerability> ~~~~~~~~~~~~~> > The attack consists in having an unprivileged user spawn a build> process, for instance with ‘guix build’, that makes its build directory> world-writable. The user then creates a hardlink within the build> directory to a root-owned file from outside of the build directory, such> as ‘/etc/shadow’. If the user passed the ‘--keep-failed’ option and the> build eventually fails, the daemon changes ownership of the whole build> tree, including the hardlink, to the user. At that point, the user has> write access to the target file.
This has been assigned CVE-2021-27851.
Soon, it should be available in the CVE database athttps://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-27851
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEsFFZSPHn08G5gDigJkb6MLrKfwgFAmBx5sgACgkQJkb6MLrKfwhIwBAAwbuP3cK68tXgVV4ho8HoCPa9Aww5EynTNrZcmPkElJgCgdg3Phb3i+MoQY8S2jOXHlSg0cMgtYjePkPeL7bSeEmRGtUSiTq/HGXucJlfDT3D0wHNgtqBcIYNMt6Z9ROHjb9GPkD29Si0H3UovwBOKDS5QlUWXGuaeC9Y5Itl7MYhhnJpsDRBhLGk+vCmWjxbgKh/qm2KYkKH/b3KR2BMSmPQhZ7MT37E9CTGo2eiodtbfUYhWh5OfpM+iLIIcyMgULjF887KqSf6RGnt5ajolSh9zOZUXDwSqoEiMyH5g7g9rJyua7DvXYfOxrpXVtu5tRxuN6ME4g9m8eoakV28fSZ6VVR9X6mODcT1fD2amB8q32/kuOt6SORXNWvDwU0qcW+RAaN97KyrdQ95EQfYa0PTEdze+VUoar6LV124zCpvbqa8FkPJ1fF7sKJw4+LvhVMWf+zhFfTeKCv+JgxZi06WFcaEMQSwvfI0wBhfsOE9P8M364t9BQ/l5f05HxoySSEPpi9POoQGe38rKLlwkhIQvyiidSpEDn5azo6LLe577PCzdlQT7xqglqvcfJIstD9jss43TdAQffZmaTrdaJe/AaY1SPh2HCzPe6dN3BGOP1FaZ94ltyZpq1/86spDa8eKW+LcLZng3TxUufjvQn5nIx8FUU+pHR110krvJ1s==bt5R-----END PGP SIGNATURE-----

?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send email to 47229@debbugs.gnu.org