From debbugs-submit-bounces@debbugs.gnu.org Thu Apr 16 10:27:39 2020 Received: (at 24937) by debbugs.gnu.org; 16 Apr 2020 14:27:39 +0000 Received: from localhost ([127.0.0.1]:39057 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jP5UR-0001R0-FD for submit@debbugs.gnu.org; Thu, 16 Apr 2020 10:27:39 -0400 Received: from sender4-of-o53.zoho.com ([136.143.188.53]:21355) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jP5UO-0001Qr-WE for 24937@debbugs.gnu.org; Thu, 16 Apr 2020 10:27:37 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1587047252; cv=none; d=zohomail.com; s=zohoarc; b=NHpeWwxNlnaJkTstl3az5qdsq3TaO/yBVa6PMI4hji50I4E6E3Z9V+hqgo8OSxBUkmepJdt8CDnACfNV+CNvh9luUzDaoFoW67JgHQzfcdK9Gdz0Yfb8pOoODf/HqREKmNSnWhMWV8vgRDsQZP/WbnsPYNeoSmWbr1CDbyq3FWM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1587047252; h=Content-Type:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=yX8r1H9jQfw6U3NqRwnbd2CfuLzuWvduuq2wCfJT+sc=; b=dCBJ76Q2ZTuq6LMeMOBHVIALZ4DdPwF0J/AgA8hcswdOAZy5EoVd/IYx4PMXcgPt3dh5XDac4ah2apbYmZD1r9j0A8H62H2nQko3eNncjnTpfxVy/496FDfkqZvD7xZjDb/9RenJTtnwVLhd/5NIEyjmv1LbjfkJc7524Lo37l0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1587047252; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type; bh=yX8r1H9jQfw6U3NqRwnbd2CfuLzuWvduuq2wCfJT+sc=; b=QMkHvnwIYRgXkq9pdA+rlA7Wbjd2JUo/Ak7FM1j/J2+zaHvfgQW2UjRJsJNKfROo chM1iBH0mEaKozWIAlhggmqAlFNpUeyw6xjJwYFKK6iHRQKM9ZAEWGwczLsPdTfim7k p52WXqLNjc18l82uaCn8zl+IHbA1pSqK6H50jPn4= Received: from localhost (p54AD4D27.dip0.t-ipconnect.de [84.173.77.39]) by mx.zohomail.com with SMTPS id 1587047251175351.5640975176159; Thu, 16 Apr 2020 07:27:31 -0700 (PDT) References: <87wpg7ffbm.fsf@gnu.org> <87lgvm4lzu.fsf@gnu.org> <87twaaa6j9.fsf@netris.org> <87twaa2vjx.fsf@gnu.org> <87lgvm9sgq.fsf@netris.org> <87d1gwvgu0.fsf@gnu.org> <87wpf4yoz0.fsf@netris.org> <87fulrsqxx.fsf@gnu.org> <87vaunbvcu.fsf@mdc-berlin.de> <87ftd3muhp.fsf@elephly.net> User-agent: mu4e 1.2.0; emacs 26.3 From: Ricardo Wurmus To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#24937: "deleting unused links" GC phase is too slow In-reply-to: <87ftd3muhp.fsf@elephly.net> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Thu, 16 Apr 2020 16:27:27 +0200 Message-ID: <87eesnmrow.fsf@elephly.net> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Zoho-Virus-Status: 2 X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24937 Cc: 24937@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Here are more benchmarks on one of the build nodes. It doesn=E2=80=99t nea= rly have as many used inodes as ci.guix.gnu.org, but I could fill it up if necessary. root@hydra-guix-127 ~# df -i /gnu/ Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda3 28950528 2796829 26153699 10% / root@hydra-guix-127 ~# ls -1 /gnu/store/.links | wc -l 2017395 I tested all three modes with statx and with lstat. The links-traversal-statx.c is attached below. * mode 1 + statx --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal-statx.c -= DMODE=3D1 -D_GNU_SOURCE=3D1 -o links-traversal links-traversal-statx.c:53:8: warning: =EF=BF=BDstat_entries=EF=BF=BD defin= ed but not used [-Wunused-function] 53 | void stat_entries (void) | ^~~~~~~~~~~~ root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 9 seconds (including stat) real 0m9.176s user 0m0.801s sys 0m4.236s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 4 seconds (including stat) real 0m3.556s user 0m0.708s sys 0m2.848s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 4 seconds (including stat) real 0m3.553s user 0m0.599s sys 0m2.954s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- * mode 2 + statx --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal-statx.c -= DMODE=3D2 -D_GNU_SOURCE=3D1 -o links-traversal root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 10 seconds (including stat) real 0m9.598s user 0m1.210s sys 0m4.257s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 4 seconds (including stat) real 0m4.094s user 0m0.988s sys 0m3.107s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 4 seconds (including stat) real 0m4.095s user 0m0.933s sys 0m3.162s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- * mode 3 + statx --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal-statx.c -= DMODE=3D3 -D_GNU_SOURCE=3D1 -o links-traversal^C root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 7 seconds stat took 3 seconds real 0m9.992s user 0m1.411s sys 0m4.221s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 1 seconds stat took 2 seconds real 0m4.265s user 0m1.120s sys 0m3.145s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 2 seconds stat took 2 seconds real 0m4.267s user 0m1.072s sys 0m3.195s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- Now with just lstat: * mode 1 + lstat --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal.c -DMODE= =3D1 -D_GNU_SOURCE=3D1 -o links-traversal links-traversal.c:49:8: warning: =EF=BF=BDstat_entries=EF=BF=BD defined but= not used [-Wunused-function] 49 | void stat_entries (void) | ^~~~~~~~~~~~ root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 9 seconds (including stat) real 0m9.303s user 0m0.748s sys 0m4.397s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 4 seconds (including stat) real 0m3.526s user 0m0.540s sys 0m2.987s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 3 seconds (including stat) real 0m3.519s user 0m0.600s sys 0m2.919s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- * mode 2 + lstat --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal.c -DMODE= =3D2 -D_GNU_SOURCE=3D1 -o links-traversal root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 9 seconds (including stat) real 0m9.614s user 0m1.205s sys 0m4.250s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 4 seconds (including stat) real 0m4.060s user 0m1.052s sys 0m3.008s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 17377 dir_entries, 4 seconds (including stat) real 0m4.057s user 0m0.984s sys 0m3.073s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- * mode 3 + lstat --8<---------------cut here---------------start------------->8--- root@hydra-guix-127 ~ [env]# gcc -Wall -std=3Dc99 links-traversal.c -DMODE= =3D3 -D_GNU_SOURCE=3D1 -o links-traversal root@hydra-guix-127 ~ [env]# echo 3 > /proc/sys/vm/drop_caches root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 6 seconds stat took 3 seconds real 0m9.767s user 0m1.270s sys 0m4.339s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 2 seconds stat took 2 seconds real 0m4.234s user 0m1.136s sys 0m3.097s root@hydra-guix-127 ~ [env]# time ./links-traversal=20 2017397 dir_entries, 1 seconds stat took 2 seconds real 0m4.222s user 0m1.052s sys 0m3.170s root@hydra-guix-127 ~ [env]#=20 --8<---------------cut here---------------end--------------->8--- They are all very close, so I think I need to work with a bigger store to see a difference. Or perhaps I did something silly because I don=E2=80=99t know C=E2=80=A6 I= f so please let me know. -- Ricardo --=-=-= Content-Type: application/octet-stream Content-Disposition: attachment; filename=links-traversal-statx.c Content-Transfer-Encoding: base64 I2luY2x1ZGUgPHVuaXN0ZC5oPgojaW5jbHVkZSA8ZGlyZW50Lmg+CiNpbmNsdWRlIDxzeXMvdHlw ZXMuaD4KI2luY2x1ZGUgPHN0ZGxpYi5oPgojaW5jbHVkZSA8c3RkaW8uaD4KI2luY2x1ZGUgPHN5 cy90aW1lLmg+CiNpbmNsdWRlIDxzdHJpbmcuaD4KI2luY2x1ZGUgPHN5cy9zdGF0Lmg+CiNpbmNs dWRlIDx1bmlzdGQuaD4KI2luY2x1ZGUgPGZjbnRsLmg+CiNpbmNsdWRlIDxhc3NlcnQuaD4KCgoK I2RlZmluZSBTVEFUX0lOVEVSTEVBVkVEIDEKI2RlZmluZSBTVEFUX1NFTUlfSU5URVJMRUFWRUQg MgojZGVmaW5lIFNUQVRfT1BUSU1BTCAzCgpzdHJ1Y3QgZW50cnkKewogIGNoYXIgKm5hbWU7CiAg aW5vX3QgaW5vZGU7Cn07CgojZGVmaW5lIE1BWF9FTlRSSUVTIDEzNTAwMDAwMApzdGF0aWMgc3Ry dWN0IGVudHJ5IGRpcl9lbnRyaWVzW01BWF9FTlRSSUVTXTsKCmludAptYWluICgpCnsKICBzdHJ1 Y3QgdGltZXZhbCBzdGFydCwgZW5kOwoKICAvKiBGb3IgdXNlZnVsIHRpbWluZ3MsIGRvOgogICAg IHN1ZG8gc2ggLWMgJ2VjaG8gMyA+IC9wcm9jL3N5cy92bS9kcm9wX2NhY2hlcycgICovCiAgZ2V0 dGltZW9mZGF5ICgmc3RhcnQsIE5VTEwpOwogIERJUiAqbGlua3MgPSBvcGVuZGlyICgiL2dudS9z dG9yZS8ubGlua3MiKTsKCiAgc2l6ZV90IGNvdW50ID0gMDsKCiNpZiBNT0RFICE9IFNUQVRfSU5U RVJMRUFWRUQKICB2b2lkIHNvcnRfZW50cmllcyAodm9pZCkKICB7CiAgICBpbnQgZW50cnlfbG93 ZXIgKGNvbnN0IHZvaWQgKmEsIGNvbnN0IHZvaWQgKmIpCiAgICB7CiAgICAgIHJldHVybiAoKHN0 cnVjdCBlbnRyeSAqKWEpLT5pbm9kZSA8ICgoc3RydWN0IGVudHJ5ICopYiktPmlub2RlOwogICAg fQoKICAgIHFzb3J0IChkaXJfZW50cmllcywgY291bnQsIHNpemVvZiAoc3RydWN0IGVudHJ5KSwK CSAgIGVudHJ5X2xvd2VyKTsKICB9CiNlbmRpZgoKICB2b2lkIHN0YXRfZW50cmllcyAodm9pZCkK ICB7CiAgICBmb3IgKHNpemVfdCBpID0gMDsgaSA8IGNvdW50OyBpKyspCiAgICAgIHsKCXN0cnVj dCBzdGF0eCBzdDsKCXN0YXR4KEFUX0ZEQ1dELCBkaXJfZW50cmllc1tpXS5uYW1lLAogICAgICAg ICAgICAgIEFUX1NZTUxJTktfTk9GT0xMT1cgfCBBVF9TVEFUWF9ET05UX1NZTkMsCiAgICAgICAg ICAgICAgU1RBVFhfU0laRSB8IFNUQVRYX05MSU5LLCAmc3QpOwoJLy9sc3RhdCAoZGlyX2VudHJp ZXNbaV0ubmFtZSwgJnN0KTsKICAgICAgfQogIH0KCiAgZm9yIChzdHJ1Y3QgZGlyZW50ICplbnRy eSA9IHJlYWRkaXIgKGxpbmtzKTsKICAgICAgIGVudHJ5ICE9IE5VTEw7CiAgICAgICBlbnRyeSA9 IHJlYWRkaXIgKGxpbmtzKSkKICAgIHsKICAgICAgYXNzZXJ0IChjb3VudCA8IE1BWF9FTlRSSUVT KTsKICAgICAgZGlyX2VudHJpZXNbY291bnRdLm5hbWUgPSBzdHJkdXAgKGVudHJ5LT5kX25hbWUp OwogICAgICBkaXJfZW50cmllc1tjb3VudF0uaW5vZGUgPSBlbnRyeS0+ZF9pbm87CiNpZiBNT0RF ID09IFNUQVRfSU5URVJMRUFWRUQKICAgICAgc3RydWN0IHN0YXR4IHN0OwogICAgICBzdGF0eChB VF9GRENXRCwgZW50cnktPmRfbmFtZSwKICAgICAgICAgICAgQVRfU1lNTElOS19OT0ZPTExPVyB8 IEFUX1NUQVRYX0RPTlRfU1lOQywgU1RBVFhfU0laRSB8IFNUQVRYX05MSU5LLCAmc3QpOwoKICAg ICAgLy9sc3RhdCAoZW50cnktPmRfbmFtZSwgJnN0KTsKI2VuZGlmCgojaWYgTU9ERSA9PSBTVEFU X1NFTUlfSU5URVJMRUFWRUQKICAgICAgaWYgKGNvdW50KysgPj0gMTAwMDAwKQoJewoJICBzb3J0 X2VudHJpZXMgKCk7CgkgIHN0YXRfZW50cmllcyAoKTsKCSAgY291bnQgPSAwOwoJfQojZWxzZQog ICAgICBjb3VudCsrOwojZW5kaWYKICAgIH0KCiNpZiBNT0RFID09IFNUQVRfU0VNSV9JTlRFUkxF QVZFRAogIHNvcnRfZW50cmllcyAoKTsKICBzdGF0X2VudHJpZXMgKCk7CiNlbmRpZgoKICBnZXR0 aW1lb2ZkYXkgKCZlbmQsIE5VTEwpOwogIHByaW50ZiAoIiV6aSBkaXJfZW50cmllcywgJXppIHNl Y29uZHMiCiNpZiBNT0RFICE9IFNUQVRfT1BUSU1BTAoJICAiIChpbmNsdWRpbmcgc3RhdCkiCiNl bmRpZgoJICAiXG4iLCBjb3VudCwKCSAgZW5kLnR2X3NlYyAtIHN0YXJ0LnR2X3NlYyk7CgojaWYg TU9ERSA9PSBTVEFUX09QVElNQUwKICBzb3J0X2VudHJpZXMgKCk7CiAgZ2V0dGltZW9mZGF5ICgm c3RhcnQsIE5VTEwpOwogIHN0YXRfZW50cmllcyAoKTsKICBnZXR0aW1lb2ZkYXkgKCZlbmQsIE5V TEwpOwoKICBwcmludGYgKCJzdGF0IHRvb2sgJXppIHNlY29uZHNcbiIsIGVuZC50dl9zZWMgLSBz dGFydC50dl9zZWMpOwojZW5kaWYKCiAgcmV0dXJuIEVYSVRfU1VDQ0VTUzsKfQo= --=-=-=--