[PATCH core-updates 0/6] Add `guix index` subcommand

  • Open
  • quality assurance status badge
Details
One participant
  • Antoine R. Dumont (@ardumont)
Owner
unassigned
Submitted by
Antoine R. Dumont (@ardumont)
Severity
normal
A
A
Antoine R. Dumont (@ardumont) wrote 5 days ago
(address . guix-patches@gnu.org)
87v8iy59j1.fsf@gmail.com
Hello,

This introduces a new `guix index` subcommand (initiated by
civodul). This is in charge of indexing guix packages to ease user
lookups. There are 2 indexation methods:
- `manifest`: fast but less efficient, expected to be used by local
users
- `store`: slow but more efficient, expected to be used on a guix build
machine. Another service could be in charge of exposing a search
package service to ease remote search of packages.

** Impact (positive)

tl;dr This is an equivalent of `nix-index/nix-locate` or `apt-file
search` cli we can find respectively in Nix or Debian's toolbox
(unbeknownst of their implementation).

This should help users transitioning from other distributions. Usually,
people will want to look up for files or packages they are already
using. They'd expect an easy way to find those back in guix. And we
currently cannot do so easily. We can ask on #guix-devel irc (as someone
mentions to me there) though that's not scalable nor really practical
for non-irc users.

** How did I test/run it?

I've been running both the:
- `guix index` subcommand to index packages (with both methods in
separated or not dbs).
- `guix index search FILE` to actually search for packages present in
the db

I've inspected time and again the db through the `sqlite3` program after
multiple runs too to ensure data were consistent. Data are consistent.

** Build?

The makefile routine will build that subcommand too.

** Development details

The backend technology is sqlite3. The command is in charge of migrating
the data model (if any). That data model migration process is
transparent for users. The data model is currently at version 3 because
it underwent some migrations during its development process
already. It's kept for dogfooding and examples reasons.

The --version flag mentions the default backend locations and data model
version:
Toggle snippet (14 lines)
$ guix index --version
Extension local cache database:
- path: /home/tony/.cache/guix/index/db.sqlite
- version: 3

guix index (GNU Guix) 1.4.0.3874-372b2
Copyright (C) 2023 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ git log | head -1
commit 372b2b9660b8293eebd6280bb46a4ec07d4192a7

** Current caveat ("help needed on test")

The code works but...

I can't make the integration test i wrote work. Hence my delay to send
the overall patch (as the subcommand has been ready for a while). As
it's been way too long already, I've posted it in the hope someone
interested enough sees what's missing. Thanks in advance.

What's happening during the test is that somehow the execution of the
`guix index` subcommand triggers an exit code > 0 (error). Which,
expectedly, stops the test immediately. I don't get why the subcommand
exits that way and I don't see how to dig further unfortunately. Any
clues would be very much appreciated.

** Use

The subcommand is documented exhaustively.

Toggle snippet (37 lines)
$ guix index --help
Usage: guix index [OPTIONS...] [search FILE...]
Without argument, indexes (package, file) relationships from the machine.
This allows indexation with 2 methods, out of the local:

- manifests: This is the fastest implementation with the caveat of indexing
less packages. That'd be typically the use case of user local indexation.

- store: This is slowest implementation. It discusses with the store
daemon. That'd be typically the use case of building the largest db in one of
the build farm node.

With 'search FILE', search for packages installing FILE.

Note: Internal cache is located at ~/.cache/guix/index/db.sqlite by default.
See --db-path for customization.

The valid values for OPTIONS are:

-h, --help Display this help and exit
-V, --version Display version information and exit
--db-path=DIR Change default location of the cache db

--method=METH Change default indexation method. By default it uses the
local "manifests" (faster). It can also uses the local
"store" (slower, typically on the farm build ci).
The valid values for ARGS are:

search FILE Search for packages installing the FILE (from cache db)

<EMPTY> Without any argument, it index packages. This fills in the
db cache using whatever indexation method is defined.
Report bugs to: bug-guix@gnu.org.
GNU Guix home page: <https://www.gnu.org/software/guix/>
General help using Guix and GNU software: <https://guix.gnu.org/en/help/>

Example of an indexation (fast method by default):

Toggle snippet (4 lines)
$ guix index
Registering 133 packages ▕█████▎ ▏

Example of a search:

Toggle snippet (5 lines)
$ guix index search sqlite3
sqlite@3.37.0 /gnu/store/jd6nn2c8ln5flv4vwl7pp1w804w2i9wj-sqlite-3.37.0/bin/sqlite3
sqlite@3.36.0 /gnu/store/xmzx5mzv4863yw9kmr2ykndgp37p8if0-sqlite-3.36.0/bin/sqlite3

Note: inspired from zimoun's way of quoting (/me like those, thx ;)

** Development process (optional read)

Note that this started as a subcommand (again from civodul's bootstrap
code). Then as a good proposal from zimoun, this got simplified into a
guix extension (awesome work zimoun ;). Fwiw, developing it as an
extension felt way simpler for dev and run. Versus, having to setup my
machines to develop with guix. It continued as a guix subcommand in the
end as civodul proposed it that way instead.

Sent from my MUA as git-send-email would not work...
Toggle snippet (12 lines)
$ git send-email outgoing/0000-cover-letter.patch -a \
--to=guix-patches@gnu.org
outgoing/0000-cover-letter.patch
Can't exec "sh": No such file or directory at
/gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real
line 231.
the editor exited uncleanly, aborting everything at
/gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real
line 252.

Hoping this finds you well nonetheless.

Cheers,
--
tony / Antoine R. Dumont (@ardumont)

-----------------------------------------------------------------
gpg fingerprint BF00 203D 741A C9D5 46A8 BE07 52E2 E984 0D10 C3B8


===File ~/repo/public/guix/guix/outgoing/0000-cover-letter.patch===
From 372b2b9660b8293eebd6280bb46a4ec07d4192a7 Mon Sep 17 00:00:00 2001
From: "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
Date: Sat, 18 Mar 2023 16:27:22 +0100
Subject: [PATCH core-updates 0/6] Add `guix index` subcommand
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

*** BLURB HERE ****

Antoine R. Dumont (@ardumont) (5):
scripts-index: Transform `guix index` extension into a Guix script
scripts-index: Store outputs alongside packages
Makefile.am: Reference new script to compile
Bootstrap tests for guix index subcommand
Allow gcroot function to exceptionally ignore error

Ludovic Courtès (1):
index: Add initial implementation from civodul

Makefile.am | 2 +
guix/scripts/home.scm | 2 +-
guix/scripts/index.scm | 595 +++++++++++++++++++++++++++++++++++++++++
guix/store/roots.scm | 10 +-
tests/guix-index.sh | 73 +++++
tests/store-roots.scm | 7 +-
6 files changed, 686 insertions(+), 3 deletions(-)
create mode 100644 guix/scripts/index.scm
create mode 100755 tests/guix-index.sh


base-commit: 962277fd4313f20c0e0333effbd88352c0a7d461
--
2.36.1

============================================================
-----BEGIN PGP SIGNATURE-----

iQJUBAEBCgA+FiEEvwAgPXQaydVGqL4HUuLphA0Qw7gFAmQV4YIgHGFudG9pbmUu
cm9tYWluLmR1bW9udEBnbWFpbC5jb20ACgkQUuLphA0Qw7i3DBAAx/QDdyBGSKd5
fowy1gAbb3two+K3jAO5LK1WYdffyse5+I26T2j1sUupPeo8LAM7dK/J4Qnp9AON
hvc18/4zzge/lw5O7MmEOcMZLUFtrT7dtRFrT2B1n9GhdQVddLgfte8oH42hcxNu
cJphYpre96xb58lCh4v8JCvj28csPdY4kh37Xv5XaQ8fqqIEJsmNcDkN2bMpQZCj
3T4rzpyH7YBvZ0PBFwB2/XuFkvn0IKJVcMGjjHXM6XtaJqccV6dzsUYS+KGrTnbN
nN/AawTkXomBCQfrvXIk+LM6OMNsxA97dye/nKcfFfKvHyeA9oJ75pPnNUURVuXt
aeXoC7XXvNRYaliHDYKdXe2jrUnTdhOq0SXhI2O6W/wZBrhqDPmXjoOB9cosazhL
ZA0CFgQNfylB8BKIZc8gYnUyR7ldGg5ybvQ4NKhnzgJySAHpznvSTTGgOYFgUDVg
FLk3s95bI6CkKSSJ2ldSkva5Xl61SDxNTUIm/RTLJr3a3Ou36tVkzdlCo73wiw2N
9szvWNojOJnzxyqHFXPWQp99w/AxMz69Pnlu5FEp+trrCo3BgGZ7ZgcWcTCHRhft
UGkrIWPG3WVxcuxAn9l6zckJ6J3bB3xG+2C8vtleLCPTuQkhh7XoWFfcCvdQOeaC
ObL/YM1n6Yi/FDrTkad9x46O6PUyS2w=
=nK7i
-----END PGP SIGNATURE-----

A
A
Antoine R. Dumont wrote 5 days ago
[PATCH core-updates 1-6/6] Add `guix index` subcommand
(address . 62264@debbugs.gnu.org)
87r0tm576h.fsf@gmail.com
Hello again,

please find enclosed the remaining patches holding the actual guix
subcommand as described in the introductory email.

Cheers,
--
tony / Antoine R. Dumont (@ardumont)

-----------------------------------------------------------------
gpg fingerprint BF00 203D 741A C9D5 46A8 BE07 52E2 E984 0D10 C3B8
From 434b27de6227f5077505c1a1688a6ae500bbe56f Mon Sep 17 00:00:00 2001
From: "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
Date: Tue, 20 Dec 2022 16:05:50 +0100
Subject: [PATCH core-updates 2/6] scripts-index: Transform `guix index`
extension into a Guix script

---
guix/{extensions => scripts}/index.scm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
rename guix/{extensions => scripts}/index.scm (99%)

Toggle diff (27 lines)
diff --git a/guix/extensions/index.scm b/guix/scripts/index.scm
similarity index 99%
rename from guix/extensions/index.scm
rename to guix/scripts/index.scm
index d9894b213e..8d68a63847 100644
--- a/guix/extensions/index.scm
+++ b/guix/scripts/index.scm
@@ -16,7 +16,7 @@
;;; You should have received a copy of the GNU General Public License
;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
-(define-module (guix extensions index)
+(define-module (guix scripts index)
#:use-module ((guix i18n) #:select (G_))
#:use-module ((guix ui) #:select (show-version-and-exit
show-bug-report-information
@@ -484,7 +484,7 @@ (define %default-options
(with-method . "manifests")))
(define-command (guix-index . args)
- (category extension)
+ (category packaging)
(synopsis "Index packages to search package for a given filename")
(define (parse-sub-command arg result)
--
2.36.1
From ecea57fd4b46a8da5b78db17ceb7d8225a9e68e6 Mon Sep 17 00:00:00 2001
From: "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
Date: Fri, 24 Feb 2023 13:54:05 +0100
Subject: [PATCH core-updates 4/6] Makefile.am: Reference new script to compile

---
Makefile.am | 2 ++
1 file changed, 2 insertions(+)

Toggle diff (22 lines)
diff --git a/Makefile.am b/Makefile.am
index 23b939b674..6edd5eb900 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -302,6 +302,7 @@ MODULES = \
guix/scripts/archive.scm \
guix/scripts/import.scm \
guix/scripts/package.scm \
+ guix/scripts/index.scm \
guix/scripts/install.scm \
guix/scripts/remove.scm \
guix/scripts/upgrade.scm \
@@ -589,6 +590,7 @@ SH_TESTS = \
tests/guix-gc.sh \
tests/guix-git-authenticate.sh \
tests/guix-hash.sh \
+ tests/guix-index.sh \
tests/guix-pack.sh \
tests/guix-pack-localstatedir.sh \
tests/guix-pack-relocatable.sh \
--
2.36.1
From ae756e5add599fe0bb07547b5ff43ffa22f47da0 Mon Sep 17 00:00:00 2001
From: "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
Date: Fri, 24 Feb 2023 13:54:17 +0100
Subject: [PATCH core-updates 5/6] Bootstrap tests for guix index subcommand

---
guix/scripts/index.scm | 4 +++
tests/guix-index.sh | 73 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 77 insertions(+)
create mode 100755 tests/guix-index.sh

Toggle diff (96 lines)
diff --git a/guix/scripts/index.scm b/guix/scripts/index.scm
index d1478042ab..adf0f31269 100644
--- a/guix/scripts/index.scm
+++ b/guix/scripts/index.scm
@@ -555,6 +555,10 @@ (define (fail)
(with-error-handling
(let* ((opts (parse-command-line args %options
(list %default-options)
+ ;; ignore $GUIX_BUILD_OPTIONS
+ ;; otherwise, subcommand is not
+ ;; detected in the tests context
+ #:build-options? #f
#:argument-handler
parse-sub-command))
(args (option-arguments opts))
diff --git a/tests/guix-index.sh b/tests/guix-index.sh
new file mode 100755
index 0000000000..2c21d45a6b
--- /dev/null
+++ b/tests/guix-index.sh
@@ -0,0 +1,73 @@
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2013, 2014, 2015, 2019, 2020, 2023 Ludovic Courtès <ludo@gnu.org>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Test the 'guix index' command-line utility.
+#
+
+set -x
+
+tmpdir="guix-index-$$"
+trap 'rm -rf "$tmpdir"' EXIT
+
+guix index --version
+
+# Basic application to install and lookup through the index subcommand
+APPLICATION=guile-bootstrap
+
+# The subcommand exposes two indexation methods so far:
+# - manifests: fast and less exhaustive
+# - store: slow, exhaustive
+
+# In the following tests, we will store in 2 different dbs for both indexation
+# methods
+tmpdb_manifests="$tmpdir/manifests/db.sqlite"
+tmpdb_store="$tmpdir/store/db.sqlite"
+
+echo "### Preparing db locations for both indexation methods"
+mkdir -p `dirname $tmpdb_manifests` `dirname $tmpdb_store`
+
+cmd_manifests="guix index --db-path=$tmpdb_manifests --method=manifests"
+cmd_store="guix index --db-path=$tmpdb_store --method=store"
+
+echo "### Lookup without any db should fail"
+! $cmd_manifests search "$APPLICATION"
+! $cmd_store search "$APPLICATION"
+
+echo "### Initializing db with bare guix store should work"
+$cmd_manifests
+# ! $cmd_store
+
+echo "### lookup without anything in db should yield no result"
+! test `$cmd_manifests search "$APPLICATION"`
+# ! test `$cmd_store search "$APPLICATION"`
+
+echo "### Add some package to the temporary store"
+guix package --bootstrap \
+ --install $APPLICATION \
+ --profile=$tmpdir/profile
+
+echo "### Both both indexation call should work"
+# Testing indexation should work for both method
+test `$cmd_manifests`
+# test `$cmd_store`
+
+echo "### lookup indexed '$APPLICATION' should yield result"
+
+test `$cmd_manifests search "$APPLICATION"`
+# test `$cmd_store search "$APPLICATION"`
--
2.36.1
From 372b2b9660b8293eebd6280bb46a4ec07d4192a7 Mon Sep 17 00:00:00 2001
From: "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
Date: Mon, 13 Mar 2023 13:52:38 +0100
Subject: [PATCH core-updates 6/6] Allow gcroot function to exceptionally
ignore error
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored with Ludovic Courtès <ludo@gnu.org>
---
guix/store/roots.scm | 10 +++++++++-
tests/store-roots.scm | 7 ++++++-
2 files changed, 15 insertions(+), 2 deletions(-)

Toggle diff (53 lines)
diff --git a/guix/store/roots.scm b/guix/store/roots.scm
index 222f69c5c0..c2b15c33f0 100644
--- a/guix/store/roots.scm
+++ b/guix/store/roots.scm
@@ -105,7 +105,15 @@ (define canonical-root
(map (match-lambda
((file . properties)
(cons (scope file) properties)))
- (scandir* directory regular?)))))
+ (catch 'system-error
+ (lambda ()
+ (scandir* directory regular?))
+ (lambda args
+ (if (= ENOENT
+ (system-error-errno
+ args))
+ '()
+ (apply throw args))))))))
(loop (append rest (map first sub-directories))
(append (map canonical-root (filter symlink? files))
roots)
diff --git a/tests/store-roots.scm b/tests/store-roots.scm
index 5bcf1bc87e..00a4fe7931 100644
--- a/tests/store-roots.scm
+++ b/tests/store-roots.scm
@@ -1,5 +1,5 @@
;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2019 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2019, 2023 Ludovic Courtès <ludo@gnu.org>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -21,6 +21,7 @@ (define-module (test-store-deduplication)
#:use-module (guix store)
#:use-module (guix store roots)
#:use-module ((guix utils) #:select (call-with-temporary-directory))
+ #:use-module ((guix config) #:select (%state-directory))
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-64))
@@ -29,6 +30,10 @@ (define %store
(test-begin "store-roots")
+(test-equal "gc-roots, initial"
+ (list (string-append %state-directory "/profiles"))
+ (gc-roots))
+
(test-assert "gc-roots, regular root"
(let* ((item (add-text-to-store %store "something"
(random-text)))
--
2.36.1
-----BEGIN PGP SIGNATURE-----

iQJUBAEBCgA+FiEEvwAgPXQaydVGqL4HUuLphA0Qw7gFAmQV7WcgHGFudG9pbmUu
cm9tYWluLmR1bW9udEBnbWFpbC5jb20ACgkQUuLphA0Qw7gQmg//atg37DoXGnRe
YJBPmGHmnGnL+IGxKJKsRWaxVDqXTAyFZFWFth4UfeLNQBy63frEBWEUQwKs/8l3
cnDBJ6W9/pyv6mLsd5tDePxLN9wm9OPRlBm6QPdzqt7S+7Xl/BP/Elb1dyvzzRC7
WX4fmqJSpT6EUR5CKoNb4V/yO4clb+bgvsz2v1KK1qdPbH2ikgEKrD2cKkt7m3uQ
vebA4h4BXkdzwpdvxLT60cfGq+xXRjrVW6SrhrW8fJP8AEs1k4bKgFTFjx038ebv
8YLiHpuwYXzj1xBDiwhpHOSE6Y999psovb11Bd3Fvyf0xMf+Z8rZ78G5JhFwCZIR
jtGFWtM9uXTC+4VwBiR62/10fFhMkRtllqQRMkJCTTxqHMDFk8zEhKuiGckXM1bn
6qe5idCn8UW/y/GhDYjcIE9Fco7ClCft8C93iEpzp6gBB+Bm61xDreBm5MGkyp5R
Iv3AHFNTJEpOA4TQBovXE473rWiQscWhaVhbK4bS8VWRZLQvCRz2zUci6w6zykjh
nR+53WczL7z/pY8AhB6HuXcQvdpGNRodj5Fk44gxZaDO6g04FMxPC4251H4H51yr
Tyez8uT61HrREjThuGF/ET61Z/jSHEVdkpn4qHVKRIUSZjKOovXCjhDhcekvq1z/
A/nzaY3gIORKAoFZ8oIIfdg7to2LVsc=
=WV38
-----END PGP SIGNATURE-----

?