[PATCH core-updates 0/6] Add `guix index` subcommand

Open

Details

One participant

Antoine R. Dumont (@ardumont)

Owner: unassigned

Submitted by: Antoine R. Dumont (@ardumont)

Severity: normal

Antoine R. Dumont (@ardumont) wrote 5 days ago

Recipients:(address . guix-patches@gnu.org)

Message-ID:87v8iy59j1.fsf@gmail.com

Hello,

This introduces a new `guix index` subcommand (initiated by

civodul). This is in charge of indexing guix packages to ease user

lookups. There are 2 indexation methods:

- `manifest`: fast but less efficient, expected to be used by local

users

- `store`: slow but more efficient, expected to be used on a guix build

machine. Another service could be in charge of exposing a search

package service to ease remote search of packages.

** Impact (positive)

tl;dr This is an equivalent of `nix-index/nix-locate` or `apt-file

search` cli we can find respectively in Nix or Debian's toolbox

(unbeknownst of their implementation).

This should help users transitioning from other distributions. Usually,

people will want to look up for files or packages they are already

using. They'd expect an easy way to find those back in guix. And we

currently cannot do so easily. We can ask on #guix-devel irc (as someone

mentions to me there) though that's not scalable nor really practical

for non-irc users.

** How did I test/run it?

I've been running both the:

- `guix index` subcommand to index packages (with both methods in

separated or not dbs).

- `guix index search FILE` to actually search for packages present in

the db

I've inspected time and again the db through the `sqlite3` program after

multiple runs too to ensure data were consistent. Data are consistent.

** Build?

The makefile routine will build that subcommand too.

** Development details

The backend technology is sqlite3. The command is in charge of migrating

the data model (if any). That data model migration process is

transparent for users. The data model is currently at version 3 because

it underwent some migrations during its development process

already. It's kept for dogfooding and examples reasons.

The --version flag mentions the default backend locations and data model

version:

Toggle snippet (14 lines)

$ guix index --version

Extension local cache database:

- path: /home/tony/.cache/guix/index/db.sqlite

- version: 3

guix index (GNU Guix) 1.4.0.3874-372b2

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.

$ git log | head -1

commit 372b2b9660b8293eebd6280bb46a4ec07d4192a7

** Current caveat ("help needed on test")

The code works but...

I can't make the integration test i wrote work. Hence my delay to send

the overall patch (as the subcommand has been ready for a while). As

it's been way too long already, I've posted it in the hope someone

interested enough sees what's missing. Thanks in advance.

What's happening during the test is that somehow the execution of the

`guix index` subcommand triggers an exit code > 0 (error). Which,

expectedly, stops the test immediately. I don't get why the subcommand

exits that way and I don't see how to dig further unfortunately. Any

clues would be very much appreciated.

** Use

The subcommand is documented exhaustively.

Toggle snippet (37 lines)

$ guix index --help

Usage: guix index [OPTIONS...] [search FILE...]

Without argument, indexes (package, file) relationships from the machine.

This allows indexation with 2 methods, out of the local:

- manifests: This is the fastest implementation with the caveat of indexing

less packages. That'd be typically the use case of user local indexation.

- store: This is slowest implementation. It discusses with the store

daemon. That'd be typically the use case of building the largest db in one of

the build farm node.

With 'search FILE', search for packages installing FILE.

Note: Internal cache is located at ~/.cache/guix/index/db.sqlite by default.

See --db-path for customization.

The valid values for OPTIONS are:

-h, --help Display this help and exit

-V, --version Display version information and exit

--db-path=DIR Change default location of the cache db

--method=METH Change default indexation method. By default it uses the

local "manifests" (faster). It can also uses the local

"store" (slower, typically on the farm build ci).

The valid values for ARGS are:

search FILE Search for packages installing the FILE (from cache db)

<EMPTY> Without any argument, it index packages. This fills in the

db cache using whatever indexation method is defined.

Report bugs to: bug-guix@gnu.org.

GNU Guix home page: <https://www.gnu.org/software/guix/>

General help using Guix and GNU software: <https://guix.gnu.org/en/help/>

Example of an indexation (fast method by default):

Toggle snippet (4 lines)

$ guix index

Registering 133 packages ▕█████▎ ▏

Example of a search:

Toggle snippet (5 lines)

$ guix index search sqlite3

sqlite@3.37.0 /gnu/store/jd6nn2c8ln5flv4vwl7pp1w804w2i9wj-sqlite-3.37.0/bin/sqlite3

sqlite@3.36.0 /gnu/store/xmzx5mzv4863yw9kmr2ykndgp37p8if0-sqlite-3.36.0/bin/sqlite3

Note: inspired from zimoun's way of quoting (/me like those, thx ;)

** Development process (optional read)

Note that this started as a subcommand (again from civodul's bootstrap

code). Then as a good proposal from zimoun, this got simplified into a

guix extension (awesome work zimoun ;). Fwiw, developing it as an

extension felt way simpler for dev and run. Versus, having to setup my

machines to develop with guix. It continued as a guix subcommand in the

end as civodul proposed it that way instead.

Sent from my MUA as git-send-email would not work...

Toggle snippet (12 lines)

$ git send-email outgoing/0000-cover-letter.patch -a \

--to=guix-patches@gnu.org

outgoing/0000-cover-letter.patch

Can't exec "sh": No such file or directory at

/gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real

line 231.

the editor exited uncleanly, aborting everything at

/gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real

line 252.

Hoping this finds you well nonetheless.

Cheers,

tony / Antoine R. Dumont (@ardumont)

-----------------------------------------------------------------

gpg fingerprint BF00 203D 741A C9D5 46A8 BE07 52E2 E984 0D10 C3B8

===File ~/repo/public/guix/guix/outgoing/0000-cover-letter.patch===

From 372b2b9660b8293eebd6280bb46a4ec07d4192a7 Mon Sep 17 00:00:00 2001

MIME-Version: 1.0

Content-Type: text/plain; charset=UTF-8

Content-Transfer-Encoding: 8bit

*** BLURB HERE ****

Antoine R. Dumont (@ardumont) (5):

scripts-index: Transform `guix index` extension into a Guix script

scripts-index: Store outputs alongside packages

Makefile.am: Reference new script to compile

Bootstrap tests for guix index subcommand

Allow gcroot function to exceptionally ignore error

Ludovic Courtès (1):

index: Add initial implementation from civodul

Makefile.am | 2 +

guix/scripts/home.scm | 2 +-

guix/scripts/index.scm | 595 +++++++++++++++++++++++++++++++++++++++++

guix/store/roots.scm | 10 +-

tests/guix-index.sh | 73 +++++

tests/store-roots.scm | 7 +-

6 files changed, 686 insertions(+), 3 deletions(-)

create mode 100644 guix/scripts/index.scm

create mode 100755 tests/guix-index.sh

base-commit: 962277fd4313f20c0e0333effbd88352c0a7d461

2.36.1

============================================================

Antoine R. Dumont wrote 5 days ago

[PATCH core-updates 1-6/6] Add `guix index` subcommand

Recipients:(address . 62264@debbugs.gnu.org)

Message-ID:87r0tm576h.fsf@gmail.com

Hello again,

please find enclosed the remaining patches holding the actual guix

subcommand as described in the introductory email.

Cheers,

tony / Antoine R. Dumont (@ardumont)

-----------------------------------------------------------------

gpg fingerprint BF00 203D 741A C9D5 46A8 BE07 52E2 E984 0D10 C3B8

Attachment: 0001-index-Add-initial-implementation-from-civodul.patch

From 434b27de6227f5077505c1a1688a6ae500bbe56f Mon Sep 17 00:00:00 2001
From: "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
Date: Tue, 20 Dec 2022 16:05:50 +0100
Subject: [PATCH core-updates 2/6] scripts-index: Transform `guix index`
 extension into a Guix script

---
 guix/{extensions => scripts}/index.scm | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
 rename guix/{extensions => scripts}/index.scm (99%)

Toggle diff (27 lines)diff --git a/guix/extensions/index.scm b/guix/scripts/index.scm
similarity index 99%
rename from guix/extensions/index.scm
rename to guix/scripts/index.scm
index d9894b213e..8d68a63847 100644
--- a/guix/extensions/index.scm
+++ b/guix/scripts/index.scm
@@ -16,7 +16,7 @@
 ;;; You should have received a copy of the GNU General Public License
 ;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
 
-(define-module (guix extensions index)
+(define-module (guix scripts index)
   #:use-module ((guix i18n) #:select (G_))
   #:use-module ((guix ui) #:select (show-version-and-exit
                                     show-bug-report-information
@@ -484,7 +484,7 @@ (define %default-options
     (with-method . "manifests")))
 
 (define-command (guix-index . args)
-  (category extension)
+  (category packaging)
   (synopsis "Index packages to search package for a given filename")
 
   (define (parse-sub-command arg result)
-- 
2.36.1

Attachment: 0003-scripts-index-Store-outputs-alongside-packages.patch

From ecea57fd4b46a8da5b78db17ceb7d8225a9e68e6 Mon Sep 17 00:00:00 2001

---

Makefile.am | 2 ++

1 file changed, 2 insertions(+)

Toggle diff (22 lines)diff --git a/Makefile.am b/Makefile.am
index 23b939b674..6edd5eb900 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -302,6 +302,7 @@ MODULES =					\
   guix/scripts/archive.scm			\
   guix/scripts/import.scm			\
   guix/scripts/package.scm			\
+  guix/scripts/index.scm			\
   guix/scripts/install.scm			\
   guix/scripts/remove.scm			\
   guix/scripts/upgrade.scm			\
@@ -589,6 +590,7 @@ SH_TESTS =					\
   tests/guix-gc.sh				\
   tests/guix-git-authenticate.sh		\
   tests/guix-hash.sh				\
+  tests/guix-index.sh				\
   tests/guix-pack.sh				\
   tests/guix-pack-localstatedir.sh		\
   tests/guix-pack-relocatable.sh		\
-- 
2.36.1

From ae756e5add599fe0bb07547b5ff43ffa22f47da0 Mon Sep 17 00:00:00 2001
From: "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
Date: Fri, 24 Feb 2023 13:54:17 +0100
Subject: [PATCH core-updates 5/6] Bootstrap tests for guix index subcommand

---
 guix/scripts/index.scm |  4 +++
 tests/guix-index.sh    | 73 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 77 insertions(+)
 create mode 100755 tests/guix-index.sh

Toggle diff (96 lines)diff --git a/guix/scripts/index.scm b/guix/scripts/index.scm
index d1478042ab..adf0f31269 100644
--- a/guix/scripts/index.scm
+++ b/guix/scripts/index.scm
@@ -555,6 +555,10 @@ (define (fail)
   (with-error-handling
     (let* ((opts        (parse-command-line args %options
                                             (list %default-options)
+                                            ;; ignore $GUIX_BUILD_OPTIONS
+                                            ;; otherwise, subcommand is not
+                                            ;; detected in the tests context
+                                            #:build-options? #f
                                             #:argument-handler
                                             parse-sub-command))
            (args        (option-arguments opts))
diff --git a/tests/guix-index.sh b/tests/guix-index.sh
new file mode 100755
index 0000000000..2c21d45a6b
--- /dev/null
+++ b/tests/guix-index.sh
@@ -0,0 +1,73 @@
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2013, 2014, 2015, 2019, 2020, 2023 Ludovic Courtès <ludo@gnu.org>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Test the 'guix index' command-line utility.
+#
+
+set -x
+
+tmpdir="guix-index-$$"
+trap 'rm -rf "$tmpdir"' EXIT
+
+guix index --version
+
+# Basic application to install and lookup through the index subcommand
+APPLICATION=guile-bootstrap
+
+# The subcommand exposes two indexation methods so far:
+# - manifests: fast and less exhaustive
+# - store: slow, exhaustive
+
+# In the following tests, we will store in 2 different dbs for both indexation
+# methods
+tmpdb_manifests="$tmpdir/manifests/db.sqlite"
+tmpdb_store="$tmpdir/store/db.sqlite"
+
+echo "### Preparing db locations for both indexation methods"
+mkdir -p `dirname $tmpdb_manifests` `dirname $tmpdb_store`
+
+cmd_manifests="guix index --db-path=$tmpdb_manifests --method=manifests"
+cmd_store="guix index --db-path=$tmpdb_store --method=store"
+
+echo "### Lookup without any db should fail"
+! $cmd_manifests search "$APPLICATION"
+! $cmd_store search "$APPLICATION"
+
+echo "### Initializing db with bare guix store should work"
+$cmd_manifests
+# ! $cmd_store
+
+echo "### lookup without anything in db should yield no result"
+! test `$cmd_manifests search "$APPLICATION"`
+# ! test `$cmd_store search "$APPLICATION"`
+
+echo "### Add some package to the temporary store"
+guix package --bootstrap \
+     --install $APPLICATION \
+     --profile=$tmpdir/profile
+
+echo "### Both both indexation call should work"
+# Testing indexation should work for both method
+test `$cmd_manifests`
+# test `$cmd_store`
+
+echo "### lookup indexed '$APPLICATION' should yield result"
+
+test `$cmd_manifests search "$APPLICATION"`
+# test `$cmd_store search "$APPLICATION"`
-- 
2.36.1

From 372b2b9660b8293eebd6280bb46a4ec07d4192a7 Mon Sep 17 00:00:00 2001
From: "Antoine R. Dumont (@ardumont)" <antoine.romain.dumont@gmail.com>
Date: Mon, 13 Mar 2023 13:52:38 +0100
Subject: [PATCH core-updates 6/6] Allow gcroot function to exceptionally
 ignore error
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Co-authored with Ludovic Courtès <ludo@gnu.org>
---
 guix/store/roots.scm  | 10 +++++++++-
 tests/store-roots.scm |  7 ++++++-
 2 files changed, 15 insertions(+), 2 deletions(-)

Toggle diff (53 lines)diff --git a/guix/store/roots.scm b/guix/store/roots.scm
index 222f69c5c0..c2b15c33f0 100644
--- a/guix/store/roots.scm
+++ b/guix/store/roots.scm
@@ -105,7 +105,15 @@ (define canonical-root
                                      (map (match-lambda
                                             ((file . properties)
                                              (cons (scope file) properties)))
-                                          (scandir* directory regular?)))))
+                                          (catch 'system-error
+                                            (lambda ()
+                                              (scandir* directory regular?))
+                                            (lambda args
+                                              (if (= ENOENT
+                                                     (system-error-errno
+                                                      args))
+                                                  '()
+                                                  (apply throw args))))))))
              (loop (append rest (map first sub-directories))
                    (append (map canonical-root (filter symlink? files))
                            roots)
diff --git a/tests/store-roots.scm b/tests/store-roots.scm
index 5bcf1bc87e..00a4fe7931 100644
--- a/tests/store-roots.scm
+++ b/tests/store-roots.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2019 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2019, 2023 Ludovic Courtès <ludo@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -21,6 +21,7 @@ (define-module (test-store-deduplication)
   #:use-module (guix store)
   #:use-module (guix store roots)
   #:use-module ((guix utils) #:select (call-with-temporary-directory))
+  #:use-module ((guix config) #:select (%state-directory))
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-64))
 
@@ -29,6 +30,10 @@ (define %store
 
 (test-begin "store-roots")
 
+(test-equal "gc-roots, initial"
+  (list (string-append %state-directory "/profiles"))
+  (gc-roots))
+
 (test-assert "gc-roots, regular root"
   (let* ((item (add-text-to-store %store "something"
                                   (random-text)))
-- 
2.36.1

Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 62264@debbugs.gnu.org

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date