Hello, This introduces a new `guix index` subcommand (initiated by civodul). This is in charge of indexing guix packages to ease user lookups. There are 2 indexation methods: - `manifest`: fast but less efficient, expected to be used by local users - `store`: slow but more efficient, expected to be used on a guix build machine. Another service could be in charge of exposing a search package service to ease remote search of packages. ** Impact (positive) tl;dr This is an equivalent of `nix-index/nix-locate` or `apt-file search` cli we can find respectively in Nix or Debian's toolbox (unbeknownst of their implementation). This should help users transitioning from other distributions. Usually, people will want to look up for files or packages they are already using. They'd expect an easy way to find those back in guix. And we currently cannot do so easily. We can ask on #guix-devel irc (as someone mentions to me there) though that's not scalable nor really practical for non-irc users. ** How did I test/run it? I've been running both the: - `guix index` subcommand to index packages (with both methods in separated or not dbs). - `guix index search FILE` to actually search for packages present in the db I've inspected time and again the db through the `sqlite3` program after multiple runs too to ensure data were consistent. Data are consistent. ** Build? The makefile routine will build that subcommand too. ** Development details The backend technology is sqlite3. The command is in charge of migrating the data model (if any). That data model migration process is transparent for users. The data model is currently at version 3 because it underwent some migrations during its development process already. It's kept for dogfooding and examples reasons. The --version flag mentions the default backend locations and data model version: --8<---------------cut here---------------end--------------->8--- $ guix index --version Extension local cache database: - path: /home/tony/.cache/guix/index/db.sqlite - version: 3 guix index (GNU Guix) 1.4.0.3874-372b2 Copyright (C) 2023 the Guix authors License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. $ git log | head -1 commit 372b2b9660b8293eebd6280bb46a4ec07d4192a7 --8<---------------cut here---------------end--------------->8--- ** Current caveat ("help needed on test") The code works but... I can't make the integration test i wrote work. Hence my delay to send the overall patch (as the subcommand has been ready for a while). As it's been way too long already, I've posted it in the hope someone interested enough sees what's missing. Thanks in advance. What's happening during the test is that somehow the execution of the `guix index` subcommand triggers an exit code > 0 (error). Which, expectedly, stops the test immediately. I don't get why the subcommand exits that way and I don't see how to dig further unfortunately. Any clues would be very much appreciated. ** Use The subcommand is documented exhaustively. --8<---------------cut here---------------end--------------->8--- $ guix index --help Usage: guix index [OPTIONS...] [search FILE...] Without argument, indexes (package, file) relationships from the machine. This allows indexation with 2 methods, out of the local: - manifests: This is the fastest implementation with the caveat of indexing less packages. That'd be typically the use case of user local indexation. - store: This is slowest implementation. It discusses with the store daemon. That'd be typically the use case of building the largest db in one of the build farm node. With 'search FILE', search for packages installing FILE. Note: Internal cache is located at ~/.cache/guix/index/db.sqlite by default. See --db-path for customization. The valid values for OPTIONS are: -h, --help Display this help and exit -V, --version Display version information and exit --db-path=DIR Change default location of the cache db --method=METH Change default indexation method. By default it uses the local "manifests" (faster). It can also uses the local "store" (slower, typically on the farm build ci). The valid values for ARGS are: search FILE Search for packages installing the FILE (from cache db) Without any argument, it index packages. This fills in the db cache using whatever indexation method is defined. Report bugs to: bug-guix@gnu.org. GNU Guix home page: General help using Guix and GNU software: --8<---------------cut here---------------end--------------->8--- Example of an indexation (fast method by default): --8<---------------cut here---------------end--------------->8--- $ guix index Registering 133 packages ▕█████▎ ▏ --8<---------------cut here---------------end--------------->8--- Example of a search: --8<---------------cut here---------------end--------------->8--- $ guix index search sqlite3 sqlite@3.37.0 /gnu/store/jd6nn2c8ln5flv4vwl7pp1w804w2i9wj-sqlite-3.37.0/bin/sqlite3 sqlite@3.36.0 /gnu/store/xmzx5mzv4863yw9kmr2ykndgp37p8if0-sqlite-3.36.0/bin/sqlite3 --8<---------------cut here---------------end--------------->8--- Note: inspired from zimoun's way of quoting (/me like those, thx ;) ** Development process (optional read) Note that this started as a subcommand (again from civodul's bootstrap code). Then as a good proposal from zimoun, this got simplified into a guix extension (awesome work zimoun ;). Fwiw, developing it as an extension felt way simpler for dev and run. Versus, having to setup my machines to develop with guix. It continued as a guix subcommand in the end as civodul proposed it that way instead. Sent from my MUA as git-send-email would not work... --8<---------------cut here---------------end--------------->8--- $ git send-email outgoing/0000-cover-letter.patch -a \ --to=guix-patches@gnu.org outgoing/0000-cover-letter.patch Can't exec "sh": No such file or directory at /gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real line 231. the editor exited uncleanly, aborting everything at /gnu/store/xsj89hs359iblyxxi74pw6pyprdfbm5m-git-2.36.1-send-email/libexec/git-core/.git-send-email-real line 252. --8<---------------cut here---------------end--------------->8--- Hoping this finds you well nonetheless. Cheers, -- tony / Antoine R. Dumont (@ardumont) ----------------------------------------------------------------- gpg fingerprint BF00 203D 741A C9D5 46A8 BE07 52E2 E984 0D10 C3B8 ===File ~/repo/public/guix/guix/outgoing/0000-cover-letter.patch=== From 372b2b9660b8293eebd6280bb46a4ec07d4192a7 Mon Sep 17 00:00:00 2001 From: "Antoine R. Dumont (@ardumont)" Date: Sat, 18 Mar 2023 16:27:22 +0100 Subject: [PATCH core-updates 0/6] Add `guix index` subcommand MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit *** BLURB HERE **** Antoine R. Dumont (@ardumont) (5): scripts-index: Transform `guix index` extension into a Guix script scripts-index: Store outputs alongside packages Makefile.am: Reference new script to compile Bootstrap tests for guix index subcommand Allow gcroot function to exceptionally ignore error Ludovic Courtès (1): index: Add initial implementation from civodul Makefile.am | 2 + guix/scripts/home.scm | 2 +- guix/scripts/index.scm | 595 +++++++++++++++++++++++++++++++++++++++++ guix/store/roots.scm | 10 +- tests/guix-index.sh | 73 +++++ tests/store-roots.scm | 7 +- 6 files changed, 686 insertions(+), 3 deletions(-) create mode 100644 guix/scripts/index.scm create mode 100755 tests/guix-index.sh base-commit: 962277fd4313f20c0e0333effbd88352c0a7d461 -- 2.36.1 ============================================================