From debbugs-submit-bounces@debbugs.gnu.org Sun May 03 14:10:29 2020 Received: (at 39258) by debbugs.gnu.org; 3 May 2020 18:10:29 +0000 Received: from localhost ([127.0.0.1]:57994 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jVJ4P-0003sE-AS for submit@debbugs.gnu.org; Sun, 03 May 2020 14:10:29 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:45766) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jVJ4N-0003s2-Cy for 39258@debbugs.gnu.org; Sun, 03 May 2020 14:10:27 -0400 Received: by mail-qt1-f196.google.com with SMTP id 71so12145831qtc.12 for <39258@debbugs.gnu.org>; Sun, 03 May 2020 11:10:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=7kYCtT1FhcggnkrKSTbjZGekDA8sRbduJ/AL7R0h/zE=; b=KTOwmFgQUVmgaEuBqP5KdVN6+FM1WF07h5BMPGJPJVa9pqo82v+5EXx5HxqifBEqeA lmWr94eiDfielYzsShW5ufY2VlixRGoIyZLUNS1AEdgIyFUia6dOdusHM3MbTAl820r6 O5YAMivzHF5hgKIwImUoXK8EZSX/MrqOMhIP3DpCU4ZGD7ElWeJ7ttZlt/YJxrZQaBaU PyBiPPJK6b1fz+vp9JQndVI1pwhSaPrnABB3VG4APHwzYi29iZKLmju/8eD95t+9fOzP i9nFNuQ67O1tCpl0xLn8JavQB3ZrhFVqAPCija6lA+Mm7ZEww/XyXQNdt00wwx0wt6RU 5fbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=7kYCtT1FhcggnkrKSTbjZGekDA8sRbduJ/AL7R0h/zE=; b=c1UiP3vRXYpAMHdK6f0TPJuSnsYjVwACasmMvLHQlAo6FJGvUw7cnsTQ9ceWpd+b+i 6rwSH1wzUx71A6bJ66sWo46luiQckLBJes+Jq9ez8N4fF7P1qpR5mRMDmpGnLR2whehy 9nPyPVIC7ODwotvGYZXJHlAfkkCpo14zsCapRsLmgpdWkHXW1aPfVOeIlR1pGXG017J+ Xz3OU+oMLPwn0QqvfV4ylWl3NmV1sPx+Zgjsfm3JAgflhMFCd974NQdYAHM4g+fclrA0 MffqP+kG4gLLa8FIOkZzbGNO9AyUuXFSMt4RlEJBssyH3cXmtx6PFh5yHJgAogK6YrKG iEwg== X-Gm-Message-State: AGi0PuYD9HSDoyUhkKmj+UC+ONpDLB0K05LW0heuraoH+j02LPqO2oph 8K4WByDCo1k645zPcuCQnf2qMAbxaZXrUSufwyQ= X-Google-Smtp-Source: APiQypJ0OEWSOv2csOgkkS35VqviNxFQjjfvPJjdc/utUeOm6v76m6Xoax3dZGu7AH6yPGB1Mom5LTliIEhU8uSRTco= X-Received: by 2002:ac8:19f5:: with SMTP id s50mr13931499qtk.186.1588529421693; Sun, 03 May 2020 11:10:21 -0700 (PDT) MIME-Version: 1.0 References: <20200503150154.26532-1-zimon.toutoune@gmail.com> <87r1w1ynnm.fsf@gnu.org> In-Reply-To: <87r1w1ynnm.fsf@gnu.org> From: zimoun Date: Sun, 3 May 2020 20:10:10 +0200 Message-ID: Subject: Re: [PATCH v4 0/3] Faster cache generation (similar as v3) To: =?UTF-8?Q?Ludovic_Court=C3=A8s?= Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 39258 Cc: Arun Isaac , Pierre Neidhardt , 39258@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludo, On Sun, 3 May 2020 at 18:43, Ludovic Court=C3=A8s wrote: > > Therefore the cache '/lib/guix/package.cache' contains more > > information. > > This breaks the binary interface, so we=E2=80=99ll have to analyze the im= pact of > such a change and devise a strategy. Interface between what and what? Because from my understanding, this file is only used by only one guix. What do I miss? Note that I have read your comment in v3 2/3 but I did not understand it. S= orry. --8<---------------cut here---------------start------------->8--- I realize the other cache also has that problem, but it would be nice to add a version tag to the cache. Basically emit something like: (package-metadata-cache (version 0) VECTOR =E2=80=A6) instead of just: (VECTOR =E2=80=A6) --8<---------------cut here---------------end--------------->8--- > > (The v4 structure of 'package.cache' is a quick draft, so details > > should be discussed and an interesting move should to have a > > structured (binary and all strings) S-exp; because it should become an > > entry point to export the packages list to JSON. WDYT?) > > It=E2=80=99s on purpose that this cache is an object file: it just needs = to be > mmap=E2=80=99d, and that=E2=80=99s it. It=E2=80=99s the cheapest possibl= e way to do it. > Parsing sexps would be more costly, and since we=E2=80=99re talking about > startup time, this is sensitive. I agree and I have badly worded or I misunderstand something. For example, 'supported-systems' is saved as a list of strings, whereas 'license' is expanded as 3 strings without be packed in a list of strings. From my point of view, it is inconsistent and I do not know what is the best (readibility, startup time, etc.). > > To be clear about BM25 and caching, what I have in mind is: > > 1. "guix search --build-index" optionally done by the user if they wa= nts for example the BM25 ranking. > > Something that must be done explicitly doesn=E2=80=99t seem great to me. = As a > user, I=E2=80=99d rather not think about search indexes and all. But I d= on=E2=80=99t > know, maybe if it happened automatically on the first =E2=80=98guix searc= h=E2=80=99 > invocation that=E2=80=99d be fine. I do not think it is an option to build the BM25 the first time "guix search" is called. Back-to-envelop estimation, it needs ~25 seconds to Xapian* to do so. From my point of view, two options: a) "guix pull" does this extra ~25 seconds (compared to 10 seconds to build the v4 cache) b) the user manually build the index (I agree it is awkward!) Well, the first question is to evaluate if it is worth -- I am using the v2 version based on Xapian to have an idea. Please if you have suggestions about query (terms an user could type) and results (packages an user could expect), there are welcome. *Xapian: I do not think we could do better but I have not checked yet if there is a bottleneck Guix, Guile-Xapian and Xapian. > > 1. The name of 'fold-packages*' should be misleading since it does not= return "true" packages. > > Did you see =E2=80=98fold-available-packages=E2=80=99? It seems you coul= d extend it > instead of introducing =E2=80=98fold-packages*=E2=80=99, no? Yes and no. a) 'fold-available-packages' requires to modify the 'lambda' in 'find-package-by-description', b) 'fold-package*' returning a 'package' is less tweaks, IMHO. Well, I agree that on the long term, what 'fold-package*' does could be done by 'fold-available-packages' with the adequate 'proc'. Thank you for the suggestion; even if once re-read correctly v3 2/3 you already mentioned it. :-) > > 2. The function 'package->recutils' in 'guix/ui.scm' is modified but i= t is not the better. > > > > (match (package-supported-systems p) > > (('cache supported-systems) > > (string-join supported-systems)) > > (_ > > (string-join (package-transitive-supported-systems p))))) > > > > However it avoids to duplicate code; as it is done in version v3. > > I made suggestions to Arun=E2=80=99s v3 about the API here. Essentially,= I > think I proposed having a procedure that takes the list of fields as > keyword parameters, and =E2=80=98package->recutils=E2=80=99 would just de= legate to that. Yes, it was already your suggestion in v3 3/3. Do you suggest to refactor 'package->recutils'? For example, --8<---------------cut here---------------start------------->8--- (define* (package->recutils name version ... all-the-other-fields ... port #:optional (width (%text-width)) #:key (hyperlinks? (supports-hyperlinks? port)) (extra-fields '())) --8<---------------cut here---------------end--------------->8--- > > 4. Impolite '@@' is used to access the private license construction. > > (guix licenses) could provide a =E2=80=98string->license=E2=80=99 procedu= re. Well, do you suggest: (define (string->license name) (license name #f #f)) ? Skipping 'uri' and 'comment'? Naive question: what is the purpose of these 2 fields? Because there are not exposed at the CLI level, AFAIK, and I do not think an user evaluate '(license-uri pkg)' in a script. Well, I think that the hyperlink feature could be used to display the license URI too. WDYT? > Stopping here for now because I=E2=80=99m sorta drowning in patch review.= :-) Thank you for all the comments. > Thanks for exploring this design space, we=E2=80=99re making progress! My pleasure. Scheme is designed to explore. ;-) Cheers, simon