[PATCH 0/21] Juliahub import script.

  • Open
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Nicolas Graves
  • Simon Tournier
Owner
unassigned
Submitted by
Nicolas Graves
Severity
normal
N
N
Nicolas Graves wrote on 15 Mar 13:47 +0100
(address . guix-patches@gnu.org)
871qlq89kz.fsf@ngraves.fr
Hi guix!

Took me quite more time than I would've liked, but I have a usable
juliahub scheme import script!

It seems there's still one edge case that isn't covered and revolves
around when Julia packagers don't properly tag their git repos (I've
only seen the case with SnoopPrecompile). There's the possibility to
rely on tree commit hashes from the General repository (since this is a
valid way to identify/store a git repo), but that needs some major
changes in the way latest-repository-commit works. Otherwise, it needs
to be done by hand. It might also not work for subpackages in
directories that are up-to-date on juliahub but not yet on github, I
haven't met this case yet.

I'm sending a patch series in the coming minutes.

It's detailed since I haven't swauased all commits, for readability, but
I can squash it further if necessary.

--
Best regards,
Nicolas Graves
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 01/21] import: juliahub: first script draft.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-1-ngraves@ngraves.fr
---
guix/import/go.scm | 6 +-
guix/import/juliahub.scm | 183 +++++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/juliahub.scm | 107 ++++++++++++++++++
4 files changed, 295 insertions(+), 3 deletions(-)
create mode 100644 guix/import/juliahub.scm
create mode 100644 guix/scripts/import/juliahub.scm

Toggle diff (335 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 69937f8a4d..f264715fbd 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -503,9 +503,11 @@ (define (transform-version version)
'(string-append "v" version)
'(go-version->git-ref version))))
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define* (vcs->origin vcs-type vcs-repo-url version
+ #:optional transform-version)
"Generate the `origin' block of a package depending on what type of source
-control system is being used."
+control system is being used. Optionally use the function TRANSFORM-VERSION
+which takes version as an input."
(case vcs-type
((git)
(git->origin vcs-repo-url `(tag-or-commit . ,version) transform-version))
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
new file mode 100644
index 0000000000..efe6abbb24
--- /dev/null
+++ b/guix/import/juliahub.scm
@@ -0,0 +1,183 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import juliahub)
+ #:use-module (ice-9 textual-ports)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 string-fun)
+ #:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-1)
+ #:use-module (guix http-client)
+ #:use-module (guix import utils)
+ #:use-module (guix import json)
+ #:use-module (guix base16)
+ #:use-module (guix base32)
+ #:use-module (guix packages)
+ #:use-module (guix upstream)
+ #:use-module (json)
+ #:use-module ((guix licenses) #:prefix license:)
+
+ #:export (juliahub->guix-package))
+
+(define (juliahub-uri name)
+ (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+ (port (http-fetch url #:text? #t))
+ (_ (get-line port))
+ (meta (get-line port))
+ (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*")
+ (redirect (match:substring (string-match regex meta))))
+ (close-port port)
+ (string-drop redirect 4)))
+
+(define (juliahub-url name)
+ (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+ (uri (juliahub-uri name)))
+ (string-append url uri "/")))
+
+(define (juliahub-slug-version name)
+ (let* ((uri (juliahub-uri name))
+ (slug (string-take uri 5))
+ (latest-version (string-drop uri 6)))
+ `(,slug ,latest-version)))
+
+(define (json->juliahub-direct-dependencies vector)
+ (if (vector? vector)
+ (filter-map
+ (lambda (el)
+ (let ((dep (json->juliahub-dependency el)))
+ (if (juliahub-dependency-direct? dep)
+ dep
+ #f)))
+ (vector->list vector))))
+
+;; Julia package.
+(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
+ json->juliahub-package
+ (homepage juliahub-package-homepage) ;string
+ (readme juliahub-package-readme) ;string
+ ;; (slug juliahub-package-slug) ;string
+ (version juliahub-package-version) ;string
+ (description juliahub-package-description) ;string
+ (dependencies
+ juliahub-package-dependencies "deps"
+ json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
+ ;; (lambda (vector)
+ ;; (map json->juliahub-dependency (vector->list vector))))
+ (url juliahub-package-url) ;string
+ (uuid juliahub-package-uuid) ;string
+ (license juliahub-package-license)) ;string
+
+(define-json-mapping <juliahub-dependency>
+ make-juliahub-dependency juliahub-dependency?
+ json->juliahub-dependency
+ (direct? juliahub-dependency-direct? "direct") ;boolean
+ (name juliahub-dependency-name) ;string
+ (uuid juliahub-dependency-uuid) ;string
+ (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
+ ;; (slug juliahub-dependency-slug) ;string
+
+(define (julia-name->guix-name name)
+ (string-append "julia-" (snake-case name)))
+
+(define* (juliahub-fetch name #:key (version #f))
+ "Return a <juliahub-package> record for package NAME, or #f on failure."
+ (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
+ json->juliahub-package))
+
+(define (make-julia-sexp name version uri hash home-page synopsis description
+ dependencies licenses)
+ "Return the `package' s-expression for a Julia package with the given NAME,
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
+ `(package
+ (name ,(julia-name->guix-name name))
+ (version ,version)
+ (source (origin
+ (method url-fetch)
+ (uri ,uri)
+ (sha256
+ (base32
+ "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
+ ;; ,(bytevector->nix-base32-string hash)
+ ))))
+ (build-system julia-build-system)
+ ,@(if (null? dependencies)
+ '()
+ `((inputs
+ (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ dependencies)))))
+ (synopsis ,synopsis)
+ (description ,description)
+ (home-page ,home-page)
+ (license ,(match licenses
+ (() #f)
+ ((license) (license->symbol license))
+ (_ `(list ,@(map license->symbol licenses)))))))
+
+(define* (juliahub->guix-package package-name
+ #:key version #:allow-other-keys)
+ "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
+`package' s-expression corresponding to that package, or #f on failure.
+Optionally include a VERSION string to fetch a specific version juliahub."
+ (let ((package (if version
+ (juliahub-fetch package-name version)
+ (juliahub-fetch package-name))))
+ (if package
+ (let* ((dependencies-names
+ (map juliahub-dependency-name
+ (juliahub-package-dependencies package)))
+ (licenses
+ (map spdx-string->license
+ (list (juliahub-package-license package)))))
+ (values (make-julia-sexp
+ package-name
+ (juliahub-package-version package)
+ (juliahub-package-url package)
+ "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
+ (juliahub-package-homepage package)
+ (juliahub-package-description package)
+ (beautify-description (juliahub-package-readme package))
+ (juliahub-package-dependencies package)
+ licenses)
+ dependencies-names))
+ (values #f '()))))
+
+(define* (import-release package #:key (version #f))
+ "Return an <upstream-source> for the latest release of PACKAGE."
+ (let* ((package-name (guix-package->juliahub-name package))
+ (package (juliahub-fetch package-name))
+ (version (or version (juliahub-version gem)))
+ (url (rubyjuliahubs-uri gem-name version)))
+ (upstream-source
+ (package (package-name package))
+ (version version)
+ (urls (list url)))))
+
+(define %juliahub-updater
+ (upstream-updater
+ (name 'juliahub)
+ (description "Updater for Juliahub packages")
+ (pred juliahub-package?)
+ (import import-release)))
+
+(define* (juliahub-recursive-import package-name #:optional version)
+ (recursive-import package-name
+ #:repo '()
+ #:repo->guix-package juliahub->guix-package
+ #:guix-name ruby-package-name
+ #:version version))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index f84a964a53..ef4e0b9cc6 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -47,7 +47,7 @@ (define %standard-import-options '())
(define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
"gem" "go" "cran" "crate" "texlive" "json" "opam"
- "minetest" "elm" "hexpm"))
+ "minetest" "elm" "hexpm" "juliahub"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/juliahub.scm b/guix/scripts/import/juliahub.scm
new file mode 100644
index 0000000000..1317c67aa3
--- /dev/null
+++ b/guix/scripts/import/juliahub.scm
@@ -0,0 +1,107 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import juliahub)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import juliahub)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:use-module (ice-9 receive)
+ #:export (guix-import-juliahub))
+
+
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import juliahub PACKAGE-NAME[@VERSION] Import and
+convert the Julia package for PACKAGE-NAME. Optionally, a version can be
+specified after the at-sign (@) character.\n"))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (display (G_ "
+ -r, --recursive generate package expressions for all Gem packages\
+ that are not yet in Guix"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import gem")))
+ (option '(#\r "recursive") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'recursive #t result)))
+ %standard-import-options))
+
+
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-juliahub . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (parse-command-line args %options (list %default-options)
+ #:build-options? #f))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((spec)
+ (receive (package-name package-version)
+ (package-name->name+version spec)
+ (let ((code (if (assoc-ref opts 'recursive)
+ (map (match-lambda
+ ((and ('package ('name name) . rest) pkg)
+ `(define-public ,(string->symbol name)
+ ,pkg))
+ (_ #f))
+ (juliahub-recursive-import package-name package-version))
+ (let ((sexp (juliahub->guix-package package-name #:version package-version)))
+ (if sexp sexp #f)))))
+ (match code
+ ((or #f '(#f))
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-name))
+ (_ code)))))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%"))))))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 02/21] import: utils: Change git->origin function to git->origin+version.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-2-ngraves@ngraves.fr
---
guix/import/go.scm | 2 +-
guix/import/utils.scm | 50 +++++++++++++++++++++++++++----------------
2 files changed, 33 insertions(+), 19 deletions(-)

Toggle diff (90 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index f264715fbd..3be82ed164 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -510,7 +510,7 @@ (define* (vcs->origin vcs-type vcs-repo-url version
which takes version as an input."
(case vcs-type
((git)
- (git->origin vcs-repo-url `(tag-or-commit . ,version) transform-version))
+ (git->origin vcs-repo-url `(tag-or-commit . ,version) transform-version))
((hg)
`(origin
(method hg-fetch)
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 529d7f11f8..171dca54e8 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -65,6 +65,7 @@ (define-module (guix import utils)
url-fetch
guix-hash-url
+ git->origin+dir
git->origin
package-names->package-inputs
@@ -156,9 +157,12 @@ (define (guix-hash-url filename)
"Return the hash of FILENAME in nix-base32 format."
(bytevector->nix-base32-string (file-sha256 filename)))
-(define* (git->origin repo-url ref #:optional ref->commit)
- "Generate the `origin' block of a package depending on the git source
-control system. REPO-URL or REF can be null."
+(define* (git->origin+dir repo-url ref #:optional ref->commit)
+ "Returns a generated `origin' block of a package depending on the git source
+control system, and the directory in the store where the package has been
+downloaded, in case further processing is necessary. REPO-URL or REF can be
+null. REF->COMMIT can be a function or #t, in which case the commit matching
+ref is used. If REF->COMMIT is not used, the value inside REF is used."
(let-values (((directory commit)
(with-store store
(latest-repository-commit store repo-url #:ref ref))))
@@ -172,21 +176,31 @@ (define* (git->origin repo-url ref #:optional ref->commit)
version)
(_
(ref->commit version)))))
- `(origin
- (method git-fetch)
- (uri (git-reference
- (url ,(and (not (eq? repo-url 'null)) repo-url))
- (commit ,vcommit)))
- (file-name (git-file-name name version))
- (sha256
- (base32
- ,(if (pair? ref)
- (bytevector->nix-base32-string
- (file-hash* directory
- ;; 'git-fetch' already filtered out the '.git' directory.
- #:select? (const #true)
- #:recursive? #true))
- #f)))))))
+ (values
+ `(origin
+ (method git-fetch)
+ (uri (git-reference
+ (url ,(and (not (eq? repo-url 'null)) repo-url))
+ (commit ,vcommit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ ,(if (pair? ref)
+ (bytevector->nix-base32-string
+ (file-hash* directory
+ ;; 'git-fetch' already filtered out the '.git' directory.
+ #:select? (const #true)
+ #:recursive? #true))
+ #f))))
+ directory))))
+
+(define* (git->origin repo-url ref #:optional ref->commit)
+ "Returns a generated `origin' block of a package depending on the git source
+control system. REPO-URL or REF can be null. REF->COMMIT can be a function or
+#t, in which case the commit matching ref is used. If REF->COMMIT is not used,
+the value inside REF is used."
+ (let-values (((origin _) (git->origin+dir repo-url ref ref->commit)))
+ origin))
(define %spdx-license-identifiers
;; https://spdx.org/licenses/
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 03/21] import: juliahub: Add support for native-inputs.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-3-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 105 ++++++++++++++++++++++++---------------
1 file changed, 64 insertions(+), 41 deletions(-)

Toggle diff (167 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index efe6abbb24..4544dee980 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -20,13 +20,14 @@ (define-module (guix import juliahub)
#:use-module (ice-9 textual-ports)
#:use-module (ice-9 regex)
#:use-module (ice-9 match)
+ #:use-module (ice-9 streams)
#:use-module (ice-9 string-fun)
- #:use-module (srfi srfi-9)
#:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-11)
#:use-module (guix http-client)
#:use-module (guix import utils)
#:use-module (guix import json)
- #:use-module (guix base16)
#:use-module (guix base32)
#:use-module (guix packages)
#:use-module (guix upstream)
@@ -66,19 +67,42 @@ (define (json->juliahub-direct-dependencies vector)
#f)))
(vector->list vector))))
+(define (ini-list->extra-dependencies lst)
+ (match lst
+ (('(extras) ooo ...)
+ (extra-list->extra-dependencies ooo))
+ (((tag) ooo ...)
+ (ini-list->extra-dependencies ooo))
+ ((attribute '= value ooo ...)
+ (ini-list->extra-dependencies ooo))
+ ('()
+ '())))
+
+(define (extra-list->extra-dependencies lst)
+ (match lst
+ ((attribute '= value ooo ...)
+ `(,(symbol->string attribute) ,@(extra-list->extra-dependencies ooo)))
+ (((tag) ooo ...)
+ '())
+ ('()
+ '())))
+
+(define (parse-extra-dependencies directory)
+ (let* ((port (open-input-file (string-append directory "/Project.toml")))
+ (ini-list (stream->list (port->stream port read))))
+ (close-port port)
+ (ini-list->extra-dependencies ini-list)))
+
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
json->juliahub-package
(homepage juliahub-package-homepage) ;string
(readme juliahub-package-readme) ;string
- ;; (slug juliahub-package-slug) ;string
(version juliahub-package-version) ;string
(description juliahub-package-description) ;string
- (dependencies
- juliahub-package-dependencies "deps"
+ (direct-dependencies
+ juliahub-package-direct-dependencies "deps"
json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
- ;; (lambda (vector)
- ;; (map json->juliahub-dependency (vector->list vector))))
(url juliahub-package-url) ;string
(uuid juliahub-package-uuid) ;string
(license juliahub-package-license)) ;string
@@ -90,7 +114,6 @@ (define-json-mapping <juliahub-dependency>
(name juliahub-dependency-name) ;string
(uuid juliahub-dependency-uuid) ;string
(versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
- ;; (slug juliahub-dependency-slug) ;string
(define (julia-name->guix-name name)
(string-append "julia-" (snake-case name)))
@@ -100,27 +123,25 @@ (define* (juliahub-fetch name #:key (version #f))
(and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
json->juliahub-package))
-(define (make-julia-sexp name version uri hash home-page synopsis description
- dependencies licenses)
+(define (make-julia-sexp name source home-page synopsis description
+ direct-dependencies test-dependencies-names licenses)
"Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
+TEST-DEPENDENCIES-NAMES and LICENSES."
`(package
(name ,(julia-name->guix-name name))
(version ,version)
- (source (origin
- (method url-fetch)
- (uri ,uri)
- (sha256
- (base32
- "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
- ;; ,(bytevector->nix-base32-string hash)
- ))))
+ (source ,source)
(build-system julia-build-system)
- ,@(if (null? dependencies)
+ ,@(if (null? direct-dependencies)
'()
- `((inputs
+ `((propagated-inputs
(list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- dependencies)))))
+ direct-dependencies)))))
+ ,@(if (null? test-dependencies-names)
+ '()
+ `((native-inputs
+ (list ,@(map julia-name->guix-name test-dependencies-names)))))
(synopsis ,synopsis)
(description ,description)
(home-page ,home-page)
@@ -135,26 +156,28 @@ (define* (juliahub->guix-package package-name
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
(let ((package (if version
- (juliahub-fetch package-name version)
- (juliahub-fetch package-name))))
+ (juliahub-fetch package-name version)
+ (juliahub-fetch package-name))))
(if package
- (let* ((dependencies-names
- (map juliahub-dependency-name
- (juliahub-package-dependencies package)))
- (licenses
- (map spdx-string->license
- (list (juliahub-package-license package)))))
- (values (make-julia-sexp
- package-name
- (juliahub-package-version package)
- (juliahub-package-url package)
- "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
- (juliahub-package-homepage package)
- (juliahub-package-description package)
- (beautify-description (juliahub-package-readme package))
- (juliahub-package-dependencies package)
- licenses)
- dependencies-names))
+ (let-values (((source directory)
+ (git->origin+dir url `(tag-or-commit . ,package-version))))
+ (let* ((dependencies-names
+ (map juliahub-dependency-name
+ (juliahub-package-direct-dependencies package)))
+ (licenses
+ (map spdx-string->license
+ (list (juliahub-package-license package))))
+ (test-dependencies-names (parse-extra-dependencies directory)))
+ (values (make-julia-sexp
+ package-name
+ source
+ (juliahub-package-homepage package)
+ (juliahub-package-description package)
+ (beautify-description (juliahub-package-readme package))
+ (juliahub-package-direct-dependencies package)
+ test-dependencies-names
+ licenses)
+ (append dependencies-names test-dependencies))))
(values #f '()))))
(define* (import-release package #:key (version #f))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 04/21] import: juliahub: Correct source parsing.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-4-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

Toggle diff (19 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 4544dee980..4c3ceed109 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -160,7 +160,11 @@ (define* (juliahub->guix-package package-name
(juliahub-fetch package-name))))
(if package
(let-values (((source directory)
- (git->origin+dir url `(tag-or-commit . ,package-version))))
+ (git->origin+dir
+ (juliahub-package-url package)
+ `(tag-or-commit
+ . ,(string-append
+ "v" (juliahub-package-version package))))))
(let* ((dependencies-names
(map juliahub-dependency-name
(juliahub-package-direct-dependencies package)))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 05/21] import: juliahub: Add indirect dependencies.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-5-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)

Toggle diff (42 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 4c3ceed109..fb361a0044 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -67,6 +67,16 @@ (define (json->juliahub-direct-dependencies vector)
#f)))
(vector->list vector))))
+(define (json->juliahub-indirect-dependencies vector)
+ (if (vector? vector)
+ (filter-map
+ (lambda (el)
+ (let ((dep (json->juliahub-dependency el)))
+ (if (not (juliahub-dependency-direct? dep))
+ dep
+ #f)))
+ (vector->list vector))))
+
(define (ini-list->extra-dependencies lst)
(match lst
(('(extras) ooo ...)
@@ -103,6 +113,9 @@ (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
(direct-dependencies
juliahub-package-direct-dependencies "deps"
json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
+ (indirect-dependencies
+ juliahub-package-indirect-dependencies "deps"
+ json->juliahub-indirect-dependencies) ;list of <juliahub-dependency>
(url juliahub-package-url) ;string
(uuid juliahub-package-uuid) ;string
(license juliahub-package-license)) ;string
@@ -181,7 +194,7 @@ (define* (juliahub->guix-package package-name
(juliahub-package-direct-dependencies package)
test-dependencies-names
licenses)
- (append dependencies-names test-dependencies))))
+ (append dependencies-names test-dependencies-names))))
(values #f '()))))
(define* (import-release package #:key (version #f))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 06/21] import: juliahub: Add updater and recursive-importer.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-6-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 22 +++++++++++++++++-----
1 file changed, 17 insertions(+), 5 deletions(-)

Toggle diff (54 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index fb361a0044..c38c830caa 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -34,7 +34,9 @@ (define-module (guix import juliahub)
#:use-module (json)
#:use-module ((guix licenses) #:prefix license:)
- #:export (juliahub->guix-package))
+ #:export (juliahub->guix-package
+ %juliahub-updater
+ juliahub-recursive-import))
(define (juliahub-uri name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
@@ -197,16 +199,26 @@ (define* (juliahub->guix-package package-name
(append dependencies-names test-dependencies-names))))
(values #f '()))))
+(define (guix-package->juliahub-name package)
+ (let* ((url (juliahub-package-url package))
+ (git-name (car (last-pair (string-split url #\/))))
+ (ungitted-name (if (string-suffix? ".git" git-name)
+ (string-drop-right git-name 4)
+ git-name))
+ (package-name (if (string-suffix? ".jl" ungitted-name)
+ (string-drop-right ungitted-name 4)
+ ungitted-name)))
+ package-name))
+
(define* (import-release package #:key (version #f))
"Return an <upstream-source> for the latest release of PACKAGE."
(let* ((package-name (guix-package->juliahub-name package))
(package (juliahub-fetch package-name))
- (version (or version (juliahub-version gem)))
- (url (rubyjuliahubs-uri gem-name version)))
+ (version (or version (juliahub-package-version package))))
(upstream-source
(package (package-name package))
(version version)
- (urls (list url)))))
+ (urls (list (juliahub-package-url package))))))
(define %juliahub-updater
(upstream-updater
@@ -219,5 +231,5 @@ (define* (juliahub-recursive-import package-name #:optional version)
(recursive-import package-name
#:repo '()
#:repo->guix-package juliahub->guix-package
- #:guix-name ruby-package-name
+ #:guix-name julia-name->guix-name
#:version version))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 07/21] import: juliahub: Filter out julia stdlibs.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-7-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 49 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 47 insertions(+), 2 deletions(-)

Toggle diff (76 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index c38c830caa..af08f3d698 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -38,6 +38,47 @@ (define-module (guix import juliahub)
%juliahub-updater
juliahub-recursive-import))
+;; To update, see file sysimg.jl
+(define %julia-stdlibs
+ (list "julia"
+ "ArgTools"
+ "Artifacts"
+ "Base64"
+ "CRC32c"
+ "FileWatching"
+ "Libdl"
+ "Logging"
+ "Mmap"
+ "NetworkOptions"
+ "SHA"
+ "Serialization"
+ "Sockets"
+ "Unicode"
+ "DelimitedFiles"
+ "LinearAlgebra"
+ "Markdown"
+ "Printf"
+ "Random"
+ "Tar"
+ "Dates"
+ "Distributed"
+ "Future"
+ "InteractiveUtils"
+ "LibGit2"
+ "Profile"
+ "SparseArrays"
+ "UUIDs"
+ "REPL"
+ "SharedArrays"
+ "Statistics"
+ "SuiteSparse"
+ "TOML"
+ "Test"
+ "LibCURL"
+ "Downloads"
+ "Pkg"
+ "LazyArtifacts"))
+
(define (juliahub-uri name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
(port (http-fetch url #:text? #t))
@@ -64,7 +105,9 @@ (define (json->juliahub-direct-dependencies vector)
(filter-map
(lambda (el)
(let ((dep (json->juliahub-dependency el)))
- (if (juliahub-dependency-direct? dep)
+ (if (and (juliahub-dependency-direct? dep)
+ (not (member (juliahub-dependency-name dep)
+ %julia-stdlibs)))
dep
#f)))
(vector->list vector))))
@@ -74,7 +117,9 @@ (define (json->juliahub-indirect-dependencies vector)
(filter-map
(lambda (el)
(let ((dep (json->juliahub-dependency el)))
- (if (not (juliahub-dependency-direct? dep))
+ (if (and (not (juliahub-dependency-direct? dep))
+ (not (member (juliahub-dependency-name dep)
+ %julia-stdlibs)))
dep
#f)))
(vector->list vector))))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 08/21] import: juliahub: Simplify juliahub dependency management.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-8-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 42 ++++++++++++++--------------------------
1 file changed, 14 insertions(+), 28 deletions(-)

Toggle diff (80 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index af08f3d698..b1eeb736a8 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -100,26 +100,13 @@ (define (juliahub-slug-version name)
(latest-version (string-drop uri 6)))
`(,slug ,latest-version)))
-(define (json->juliahub-direct-dependencies vector)
+(define (json->juliahub-dependencies vector)
(if (vector? vector)
(filter-map
(lambda (el)
(let ((dep (json->juliahub-dependency el)))
- (if (and (juliahub-dependency-direct? dep)
- (not (member (juliahub-dependency-name dep)
- %julia-stdlibs)))
- dep
- #f)))
- (vector->list vector))))
-
-(define (json->juliahub-indirect-dependencies vector)
- (if (vector? vector)
- (filter-map
- (lambda (el)
- (let ((dep (json->juliahub-dependency el)))
- (if (and (not (juliahub-dependency-direct? dep))
- (not (member (juliahub-dependency-name dep)
- %julia-stdlibs)))
+ (if (not (member (juliahub-dependency-name dep)
+ %julia-stdlibs))
dep
#f)))
(vector->list vector))))
@@ -157,12 +144,9 @@ (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
(readme juliahub-package-readme) ;string
(version juliahub-package-version) ;string
(description juliahub-package-description) ;string
- (direct-dependencies
- juliahub-package-direct-dependencies "deps"
- json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
- (indirect-dependencies
- juliahub-package-indirect-dependencies "deps"
- json->juliahub-indirect-dependencies) ;list of <juliahub-dependency>
+ (dependencies
+ juliahub-package-dependencies "deps"
+ json->juliahub-dependencies) ;list of <juliahub-dependency>
(url juliahub-package-url) ;string
(uuid juliahub-package-uuid) ;string
(license juliahub-package-license)) ;string
@@ -184,7 +168,7 @@ (define* (juliahub-fetch name #:key (version #f))
json->juliahub-package))
(define (make-julia-sexp name source home-page synopsis description
- direct-dependencies test-dependencies-names licenses)
+ dependencies test-dependencies-names licenses)
"Return the `package' s-expression for a Julia package with the given NAME,
VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
TEST-DEPENDENCIES-NAMES and LICENSES."
@@ -193,11 +177,13 @@ (define (make-julia-sexp name source home-page synopsis description
(version ,version)
(source ,source)
(build-system julia-build-system)
- ,@(if (null? direct-dependencies)
- '()
- `((propagated-inputs
- (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- direct-dependencies)))))
+ ,@(let ((direct-dependencies
+ (filter julia-dependency-direct? dependencies)))
+ (if (null? direct-dependencies)
+ '()
+ `((propagated-inputs
+ (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ direct-dependencies))))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 09/21] import: juliahub: Improve dependency management.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-9-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 25 +++++++++++++------------
1 file changed, 13 insertions(+), 12 deletions(-)

Toggle diff (60 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index b1eeb736a8..fc25ba1d42 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -168,22 +168,20 @@ (define* (juliahub-fetch name #:key (version #f))
json->juliahub-package))
(define (make-julia-sexp name source home-page synopsis description
- dependencies test-dependencies-names licenses)
+ direct-dependencies test-dependencies-names licenses)
"Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
TEST-DEPENDENCIES-NAMES and LICENSES."
`(package
(name ,(julia-name->guix-name name))
(version ,version)
(source ,source)
(build-system julia-build-system)
- ,@(let ((direct-dependencies
- (filter julia-dependency-direct? dependencies)))
- (if (null? direct-dependencies)
- '()
- `((propagated-inputs
- (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- direct-dependencies))))))
+ ,@(if (null? direct-dependencies)
+ '()
+ `((propagated-inputs
+ (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ direct-dependencies)))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
@@ -211,9 +209,12 @@ (define* (juliahub->guix-package package-name
`(tag-or-commit
. ,(string-append
"v" (juliahub-package-version package))))))
- (let* ((dependencies-names
+ (let* ((direct-dependencies
+ (filter juliahub-dependency-direct?
+ (juliahub-package-dependencies package)))
+ (dependencies-names
(map juliahub-dependency-name
- (juliahub-package-direct-dependencies package)))
+ direct-dependencies))
(licenses
(map spdx-string->license
(list (juliahub-package-license package))))
@@ -224,7 +225,7 @@ (define* (juliahub->guix-package package-name
(juliahub-package-homepage package)
(juliahub-package-description package)
(beautify-description (juliahub-package-readme package))
- (juliahub-package-direct-dependencies package)
+ direct-dependencies
test-dependencies-names
licenses)
(append dependencies-names test-dependencies-names))))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 11/21] import: juliahub: Improve test dependencies parsing.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-11-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 30 +++++++++++++-----------------
1 file changed, 13 insertions(+), 17 deletions(-)

Toggle diff (59 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 5327e92325..2ea461b72a 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -154,31 +154,27 @@ (define (json->juliahub-dependencies vector)
#f)))
(vector->list vector))))
-(define (ini-list->extra-dependencies lst)
+(define (ini-list->test-dependencies lst)
(match lst
- (('(extras) ooo ...)
- (extra-list->extra-dependencies ooo))
- (((tag) ooo ...)
- (ini-list->extra-dependencies ooo))
- ((attribute '= value ooo ...)
- (ini-list->extra-dependencies ooo))
+ (('test '= ooo ...)
+ `(,(caar ooo) ,@(test-list->test-dependencies (cdar ooo))))
+ ((value ooo ...)
+ (ini-list->test-dependencies ooo))
('()
- '())))
+ '())))
-(define (extra-list->extra-dependencies lst)
+(define (test-list->test-dependencies lst)
(match lst
- ((attribute '= value ooo ...)
- `(,(symbol->string attribute) ,@(extra-list->extra-dependencies ooo)))
- (((tag) ooo ...)
- '())
+ ((('unquote value) ooo ...)
+ `(,value ,@(test-list->test-dependencies ooo)))
('()
- '())))
+ '())))
-(define (parse-extra-dependencies directory)
+(define (parse-test-dependencies directory)
(let* ((port (open-input-file (string-append directory "/Project.toml")))
(ini-list (stream->list (port->stream port read))))
(close-port port)
- (ini-list->extra-dependencies ini-list)))
+ (ini-list->test-dependencies ini-list)))
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
@@ -268,7 +264,7 @@ (define* (juliahub->guix-package package-name
(licenses
(map spdx-string->license
(list (juliahub-package-license package))))
- (test-dependencies-names (parse-extra-dependencies directory)))
+ (test-dependencies-names (parse-test-dependencies directory)))
(values (make-julia-sexp
package-name
source
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 10/21] import: juliahub: Add functions to parse the git repo for a git tag.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-10-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 62 ++++++++++++++++++++++++++++++++++++----
1 file changed, 56 insertions(+), 6 deletions(-)

Toggle diff (105 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index fc25ba1d42..5327e92325 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -26,6 +26,7 @@ (define-module (guix import juliahub)
#:use-module (srfi srfi-9)
#:use-module (srfi srfi-11)
#:use-module (guix http-client)
+ #:use-module (guix git)
#:use-module (guix import utils)
#:use-module (guix import json)
#:use-module (guix base32)
@@ -38,6 +39,48 @@ (define-module (guix import juliahub)
%juliahub-updater
juliahub-recursive-import))
+;; Juliahub may be more up-to-date than the General registry or the actual git
+;; tag (it seems around 6 hours pass between the time a commit is supplied to
+;; JuliaRegistrator as a release, and the time Julia TagBot Github Action makes
+;; the git tag). We have no simple way to get the commit of the latest-version.
+;; Thus the best simple thing we can do is get the latest-git-version, and
+;; import this version instead. We do this by parsing Package.toml in the General
+;; registry, and then getting the refs of the git repo supplied by this
+;; file. Parsing this file is also necessary if the package is in a subdir of a
+;; git repository, because the information isn't present in Juliahub.
+;; There's a last case where some Julia packages are not based on a particular
+;; git tag, and where looking for the commit is tedious and artificial. In these
+;; cases, we introduce the tree-commit which is available in the Versions.toml
+;; file in the General repository. This is equivalent to a commit, since we have
+;; a unique hash of the listing of files and directories, thus it can be used to
+;; identify the state of a repository.
+
+(define %general-base-url
+ "https://raw.githubusercontent.com/JuliaRegistries/General/master/")
+
+(define (general-url package-name file)
+ (let ((folder (string-capitalize (string-take package-name 1))))
+ (string-append
+ %general-base-url folder "/" package-name "/" file)))
+
+(define (ini-list->alist lst)
+ (match lst
+ ((attribute '= value ooo ...)
+ `((,attribute . ,value) ,@(ini-list->alist ooo)))
+ ('()
+ '())))
+
+(define (ini-fetch url)
+ (let* ((port (http-fetch url #:text? #t))
+ (ini-list (stream->list (port->stream port read))))
+ (close-port port)
+ (ini-list->alist ini-list)))
+
+(define (latest-git-tag repo)
+ (let* ((last-ref (last (remote-refs repo #:tags? #t)))
+ (last-git-tag (last (string-split last-ref #\/))))
+ (string-drop last-git-tag 1)))
+
;; To update, see file sysimg.jl
(define %julia-stdlibs
(list "julia"
@@ -194,14 +237,21 @@ (define (make-julia-sexp name source home-page synopsis description
((license) (license->symbol license))
(_ `(list ,@(map license->symbol licenses)))))))
+;; TODO handle subdir case properly.
+
(define* (juliahub->guix-package package-name
#:key version #:allow-other-keys)
"Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
- (let ((package (if version
- (juliahub-fetch package-name version)
- (juliahub-fetch package-name))))
+ (let* ((package-toml (ini-fetch (general-url name "Package.toml")))
+ (subdir (assoc-ref package-toml 'subdir))
+ (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+ (package (if version
+ (juliahub-fetch package-name version)
+ (if tag
+ (juliahub-fetch package-name tag)
+ (juliahub-fetch package-name)))))
(if package
(let-values (((source directory)
(git->origin+dir
@@ -233,13 +283,13 @@ (define* (juliahub->guix-package package-name
(define (guix-package->juliahub-name package)
(let* ((url (juliahub-package-url package))
- (git-name (car (last-pair (string-split url #\/))))
+ (git-name (last (string-split url #\/)))
(ungitted-name (if (string-suffix? ".git" git-name)
(string-drop-right git-name 4)
git-name))
(package-name (if (string-suffix? ".jl" ungitted-name)
- (string-drop-right ungitted-name 4)
- ungitted-name)))
+ (string-drop-right ungitted-name 4)
+ ungitted-name)))
package-name))
(define* (import-release package #:key (version #f))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 12/21] import: juliahub: Handle the case where we have a subdirectory.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-12-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)

Toggle diff (35 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 2ea461b72a..b646f93295 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -233,14 +233,12 @@ (define (make-julia-sexp name source home-page synopsis description
((license) (license->symbol license))
(_ `(list ,@(map license->symbol licenses)))))))
-;; TODO handle subdir case properly.
-
(define* (juliahub->guix-package package-name
#:key version #:allow-other-keys)
"Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
- (let* ((package-toml (ini-fetch (general-url name "Package.toml")))
+ (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
(subdir (assoc-ref package-toml 'subdir))
(tag (latest-git-tag (assoc-ref package-toml 'repo)))
(package (if version
@@ -264,7 +262,11 @@ (define* (juliahub->guix-package package-name
(licenses
(map spdx-string->license
(list (juliahub-package-license package))))
- (test-dependencies-names (parse-test-dependencies directory)))
+ (test-dependencies-names
+ (if subdir
+ (parse-test-dependencies
+ (string-append subdir "/" directory))
+ (parse-test-dependencies directory))))
(values (make-julia-sexp
package-name
source
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 13/21] import: juliahub: Add support for versions for juliahub-fetch.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-13-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 20 ++++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)

Toggle diff (57 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index b646f93295..6ce0487dba 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -122,7 +122,7 @@ (define %julia-stdlibs
"Pkg"
"LazyArtifacts"))
-(define (juliahub-uri name)
+(define (juliahub-redirect-uri name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
(port (http-fetch url #:text? #t))
(_ (get-line port))
@@ -134,11 +134,11 @@ (define (juliahub-uri name)
(define (juliahub-url name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
- (uri (juliahub-uri name)))
+ (uri (juliahub-redirect-uri name)))
(string-append url uri "/")))
-(define (juliahub-slug-version name)
- (let* ((uri (juliahub-uri name))
+(define (juliahub-slug+version name)
+ (let* ((uri (juliahub-redirect-uri name))
(slug (string-take uri 5))
(latest-version (string-drop uri 6)))
`(,slug ,latest-version)))
@@ -203,8 +203,12 @@ (define (julia-name->guix-name name)
(define* (juliahub-fetch name #:key (version #f))
"Return a <juliahub-package> record for package NAME, or #f on failure."
- (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
- json->juliahub-package))
+ (let ((url (if version
+ (string-append "https://docs.juliahub.com/" name "/"
+ (car (juliahub-slug+version name)) "/"
+ version "/pkg.json")
+ (string-append (juliahub-url name) "pkg.json"))))
+ (and=> (json-fetch url) json->juliahub-package)))
(define (make-julia-sexp name source home-page synopsis description
direct-dependencies test-dependencies-names licenses)
@@ -242,9 +246,9 @@ (define* (juliahub->guix-package package-name
(subdir (assoc-ref package-toml 'subdir))
(tag (latest-git-tag (assoc-ref package-toml 'repo)))
(package (if version
- (juliahub-fetch package-name version)
+ (juliahub-fetch package-name #:version version)
(if tag
- (juliahub-fetch package-name tag)
+ (juliahub-fetch package-name #:version tag)
(juliahub-fetch package-name)))))
(if package
(let-values (((source directory)
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 14/21] import: juliahub: Filter out stdlibs from test-dependencies.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-14-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

Toggle diff (16 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 6ce0487dba..1c7b029296 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -174,7 +174,8 @@ (define (parse-test-dependencies directory)
(let* ((port (open-input-file (string-append directory "/Project.toml")))
(ini-list (stream->list (port->stream port read))))
(close-port port)
- (ini-list->test-dependencies ini-list)))
+ (filter (lambda (x) (not (member x %julia-stdlibs)))
+ (ini-list->test-dependencies ini-list))))
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 15/21] import: juliahub: More robust toml regex parser.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-15-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 29 ++++++++---------------------
1 file changed, 8 insertions(+), 21 deletions(-)

Toggle diff (53 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 1c7b029296..3985d8d0be 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -154,28 +154,15 @@ (define (json->juliahub-dependencies vector)
#f)))
(vector->list vector))))
-(define (ini-list->test-dependencies lst)
- (match lst
- (('test '= ooo ...)
- `(,(caar ooo) ,@(test-list->test-dependencies (cdar ooo))))
- ((value ooo ...)
- (ini-list->test-dependencies ooo))
- ('()
- '())))
-
-(define (test-list->test-dependencies lst)
- (match lst
- ((('unquote value) ooo ...)
- `(,value ,@(test-list->test-dependencies ooo)))
- ('()
- '())))
-
(define (parse-test-dependencies directory)
(let* ((port (open-input-file (string-append directory "/Project.toml")))
- (ini-list (stream->list (port->stream port read))))
+ (project.toml (get-string-all port))
+ (regex "\ntest = \\[.*\\]")
+ (deps (match:substring (string-match regex project.toml)))
+ (pure (string-delete (list->char-set (list #\" #\ )) deps)))
(close-port port)
(filter (lambda (x) (not (member x %julia-stdlibs)))
- (ini-list->test-dependencies ini-list))))
+ (string-split (string-drop (string-drop-right pure 1) 7) #\,))))
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
@@ -243,9 +230,9 @@ (define* (juliahub->guix-package package-name
"Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
- (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
- (subdir (assoc-ref package-toml 'subdir))
- (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+ (let* ((package.toml (ini-fetch (general-url package-name "Package.toml")))
+ (subdir (assoc-ref package.toml 'subdir))
+ (tag (latest-git-tag (assoc-ref package.toml 'repo)))
(package (if version
(juliahub-fetch package-name #:version version)
(if tag
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 16/21] import: juliahub: Beautify description.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-16-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 60 +++++++++++++++++++++++++++++++++++++---
1 file changed, 56 insertions(+), 4 deletions(-)

Toggle diff (95 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 3985d8d0be..338f042441 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -32,8 +32,10 @@ (define-module (guix import juliahub)
#:use-module (guix base32)
#:use-module (guix packages)
#:use-module (guix upstream)
- #:use-module (json)
#:use-module ((guix licenses) #:prefix license:)
+ #:use-module (json)
+ #:use-module (htmlprag)
+ #:use-module (sxml transform)
#:export (juliahub->guix-package
%juliahub-updater
@@ -164,6 +166,53 @@ (define (parse-test-dependencies directory)
(filter (lambda (x) (not (member x %julia-stdlibs)))
(string-split (string-drop (string-drop-right pure 1) 7) #\,))))
+(define %juliahub-beautify-description-rules
+ `((h1 *preorder* . ,(lambda args #f))
+ (h2 *preorder* . ,(lambda args #f))
+ (h3 *preorder* . ,(lambda args #f))
+ (h4 *preorder* . ,(lambda args #f))
+ (hr *preorder* . ,(lambda args #f))
+ (span *preorder* . ,(lambda args #f))
+ (img *preorder* . ,(lambda args #f))
+ (pre *preorder* . ,(lambda args #f))
+ (div *preorder* . ,(lambda args #f))
+ (table *preorder* . ,(lambda args #f))
+ (imgalt *preorder* . ,(lambda args #f))
+ (@ *preorder* . ,(lambda args #f))
+ (*TOP* . ,(lambda args (cdr args)))
+ (p . ,(lambda args (cdr args)))
+ (em . ,(lambda args (cdr args)))
+ (strong . ,(lambda args (cdr args)))
+ (a . ,(lambda args
+ (match args
+ ((tag link ref)
+ (if ref ref #f))
+ (_ #f))))
+ (ul . ,(lambda args
+ `("@itemize" ,@(cdr args) "\n@end itemize")))
+ (ol . ,(lambda args
+ `("@enumerate" ,@(cdr args) "@end enumerate")))
+ (blockquote . ,(lambda args
+ `("@quotation" ,@(cdr args) "@end quotation")))
+ (li . ,(lambda args
+ `("\n@item" ,@(cdr args))))
+ (code . ,(lambda args
+ (string-append
+ "@code{"
+ (string-join (cdr args) " ")
+ "}")))
+ (*text* . ,(lambda (tag x) x))
+ (*default* . ,(lambda (tag . body)
+ (cons tag body)))))
+
+(define (juliahub-beautify-description description)
+ (string-join
+ (filter (lambda (x) (if (equal? x " ") #f x))
+ (flatten
+ (pre-post-order (html->sxml description)
+ %juliahub-beautify-description-rules)))
+ " "))
+
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
json->juliahub-package
@@ -211,8 +260,9 @@ (define (make-julia-sexp name source home-page synopsis description
,@(if (null? direct-dependencies)
'()
`((propagated-inputs
- (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- direct-dependencies)))))
+ (list
+ ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ direct-dependencies)))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
@@ -264,7 +314,9 @@ (define* (juliahub->guix-package package-name
source
(juliahub-package-homepage package)
(juliahub-package-description package)
- (beautify-description (juliahub-package-readme package))
+ ((compose beautify-description
+ juliahub-beautify-description)
+ (juliahub-package-readme package))
direct-dependencies
test-dependencies-names
licenses)
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 17/21] import: juliahub: Fix license management.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-17-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 15 +++++----------
1 file changed, 5 insertions(+), 10 deletions(-)

Toggle diff (51 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 338f042441..e4b26bea34 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -248,10 +248,10 @@ (define* (juliahub-fetch name #:key (version #f))
(and=> (json-fetch url) json->juliahub-package)))
(define (make-julia-sexp name source home-page synopsis description
- direct-dependencies test-dependencies-names licenses)
+ direct-dependencies test-dependencies-names license)
"Return the `package' s-expression for a Julia package with the given NAME,
VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
-TEST-DEPENDENCIES-NAMES and LICENSES."
+TEST-DEPENDENCIES-NAMES and LICENSE."
`(package
(name ,(julia-name->guix-name name))
(version ,version)
@@ -270,10 +270,8 @@ (define (make-julia-sexp name source home-page synopsis description
(synopsis ,synopsis)
(description ,description)
(home-page ,home-page)
- (license ,(match licenses
- (() #f)
- ((license) (license->symbol license))
- (_ `(list ,@(map license->symbol licenses)))))))
+ (license
+ ,(if license (spdx-string->license license) #f))))
(define* (juliahub->guix-package package-name
#:key version #:allow-other-keys)
@@ -301,9 +299,6 @@ (define* (juliahub->guix-package package-name
(dependencies-names
(map juliahub-dependency-name
direct-dependencies))
- (licenses
- (map spdx-string->license
- (list (juliahub-package-license package))))
(test-dependencies-names
(if subdir
(parse-test-dependencies
@@ -319,7 +314,7 @@ (define* (juliahub->guix-package package-name
(juliahub-package-readme package))
direct-dependencies
test-dependencies-names
- licenses)
+ (juliahub-package-license package))
(append dependencies-names test-dependencies-names))))
(values #f '()))))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 18/21] import: juliahub: Fix version management.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-18-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

Toggle diff (27 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index e4b26bea34..94d4ae8233 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -247,10 +247,10 @@ (define* (juliahub-fetch name #:key (version #f))
(string-append (juliahub-url name) "pkg.json"))))
(and=> (json-fetch url) json->juliahub-package)))
-(define (make-julia-sexp name source home-page synopsis description
+(define (make-julia-sexp name version source home-page synopsis description
direct-dependencies test-dependencies-names license)
"Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
+VERSION, SOURCE, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
TEST-DEPENDENCIES-NAMES and LICENSE."
`(package
(name ,(julia-name->guix-name name))
@@ -306,6 +306,7 @@ (define* (juliahub->guix-package package-name
(parse-test-dependencies directory))))
(values (make-julia-sexp
package-name
+ (juliahub-package-version package)
source
(juliahub-package-homepage package)
(juliahub-package-description package)
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 19/21] import: juliahub: Fix undefined homepages.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-19-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)

Toggle diff (25 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 94d4ae8233..06574db724 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -303,12 +303,16 @@ (define* (juliahub->guix-package package-name
(if subdir
(parse-test-dependencies
(string-append subdir "/" directory))
- (parse-test-dependencies directory))))
+ (parse-test-dependencies directory)))
+ (homepage (juliahub-package-homepage package)))
(values (make-julia-sexp
package-name
(juliahub-package-version package)
source
- (juliahub-package-homepage package)
+ (match homepage
+ ("" (juliahub-package-url package))
+ ((? string?) homepage)
+ (_ (juliahub-package-url package)))
(juliahub-package-description package)
((compose beautify-description
juliahub-beautify-description)
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 20/21] import: utils: Rule out texinfo common syntax from @ escape.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-20-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 5 +----
guix/import/utils.scm | 8 +++++++-
2 files changed, 8 insertions(+), 5 deletions(-)

Toggle diff (37 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 06574db724..0d3f89ad61 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -197,10 +197,7 @@ (define %juliahub-beautify-description-rules
(li . ,(lambda args
`("\n@item" ,@(cdr args))))
(code . ,(lambda args
- (string-append
- "@code{"
- (string-join (cdr args) " ")
- "}")))
+ `("@code{" ,@(cdr args) "}")))
(*text* . ,(lambda (tag x) x))
(*default* . ,(lambda (tag . body)
(cons tag body)))))
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 171dca54e8..5ed1dfd815 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -376,7 +376,13 @@ (define* (beautify-description description #:optional (length 80))
(cut string-trim-both <> #\')
;; Escape single @ to prevent it from being understood as
;; invalid Texinfo syntax.
- (cut regexp-substitute/global #f "@" <> 'pre "@@" 'post)))))
+ (lambda (word)
+ (if
+ (member word '("@itemize" "@item" "@end" "@quotation"
+ "@enumerate" "@code" "@code{"))
+ word
+ ((cut regexp-substitute/global
+ #f "@" <> 'pre "@@" 'post) word)))))))
(words
(string-tokenize (string-trim-both description)
(char-set-complement
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 13:51 +0100
[PATCH 21/21] import: juliahub: output package names as symbols.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-21-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)

Toggle diff (24 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 0d3f89ad61..3e5735b950 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -258,12 +258,15 @@ (define (make-julia-sexp name version source home-page synopsis description
'()
`((propagated-inputs
(list
- ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ ,@(map (compose string->symbol
+ julia-name->guix-name
+ juliahub-dependency-name)
direct-dependencies)))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
- (list ,@(map julia-name->guix-name test-dependencies-names)))))
+ (list ,@(map (compose string->symbol julia-name->guix-name)
+ test-dependencies-names)))))
(synopsis ,synopsis)
(description ,description)
(home-page ,home-page)
--
2.39.2
S
S
Simon Tournier wrote on 7 Apr 18:14 +0200
Re: [bug#62202] [PATCH 0/21] Juliahub import script.
87v8i7k6uv.fsf@gmail.com
Hi Nicolas,

Sorry for the delay. (I was in holidays \o/ :-))

I am not able to apply the series. Could you rebase it or provide here
by email the commit against which the series apply. Thanks in advance.


On mer., 15 mars 2023 at 13:47, Nicolas Graves via Guix-patches via <guix-patches@gnu.org> wrote:

Toggle quote (3 lines)
> Took me quite more time than I would've liked, but I have a usable
> juliahub scheme import script!

Really cool! Thank you!


Toggle quote (12 lines)
> It seems there's still one edge case that isn't covered and revolves
> around when Julia packagers don't properly tag their git repos (I've
> only seen the case with SnoopPrecompile). There's the possibility to
> rely on tree commit hashes from the General repository (since this is a
> valid way to identify/store a git repo), but that needs some major
> changes in the way latest-repository-commit works. Otherwise, it needs
> to be done by hand. It might also not work for subpackages in
> directories that are up-to-date on juliahub but not yet on github, I
> haven't met this case yet.
>
> I'm sending a patch series in the coming minutes.

Well, I have not read all series. :-)

Cheers,
simon
L
L
Ludovic Courtès wrote on 9 Apr 00:07 +0200
Re: bug#62202: [PATCH 0/21] Juliahub import script.
(name . Nicolas Graves)(address . ngraves@ngraves.fr)(address . 62202@debbugs.gnu.org)
87ile69geh.fsf@gnu.org
Hi!

Nicolas Graves <ngraves@ngraves.fr> skribis:

Toggle quote (18 lines)
> Took me quite more time than I would've liked, but I have a usable
> juliahub scheme import script!
>
> It seems there's still one edge case that isn't covered and revolves
> around when Julia packagers don't properly tag their git repos (I've
> only seen the case with SnoopPrecompile). There's the possibility to
> rely on tree commit hashes from the General repository (since this is a
> valid way to identify/store a git repo), but that needs some major
> changes in the way latest-repository-commit works. Otherwise, it needs
> to be done by hand. It might also not work for subpackages in
> directories that are up-to-date on juliahub but not yet on github, I
> haven't met this case yet.
>
> I'm sending a patch series in the coming minutes.
>
> It's detailed since I haven't swauased all commits, for readability, but
> I can squash it further if necessary.

I’ll let Simon comment on the actual code since I’m not a Julia person.
:-)

Some more general comments:

• Please make sure to document it in ‘doc/guix.texi’ under “Invoking
guix import”, following the same template as the others there.

• Please write ‘tests/juliahub.scm’. I recommend the same strategy as
‘tests/cpan.scm’, which is to mock the upstream HTTP server.

• Prefer (srfi srfi-41) over (ice-9 streams) (see rationale at

• Prefer (srfi srfi-71) over (srfi srfi-11) for multiple-value
bindings.

Thanks,
Ludo’.
?