[PATCH 0/1] npm binary importer

  • Open
  • quality assurance status badge
Details
5 participants
  • jlicht
  • Jonathan Brielmaier
  • Ludovic Courtès
  • Nicolas Graves
  • Pablo Zamora
Owner
unassigned
Submitted by
jlicht
Severity
normal
J
J
jlicht wrote on 22 Mar 2023 12:25
(address . guix-patches@gnu.org)
cover.1679484068.git.jlicht@fsfe.org
From: Jelle Licht <jlicht@fsfe.org>

Folks,

Here a revised patch to add the npm binary importer. To give some context,
'binary' here refers to the fact that this downloads archives straight from
the npm registry at https://registry.npmjs.org. Some of these downloaded
archives may not contain the original sources, so unless properly vetted the
output of this importer does not allow one to easily generate package
expressions for inclusion in guix's collection of packages.

It should work as-is for most simple NPM packages. As noted in an inline
comment somewhere, the way both npm and our very own node-build-system treats
peer dependencies may require some manual intervention from time to time to
hook in the right dependency at the right spot. The upside here is that when
this happens, it's either trivial for a human to spot and fix, or wholly
incompatible with our current approach so not fixable without writing a custom
build system to deal with dependency cycles :-).

Please test and review, let us get this merged as it might be useful to many
people for building out their personal channels and/or package expressions.

Special thanks to Timothy Sample and Lars-Dominik Braun for involvement in
realising this, way too long ago.


Jelle Licht (1):
import: Add binary npm importer.

Makefile.am | 2 +
guix/import/npm-binary.scm | 269 +++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/npm-binary.scm | 113 ++++++++++++
4 files changed, 385 insertions(+), 1 deletion(-)
create mode 100644 guix/import/npm-binary.scm
create mode 100644 guix/scripts/import/npm-binary.scm

--
2.39.2
J
J
jlicht wrote on 22 Mar 2023 12:27
[PATCH] import: Add binary npm importer.
(address . 62375@debbugs.gnu.org)
a41fe3ee10c48d36d3c84cf9f22ace8ca5bcd7c3.1679484420.git.jlicht@fsfe.org
From: Jelle Licht <jlicht@fsfe.org>

* guix/scripts/import.scm: (importers): Add "npm-binary".
* guix/import/npm-binary.scm: New file.
* guix/scripts/import/npm-binary.scm: New file.
* Makefile.am: Add them.

Co-authored-by: Timothy Sample <samplet@ngyro.com>
Co-authored-by: Lars-Dominik Braun <lars@6xq.net>

---

Makefile.am | 2 +
guix/import/npm-binary.scm | 269 +++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/npm-binary.scm | 113 ++++++++++++
4 files changed, 385 insertions(+), 1 deletion(-)
create mode 100644 guix/import/npm-binary.scm
create mode 100644 guix/scripts/import/npm-binary.scm

Toggle diff (429 lines)
diff --git a/Makefile.am b/Makefile.am
index 23b939b674..52def58ae2 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -288,6 +288,7 @@ MODULES = \
guix/import/kde.scm \
guix/import/launchpad.scm \
guix/import/minetest.scm \
+ guix/import/npm-binary.scm \
guix/import/opam.scm \
guix/import/print.scm \
guix/import/pypi.scm \
@@ -339,6 +340,7 @@ MODULES = \
guix/scripts/import/hexpm.scm \
guix/scripts/import/json.scm \
guix/scripts/import/minetest.scm \
+ guix/scripts/import/npm-binary.scm \
guix/scripts/import/opam.scm \
guix/scripts/import/pypi.scm \
guix/scripts/import/stackage.scm \
diff --git a/guix/import/npm-binary.scm b/guix/import/npm-binary.scm
new file mode 100644
index 0000000000..f9b54263e4
--- /dev/null
+++ b/guix/import/npm-binary.scm
@@ -0,0 +1,269 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2019, 2020 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2020, 2023 Jelle Licht <jlicht@fsfe.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import npm-binary)
+ #:use-module (guix import json)
+ #:use-module (guix import utils)
+ #:use-module (guix memoization)
+ #:use-module ((gnu services configuration) #:select (alist?))
+ #:use-module (guix utils)
+ #:use-module (gnu packages)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 receive)
+ #:use-module (json)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-41)
+ #:use-module (web client)
+ #:use-module (web response)
+ #:use-module (web uri)
+ #:export (npm-binary-recursive-import
+ npm-binary->guix-package
+ package-json->guix-package
+ make-versioned-package
+ name+version->symbol))
+
+;; Autoload Guile-Semver so we only have a soft dependency.
+(module-autoload! (current-module)
+ '(semver) '(string->semver semver? semver->string semver=? semver>?))
+(module-autoload! (current-module)
+ '(semver ranges) '(*semver-range-any* string->semver-range semver-range-contains?))
+
+;; Dist-tags
+(define-json-mapping <dist-tags> make-dist-tags dist-tags?
+ json->dist-tags
+ (latest dist-tags-latest "latest" string->semver))
+
+(define-record-type <versioned-package>
+ (make-versioned-package name version)
+ versioned-package?
+ (name versioned-package-name) ;string
+ (version versioned-package-version)) ;string
+
+(define (dependencies->versioned-packages entries)
+ (match entries
+ (((names . versions) ...)
+ (map make-versioned-package names versions))
+ (_ '())))
+
+(define (extract-license license-string)
+ (if (unspecified? license-string)
+ 'unspecified!
+ (spdx-string->license license-string)))
+
+(define-json-mapping <dist> make-dist dist?
+ json->dist
+ (tarball dist-tarball))
+
+(define (empty-or-string s)
+ (if (string? s) s ""))
+
+(define-json-mapping <package-revision> make-package-revision package-revision?
+ json->package-revision
+ (name package-revision-name)
+ (version package-revision-version "version" string->semver) ;semver
+ (home-page package-revision-home-page "homepage") ;string
+ (dependencies package-revision-dependencies "dependencies" ;list of versioned-package
+ dependencies->versioned-packages)
+ (dev-dependencies package-revision-dev-dependencies ;list of versioned-package
+ "devDependencies" dependencies->versioned-packages)
+ (peer-dependencies package-revision-peer-dependencies ;list of versioned-package
+ "peerDependencies" dependencies->versioned-packages)
+ (license package-revision-license "license" ;license | #f
+ (match-lambda
+ ((? unspecified?) #f)
+ ((? string? str) (spdx-string->license str))
+ ((? alist? alist)
+ (match (assoc "type" alist)
+ ((_ . (? string? type))
+ (spdx-string->license type))
+ (_ #f)))))
+ (description package-revision-description ; string
+ "description" empty-or-string)
+ (dist package-revision-dist "dist" json->dist)) ;dist
+
+(define (versions->package-revisions versions)
+ (match versions
+ (((version . package-spec) ...)
+ (map json->package-revision package-spec))
+ (_ '())))
+
+(define (versions->package-versions versions)
+ (match versions
+ (((version . package-spec) ...)
+ (map string->semver versions))
+ (_ '())))
+
+(define-json-mapping <meta-package> make-meta-package meta-package?
+ json->meta-package
+ (name meta-package-name) ;string
+ (description meta-package-description) ;string
+ (dist-tags meta-package-dist-tags "dist-tags" json->dist-tags) ;dist-tags
+ (revisions meta-package-revisions "versions" versions->package-revisions))
+
+;; TODO: Support other registries
+(define *registry* "https://registry.npmjs.org")
+(define *default-page* "https://www.npmjs.com/package")
+
+(define (lookup-meta-package name)
+ (let ((json (json-fetch (string-append *registry* "/" (uri-encode name)))))
+ (and=> json json->meta-package)))
+
+(define lookup-meta-package* (memoize lookup-meta-package))
+
+(define (http-error-code arglist)
+ (match arglist
+ (('http-error _ _ _ (code)) code)
+ (_ #f)))
+
+(define (meta-package-versions meta)
+ (map package-revision-version
+ (meta-package-revisions meta)))
+
+(define (meta-package-latest meta)
+ (and=> (meta-package-dist-tags meta) dist-tags-latest))
+
+(define* (meta-package-package meta #:optional
+ (version (meta-package-latest meta)))
+ (match version
+ ((? semver?) (find (lambda (revision)
+ (semver=? version (package-revision-version revision)))
+ (meta-package-revisions meta)))
+ ((? string?) (meta-package-package meta (string->semver version)))
+ (_ #f)))
+
+(define* (semver-latest svs #:optional (svr *semver-range-any*))
+ (find (cut semver-range-contains? svr <>)
+ (sort svs semver>?)))
+
+(define* (resolve-package name #:optional (svr *semver-range-any*))
+ (let ((meta (lookup-meta-package* name)))
+ (and meta
+ (let* ((version (semver-latest (or (meta-package-versions meta) '()) svr))
+ (pkg (meta-package-package meta version)))
+ pkg))))
+
+
+;;;
+;;; Converting packages
+;;;
+
+(define (hash-url url)
+ "Downloads the resource at URL and computes the base32 hash for it."
+ (call-with-temporary-output-file
+ (lambda (temp port)
+ (begin ((@ (guix import utils) url-fetch) url temp)
+ (guix-hash-url temp)))))
+
+(define (npm-name->name npm-name)
+ "Return a Guix package name for the npm package with name NPM-NAME."
+ (define (clean name)
+ (string-map (lambda (chr) (if (char=? chr #\/) #\- chr))
+ (string-filter (negate (cut char=? <> #\@)) name)))
+ (guix-name "node-" (clean npm-name)))
+
+(define (name+version->symbol name version)
+ (string->symbol (string-append name "-" version)))
+
+(define (package-revision->symbol package)
+ (let* ((npm-name (package-revision-name package))
+ (version (semver->string (package-revision-version package)))
+ (name (npm-name->name npm-name)))
+ (name+version->symbol name version)))
+
+(define (package-revision->input package)
+ "Return the `inputs' entry for PACKAGE."
+ (let* ((npm-name (package-revision-name package))
+ (name (npm-name->name npm-name)))
+ `(,name
+ (,'unquote ,(package-revision->symbol package)))))
+
+(define (npm-package->package-sexp npm-package)
+ "Return the `package' s-expression for an NPM-PACKAGE."
+ (define (new-or-existing-inputs resolved-deps)
+ (map package-revision->input resolved-deps))
+
+ (match npm-package
+ (($ <package-revision> name version home-page dependencies dev-dependencies peer-dependencies license description dist)
+ (let* ((name (npm-name->name name))
+ (url (dist-tarball dist))
+ (home-page (if (string? home-page)
+ home-page
+ (string-append *default-page* "/" (uri-encode name))))
+ (synopsis description)
+ (resolved-deps (map (match-lambda (($ <versioned-package> name version)
+ (resolve-package name (string->semver-range version)))) (append dependencies peer-dependencies)))
+ (peer-names (map versioned-package-name peer-dependencies))
+ ;; lset-difference for treating peer-dependencies as dependencies, which leads to dependency cycles.
+ ;; lset-union for treating them as (ignored) dev-dependencies, which leads to broken packages.
+ (dev-names (lset-union string= (map versioned-package-name dev-dependencies) peer-names))
+ (extra-phases (match dev-names
+ (() '())
+ ((dev-names ...)
+ `((add-after 'patch-dependencies 'delete-dev-dependencies
+ (lambda _
+ (delete-dependencies '(,@(reverse dev-names))))))))))
+ (values
+ `(package
+ (name ,name)
+ (version ,(semver->string (package-revision-version npm-package)))
+ (source (origin
+ (method url-fetch)
+ (uri ,url)
+ (sha256 (base32 ,(hash-url url)))))
+ (build-system node-build-system)
+ (arguments
+ '(#:tests? #f
+ #:phases (modify-phases %standard-phases
+ (delete 'build)
+ ,@extra-phases)))
+ ,@(match dependencies
+ (() '())
+ ((dependencies ...)
+ `((inputs
+ (,'quasiquote ,(map package-revision->input resolved-deps))))))
+ (home-page ,home-page)
+ (synopsis ,synopsis)
+ (description ,description)
+ (license ,license))
+ (map (match-lambda (($ <package-revision> name version)
+ (list name (semver->string version))))
+ resolved-deps))))
+ (_ #f)))
+
+
+;;;
+;;; Interface
+;;;
+
+(define npm-binary->guix-package
+ (lambda* (name #:key (version *semver-range-any*) #:allow-other-keys)
+ (let* ((svr (match version
+ ((? string?) (string->semver-range version))
+ (_ version)))
+ (pkg (resolve-package name svr)))
+ (and=> pkg npm-package->package-sexp))))
+
+(define* (npm-binary-recursive-import package-name #:key version)
+ (recursive-import package-name
+ #:repo->guix-package (memoize npm-binary->guix-package)
+ #:version version
+ #:guix-name npm-name->name))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index f84a964a53..dccf6488b2 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -47,7 +47,7 @@ (define %standard-import-options '())
(define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
"gem" "go" "cran" "crate" "texlive" "json" "opam"
- "minetest" "elm" "hexpm"))
+ "minetest" "elm" "hexpm" "npm-binary"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/npm-binary.scm b/guix/scripts/import/npm-binary.scm
new file mode 100644
index 0000000000..825c43bbc3
--- /dev/null
+++ b/guix/scripts/import/npm-binary.scm
@@ -0,0 +1,113 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2014 David Thompson <davet@gnu.org>
+;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2019 Timothy Sample <samplet@ngyro.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import npm-binary)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import npm-binary)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (srfi srfi-41)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:export (guix-import-npm-binary))
+
+
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import npm-binary PACKAGE-NAME [VERSION]
+Import and convert the NPM package PACKAGE-NAME using the
+`npm-build-system' (but without building the package from source)."))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -r, --recursive import packages recursively"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import npm-binary")))
+ (option '(#\r "recursive") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'recursive #t result)))
+ %standard-import-options))
+
+
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-npm-binary . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (args-fold* args %options
+ (lambda (opt name arg result)
+ (leave (G_ "~A: unrecognized option~%") name))
+ (lambda (arg result)
+ (alist-cons 'argument arg result))
+ %default-options))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (let loop ((args args))
+ (match args
+ ((package-name version)
+ (if (assoc-ref opts 'recursive)
+ ;; Recursive import
+ (map (match-lambda
+ ((and ('package ('name name) ('version version) . rest) pkg)
+ `(define-public ,(name+version->symbol name version)
+ ,pkg))
+ (_ #f))
+ (npm-binary-recursive-import package-name #:version version))
+ ;; Single import
+ (let ((sexp (npm-binary->guix-package package-name #:version version)))
+ (unless sexp
+ (leave (G_ "failed to download meta-data for package '~a@~a'~%")
+ package-name version))
+ sexp)))
+ ((package-name)
+ (loop (list package-name "*")))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%")))))))
--
2.39.2
L
L
Ludovic Courtès wrote on 28 Mar 2023 17:49
(address . jlicht@fsfe.org)
87tty4g9kj.fsf@gnu.org
Hello Jelle!

jlicht@fsfe.org skribis:

Toggle quote (10 lines)
> From: Jelle Licht <jlicht@fsfe.org>
>
> * guix/scripts/import.scm: (importers): Add "npm-binary".
> * guix/import/npm-binary.scm: New file.
> * guix/scripts/import/npm-binary.scm: New file.
> * Makefile.am: Add them.
>
> Co-authored-by: Timothy Sample <samplet@ngyro.com>
> Co-authored-by: Lars-Dominik Braun <lars@6xq.net>

Woohoo! I think it’ll be useful to many, even if we know it’s far from
“ideal” by our standards.

We’ll need doc under “Invoking guix import” and tests in
‘tests/npm-binary.scm’, similar to what’s done for the other importers.
The doc should clarify why it’s called that way and what the limitations
are.

For tests, I’d recommend mocking the npmjs.org HTTP servers using
‘with-http-server’ as in ‘tests/cpan.scm’.

Also, please add docstrings to top-level procedures.

Toggle quote (4 lines)
> +;; TODO: Support other registries
> +(define *registry* "https://registry.npmjs.org")
> +(define *default-page* "https://www.npmjs.com/package")

The convention currently is more like ‘%registry’.

Toggle quote (5 lines)
> +(define (http-error-code arglist)
> + (match arglist
> + (('http-error _ _ _ (code)) code)
> + (_ #f)))

Unused. :-)

Toggle quote (7 lines)
> +(define (hash-url url)
> + "Downloads the resource at URL and computes the base32 hash for it."
> + (call-with-temporary-output-file
> + (lambda (temp port)
> + (begin ((@ (guix import utils) url-fetch) url temp)
> + (guix-hash-url temp)))))

Maybe something more like: (port-sha256 (http-fetch …)) ?

Toggle quote (8 lines)
> +(define (npm-package->package-sexp npm-package)
> + "Return the `package' s-expression for an NPM-PACKAGE."
> + (define (new-or-existing-inputs resolved-deps)
> + (map package-revision->input resolved-deps))
> +
> + (match npm-package
> + (($ <package-revision> name version home-page dependencies dev-dependencies peer-dependencies license description dist)

Please use ‘match-record’ instead and keep lines below 80 chars. :-)

Toggle quote (9 lines)
> + (let* ((name (npm-name->name name))
> + (url (dist-tarball dist))
> + (home-page (if (string? home-page)
> + home-page
> + (string-append *default-page* "/" (uri-encode name))))
> + (synopsis description)
> + (resolved-deps (map (match-lambda (($ <versioned-package> name version)
> + (resolve-package name (string->semver-range version)))) (append dependencies peer-dependencies)))

Likewise.

That’s it! Could you send an updated patch?

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 6 Apr 2023 22:18
control message for bug #62375
(address . control@debbugs.gnu.org)
87lej4hii4.fsf@gnu.org
tags 62375 + moreinfo
quit
J
J
Jelle Licht wrote on 8 Apr 2023 20:29
Re: [PATCH] import: Add binary npm importer.
(name . Ludovic Courtès)(address . ludo@gnu.org)
871qkurzwh.fsf@fsfe.org
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (59 lines)
> Hello Jelle!
>
> jlicht@fsfe.org skribis:
>
>> From: Jelle Licht <jlicht@fsfe.org>
>>
>> * guix/scripts/import.scm: (importers): Add "npm-binary".
>> * guix/import/npm-binary.scm: New file.
>> * guix/scripts/import/npm-binary.scm: New file.
>> * Makefile.am: Add them.
>>
>> Co-authored-by: Timothy Sample <samplet@ngyro.com>
>> Co-authored-by: Lars-Dominik Braun <lars@6xq.net>
>
> Woohoo! I think it’ll be useful to many, even if we know it’s far from
> “ideal” by our standards.
>
> We’ll need doc under “Invoking guix import” and tests in
> ‘tests/npm-binary.scm’, similar to what’s done for the other importers.
> The doc should clarify why it’s called that way and what the limitations
> are.
>
> For tests, I’d recommend mocking the npmjs.org HTTP servers using
> ‘with-http-server’ as in ‘tests/cpan.scm’.
>
> Also, please add docstrings to top-level procedures.
>
>> +;; TODO: Support other registries
>> +(define *registry* "https://registry.npmjs.org")
>> +(define *default-page* "https://www.npmjs.com/package")
>
> The convention currently is more like ‘%registry’.
>
>> +(define (http-error-code arglist)
>> + (match arglist
>> + (('http-error _ _ _ (code)) code)
>> + (_ #f)))
>
> Unused. :-)
>
>> +(define (hash-url url)
>> + "Downloads the resource at URL and computes the base32 hash for it."
>> + (call-with-temporary-output-file
>> + (lambda (temp port)
>> + (begin ((@ (guix import utils) url-fetch) url temp)
>> + (guix-hash-url temp)))))
>
> Maybe something more like: (port-sha256 (http-fetch …)) ?
>
>> +(define (npm-package->package-sexp npm-package)
>> + "Return the `package' s-expression for an NPM-PACKAGE."
>> + (define (new-or-existing-inputs resolved-deps)
>> + (map package-revision->input resolved-deps))
>> +
>> + (match npm-package
>> + (($ <package-revision> name version home-page dependencies dev-dependencies peer-dependencies license description dist)
>
> Please use ‘match-record’ instead and keep lines below 80 chars. :-)

The records generated by `define-json-mapping' through
`define-record-type' seem to not work with `match-record'.

I could define our very own `define-json-mapping*' that works /w
`define-record-type*' instead (and through that, `match-record'), but I
thought I'd ask if you had something else in mind first :).

Toggle quote (13 lines)
>> + (let* ((name (npm-name->name name))
>> + (url (dist-tarball dist))
>> + (home-page (if (string? home-page)
>> + home-page
>> + (string-append *default-page* "/" (uri-encode name))))
>> + (synopsis description)
>> + (resolved-deps (map (match-lambda (($ <versioned-package> name version)
>> + (resolve-package name (string->semver-range version)))) (append dependencies peer-dependencies)))
>
> Likewise.
>
> That’s it! Could you send an updated patch?

Will do, once I got some proper test cases set up.

Thanks again for the review!
- Jelle
L
L
Ludovic Courtès wrote on 17 Apr 2023 23:14
Re: bug#62375: [PATCH 0/1] npm binary importer
(name . Jelle Licht)(address . jlicht@fsfe.org)
874jpecisz.fsf_-_@gnu.org
Hi,

Jelle Licht <jlicht@fsfe.org> skribis:

Toggle quote (2 lines)
> Ludovic Courtès <ludo@gnu.org> writes:

[...]

Toggle quote (8 lines)
>>> + (match npm-package
>>> + (($ <package-revision> name version home-page dependencies dev-dependencies peer-dependencies license description dist)
>>
>> Please use ‘match-record’ instead and keep lines below 80 chars. :-)
>
> The records generated by `define-json-mapping' through
> `define-record-type' seem to not work with `match-record'.

Oh right, my bad.

The other option is to call record accessors; it would avoid the risk
associated with index-based matches but may be more verbose. Your call!

Ludo’.
L
L
Ludovic Courtès wrote on 18 Jun 2023 23:03
(name . Jelle Licht)(address . jlicht@fsfe.org)
87pm5spjo8.fsf_-_@gnu.org
Hey Jelle,

How far are we from merging? :-)

TIA,
Ludo’.

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (22 lines)
> Hi,
>
> Jelle Licht <jlicht@fsfe.org> skribis:
>
>> Ludovic Courtès <ludo@gnu.org> writes:
>
> [...]
>
>>>> + (match npm-package
>>>> + (($ <package-revision> name version home-page dependencies dev-dependencies peer-dependencies license description dist)
>>>
>>> Please use ‘match-record’ instead and keep lines below 80 chars. :-)
>>
>> The records generated by `define-json-mapping' through
>> `define-record-type' seem to not work with `match-record'.
>
> Oh right, my bad.
>
> The other option is to call record accessors; it would avoid the risk
> associated with index-based matches but may be more verbose. Your call!
>
> Ludo’.
J
J
Jelle Licht wrote on 22 Jun 2023 11:39
(name . Ludovic Courtès)(address . ludo@gnu.org)
87352j7s54.fsf@fsfe.org
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (4 lines)
> Hey Jelle,
>
> How far are we from merging? :-)

Hey Ludo,

Thanks for checking in! I haven't had a whole lot of time recently due
to a bunch of (positive) personal things, but I'll reach out soon~ish
with the next iteration.

Cheers,
- Jelle
N
N
Nicolas Graves wrote on 8 Feb 01:59 +0100
Re: [bug#62375] [PATCH 0/1] npm binary importer
87eddn4vq3.fsf@ngraves.fr
On 2023-03-22 12:25, jlicht@fsfe.org wrote:

Toggle quote (37 lines)
> From: Jelle Licht <jlicht@fsfe.org>
>
> Folks,
>
> Here a revised patch to add the npm binary importer. To give some context,
> 'binary' here refers to the fact that this downloads archives straight from
> the npm registry at https://registry.npmjs.org. Some of these downloaded
> archives may not contain the original sources, so unless properly vetted the
> output of this importer does not allow one to easily generate package
> expressions for inclusion in guix's collection of packages.
>
> It should work as-is for most simple NPM packages. As noted in an inline
> comment somewhere, the way both npm and our very own node-build-system treats
> peer dependencies may require some manual intervention from time to time to
> hook in the right dependency at the right spot. The upside here is that when
> this happens, it's either trivial for a human to spot and fix, or wholly
> incompatible with our current approach so not fixable without writing a custom
> build system to deal with dependency cycles :-).
>
> Please test and review, let us get this merged as it might be useful to many
> people for building out their personal channels and/or package expressions.
>
> Special thanks to Timothy Sample and Lars-Dominik Braun for involvement in
> realising this, way too long ago.
>
>
> Jelle Licht (1):
> import: Add binary npm importer.
>
> Makefile.am | 2 +
> guix/import/npm-binary.scm | 269 +++++++++++++++++++++++++++++
> guix/scripts/import.scm | 2 +-
> guix/scripts/import/npm-binary.scm | 113 ++++++++++++
> 4 files changed, 385 insertions(+), 1 deletion(-)
> create mode 100644 guix/import/npm-binary.scm
> create mode 100644 guix/scripts/import/npm-binary.scm

Hey Jelle,

Just to understand, is there a reason why you didn't went further with
the sucrase approach?

--
Best regards,
Nicolas Graves
P
P
Pablo Zamora wrote on 24 Mar 15:54 +0100
Continue the npm-binary importer
(name . 62375@debbugs.gnu.org)(address . 62375@debbugs.gnu.org)
AS8P189MB20242416E3975E7D4AC12F3BFD372@AS8P189MB2024.EURP189.PROD.OUTLOOK.COM
Dear Guix,

I am stuck while trying to define the 'bitwarden' package definition on my personal channel here https://github.com/Pablo12345678901/guix-custom-channel/blob/master/dev/my-bitwarden.scm. I am facing issues due to the npm dependencies.

One of the potential solution would be this importer. I would like to finish writing it and perform the tests.

On which branch should I send git pull request to continue what has been done so far ? This would be my first commit with guix.

Pablo Zamora
Attachment: file
J
J
jlicht wrote on 31 Mar 21:46 +0200
[PATCH v2] import: Add binary npm importer.
(address . 62375@debbugs.gnu.org)
b143ed0e0bea73cbfde69c24758adf2e4e93839f.1711914054.git.jlicht@fsfe.org
From: Jelle Licht <jlicht@fsfe.org>

* guix/scripts/import.scm: (importers): Add "npm-binary".
* guix/import/npm-binary.scm: New file.
* guix/scripts/import/npm-binary.scm: New file.
* Makefile.am: Add them.

Co-authored-by: Timothy Sample <samplet@ngyro.com>
Co-authored-by: Lars-Dominik Braun <lars@6xq.net>

Change-Id: I98a45068cf5b9c42790664cc743feaa7ac76f807
---

Changes in v2:
- Change *SOME-VAR* to %SOME-VAR
- Removed unused http-error-code
- Rebase on master
- Refactor hash-url to use port-sha256 helper
- use explicit record accessors instead of order-sensitive destructuring
- address line-width styling issues
- added basic documentation
- added some basic tests (using simple mocks instead of with-http-server)
- simplify import script entrypoint

Makefile.am | 3 +
doc/guix.texi | 33 ++++
guix/import/npm-binary.scm | 277 +++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/npm-binary.scm | 108 +++++++++++
tests/npm-binary.scm | 146 +++++++++++++++
6 files changed, 568 insertions(+), 1 deletion(-)
create mode 100644 guix/import/npm-binary.scm
create mode 100644 guix/scripts/import/npm-binary.scm
create mode 100755 tests/npm-binary.scm

Toggle diff (500 lines)
diff --git a/Makefile.am b/Makefile.am
index 1c5688ac13..459c47a954 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -305,6 +305,7 @@ MODULES = \
guix/import/kde.scm \
guix/import/launchpad.scm \
guix/import/minetest.scm \
+ guix/import/npm-binary.scm \
guix/import/opam.scm \
guix/import/print.scm \
guix/import/pypi.scm \
@@ -359,6 +360,7 @@ MODULES = \
guix/scripts/import/hexpm.scm \
guix/scripts/import/json.scm \
guix/scripts/import/minetest.scm \
+ guix/scripts/import/npm-binary.scm \
guix/scripts/import/opam.scm \
guix/scripts/import/pypi.scm \
guix/scripts/import/stackage.scm \
@@ -557,6 +559,7 @@ SCM_TESTS = \
tests/modules.scm \
tests/monads.scm \
tests/nar.scm \
+ tests/npm-binary.scm \
tests/networking.scm \
tests/opam.scm \
tests/openpgp.scm \
diff --git a/doc/guix.texi b/doc/guix.texi
index 69a904473c..566af6e849 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -14641,6 +14641,39 @@ Invoking guix import
in Guix.
@end table
+@item npm-binary
+@cindex npm
+@cindex Node.js
+Import metadata from the @uref{https://registry.npmjs.org, npm
+Registry}, as in this example:
+
+@example
+guix import npm-binary buffer-crc32
+@end example
+
+The npm-binary importer also allows you to specify a version string:
+
+@example
+guix import npm-binary buffer-crc32 1.0.0
+@end example
+
+@quotation Note
+Generated package expressions skip the build step of the
+@code{node-build-system}. As such, generated package expressions often
+refer to transpiled or generated files, instead of being built from
+source.
+@end quotation
+
+Additional options include:
+
+@table @code
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
+@end table
+
@item opam
@cindex OPAM
@cindex OCaml
diff --git a/guix/import/npm-binary.scm b/guix/import/npm-binary.scm
new file mode 100644
index 0000000000..57c985baf2
--- /dev/null
+++ b/guix/import/npm-binary.scm
@@ -0,0 +1,277 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2019, 2020 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2020, 2023, 2024 Jelle Licht <jlicht@fsfe.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import npm-binary)
+ #:use-module ((gnu services configuration) #:select (alist?))
+ #:use-module (gcrypt hash)
+ #:use-module (gnu packages)
+ #:use-module (guix base32)
+ #:use-module (guix http-client)
+ #:use-module (guix import json)
+ #:use-module (guix import utils)
+ #:use-module (guix memoization)
+ #:use-module (guix utils)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 receive)
+ #:use-module (ice-9 regex)
+ #:use-module (json)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-41)
+ #:use-module (srfi srfi-9)
+ #:use-module (web client)
+ #:use-module (web response)
+ #:use-module (web uri)
+ #:export (npm-binary-recursive-import
+ npm-binary->guix-package
+ make-versioned-package
+ name+version->symbol))
+
+;; Autoload Guile-Semver so we only have a soft dependency.
+(module-autoload! (current-module)
+ '(semver)
+ '(string->semver semver? semver->string semver=? semver>?))
+(module-autoload! (current-module)
+ '(semver ranges)
+ '(*semver-range-any* string->semver-range semver-range-contains?))
+
+;; Dist-tags
+(define-json-mapping <dist-tags> make-dist-tags dist-tags?
+ json->dist-tags
+ (latest dist-tags-latest "latest" string->semver))
+
+(define-record-type <versioned-package>
+ (make-versioned-package name version)
+ versioned-package?
+ (name versioned-package-name) ;string
+ (version versioned-package-version)) ;string
+
+(define (dependencies->versioned-packages entries)
+ (match entries
+ (((names . versions) ...)
+ (map make-versioned-package names versions))
+ (_ '())))
+
+(define (extract-license license-string)
+ (if (unspecified? license-string)
+ 'unspecified!
+ (spdx-string->license license-string)))
+
+(define-json-mapping <dist> make-dist dist?
+ json->dist
+ (tarball dist-tarball))
+
+(define (empty-or-string s)
+ (if (string? s) s ""))
+
+(define-json-mapping <package-revision> make-package-revision package-revision?
+ json->package-revision
+ (name package-revision-name)
+ (version package-revision-version "version" ;semver
+ string->semver)
+ (home-page package-revision-home-page "homepage") ;string
+ (dependencies package-revision-dependencies ;list of versioned-package
+ "dependencies"
+ dependencies->versioned-packages)
+ (dev-dependencies package-revision-dev-dependencies ;list of versioned-package
+ "devDependencies" dependencies->versioned-packages)
+ (peer-dependencies package-revision-peer-dependencies ;list of versioned-package
+ "peerDependencies" dependencies->versioned-packages)
+ (license package-revision-license "license" ;license | #f
+ (match-lambda
+ ((? unspecified?) #f)
+ ((? string? str) (spdx-string->license str))
+ ((? alist? alist)
+ (match (assoc "type" alist)
+ ((_ . (? string? type))
+ (spdx-string->license type))
+ (_ #f)))))
+ (description package-revision-description ;string
+ "description" empty-or-string)
+ (dist package-revision-dist "dist" json->dist)) ;dist
+
+(define (versions->package-revisions versions)
+ (match versions
+ (((version . package-spec) ...)
+ (map json->package-revision package-spec))
+ (_ '())))
+
+(define (versions->package-versions versions)
+ (match versions
+ (((version . package-spec) ...)
+ (map string->semver versions))
+ (_ '())))
+
+(define-json-mapping <meta-package> make-meta-package meta-package?
+ json->meta-package
+ (name meta-package-name) ;string
+ (description meta-package-description) ;string
+ (dist-tags meta-package-dist-tags "dist-tags" json->dist-tags) ;dist-tags
+ (revisions meta-package-revisions "versions" versions->package-revisions))
+
+;; TODO: Support other registries
+(define %registry "https://registry.npmjs.org")
+(define %default-page "https://www.npmjs.com/package")
+
+(define (lookup-meta-package name)
+ (let ((json (json-fetch (string-append %registry "/" (uri-encode name)))))
+ (and=> json json->meta-package)))
+
+(define lookup-meta-package* (memoize lookup-meta-package))
+
+(define (meta-package-versions meta)
+ (map package-revision-version
+ (meta-package-revisions meta)))
+
+(define (meta-package-latest meta)
+ (and=> (meta-package-dist-tags meta) dist-tags-latest))
+
+(define* (meta-package-package meta #:optional
+ (version (meta-package-latest meta)))
+ (match version
+ ((? semver?) (find (lambda (revision)
+ (semver=? version (package-revision-version revision)))
+ (meta-package-revisions meta)))
+ ((? string?) (meta-package-package meta (string->semver version)))
+ (_ #f)))
+
+(define* (semver-latest svs #:optional (svr *semver-range-any*))
+ (find (cut semver-range-contains? svr <>)
+ (sort svs semver>?)))
+
+(define* (resolve-package name #:optional (svr *semver-range-any*))
+ (let ((meta (lookup-meta-package* name)))
+ (and meta
+ (let* ((version (semver-latest (or (meta-package-versions meta) '()) svr))
+ (pkg (meta-package-package meta version)))
+ pkg))))
+
+
+;;;
+;;; Converting packages
+;;;
+
+(define (hash-url url)
+ "Downloads the resource at URL and computes the base32 hash for it."
+ (bytevector->nix-base32-string (port-sha256 (http-fetch url))))
+
+(define (npm-name->name npm-name)
+ "Return a Guix package name for the npm package with name NPM-NAME."
+ (define (clean name)
+ (string-map (lambda (chr) (if (char=? chr #\/) #\- chr))
+ (string-filter (negate (cut char=? <> #\@)) name)))
+ (guix-name "node-" (clean npm-name)))
+
+(define (name+version->symbol name version)
+ (string->symbol (string-append name "-" version)))
+
+(define (package-revision->symbol package)
+ (let* ((npm-name (package-revision-name package))
+ (version (semver->string (package-revision-version package)))
+ (name (npm-name->name npm-name)))
+ (name+version->symbol name version)))
+
+(define (npm-package->package-sexp npm-package)
+ "Return the `package' s-expression for an NPM-PACKAGE."
+ (define resolve-spec
+ (match-lambda
+ (($ <versioned-package> name version)
+ (resolve-package name (string->semver-range version)))))
+
+ (if (package-revision? npm-package)
+ (let ((name (package-revision-name npm-package))
+ (version (package-revision-version npm-package))
+ (home-page (package-revision-home-page npm-package))
+ (dependencies (package-revision-dependencies npm-package))
+ (dev-dependencies (package-revision-dev-dependencies npm-package))
+ (peer-dependencies (package-revision-peer-dependencies npm-package))
+ (license (package-revision-license npm-package))
+ (description (package-revision-description npm-package))
+ (dist (package-revision-dist npm-package)))
+ (let* ((name (npm-name->name name))
+ (url (dist-tarball dist))
+ (home-page (if (string? home-page)
+ home-page
+ (string-append %default-page "/" (uri-encode name))))
+ (synopsis description)
+ (resolved-deps (map resolve-spec
+ (append dependencies peer-dependencies)))
+ (peer-names (map versioned-package-name peer-dependencies))
+ ;; lset-difference for treating peer-dependencies as dependencies,
+ ;; which leads to dependency cycles. lset-union for treating them as
+ ;; (ignored) dev-dependencies, which leads to broken packages.
+ (dev-names
+ (lset-union string=
+ (map versioned-package-name dev-dependencies)
+ peer-names))
+ (extra-phases
+ (match dev-names
+ (() '())
+ ((dev-names ...)
+ `((add-after 'patch-dependencies 'delete-dev-dependencies
+ (lambda _
+ (delete-dependencies '(,@(reverse dev-names))))))))))
+ (values
+ `(package
+ (name ,name)
+ (version ,(semver->string (package-revision-version npm-package)))
+ (source (origin
+ (method url-fetch)
+ (uri ,url)
+ (sha256 (base32 ,(hash-url url)))))
+ (build-system node-build-system)
+ (arguments
+ (list
+ #:tests? #f
+ #:phases
+ #~(modify-phases %standard-phases
+ (delete 'build)
+ ,@extra-phases)))
+ ,@(match dependencies
+ (() '())
+ ((dependencies ...)
+ `((inputs
+ (list ,@(map package-revision->symbol resolved-deps))))))
+ (home-page ,home-page)
+ (synopsis ,synopsis)
+ (description ,description)
+ (license ,license))
+ (map (match-lambda (($ <package-revision> name version)
+ (list name (semver->string version))))
+ resolved-deps))))
+ (values #f '())))
+
+
+;;;
+;;; Interface
+;;;
+
+(define npm-binary->guix-package
+ (lambda* (name #:key (version *semver-range-any*) #:allow-other-keys)
+ (let* ((svr (match version
+ ((? string?) (string->semver-range version))
+ (_ version)))
+ (pkg (resolve-package name svr)))
+ (npm-package->package-sexp pkg))))
+
+(define* (npm-binary-recursive-import package-name #:key version)
+ (recursive-import package-name
+ #:repo->guix-package (memoize npm-binary->guix-package)
+ #:version version
+ #:guix-name npm-name->name))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 1f34cab088..d724f2bca3 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -49,7 +49,7 @@ (define %standard-import-options '())
(define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
"gem" "go" "cran" "crate" "texlive" "json" "opam"
- "minetest" "elm" "hexpm" "composer"))
+ "minetest" "elm" "hexpm" "composer" "npm-binary"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/npm-binary.scm b/guix/scripts/import/npm-binary.scm
new file mode 100644
index 0000000000..d16b0f15b0
--- /dev/null
+++ b/guix/scripts/import/npm-binary.scm
@@ -0,0 +1,108 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2014 David Thompson <davet@gnu.org>
+;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2019 Timothy Sample <samplet@ngyro.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import npm-binary)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import npm-binary)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (srfi srfi-41)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:export (guix-import-npm-binary))
+
+
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import npm-binary PACKAGE-NAME [VERSION]
+Import and convert the npm package PACKAGE-NAME using the
+`node-build-system' (but without building the package from source)."))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -r, --recursive import packages recursively"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import npm-binary")))
+ (option '(#\r "recursive") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'recursive #t result)))
+ %standard-import-options))
+
+
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-npm-binary . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (args-fold* args %options
+ (lambda (opt name arg result)
+ (leave (G_ "~A: unrecognized option~%") name))
+ (lambda (arg result)
+ (alist-cons 'argument arg result))
+ %default-options))
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (let loop ((args args))
+ (match args
+ ((package-name version)
+ (match (if (assoc-ref opts 'recursive)
+ ;; Recursive import
+ (npm-binary-recursive-import package-name #:version version)
+ ;; Single import
+ (npm-binary->guix-package package-name #:version version))
+ ((or #f '())
+ (leave (G_ "failed to download meta-data for package '~a@~a'~%")
+ package-name version))
+ ((? list? sexps) sexps)
+ (sexp (list sexp))))
+ ((package-name)
+ (loop (list package-name "*")))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%")))))))
diff --git a/tests/npm-binary.scm b/tests/npm-binary.scm
new file mode 100755
index 0000000000..cf85e572b3
--- /dev/null
+++ b/tests/npm-binary.scm
@@ -0,0 +1,146 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2024 Jelle Licht <jlicht@fsfe.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even th
This message was truncated. Download the full message here.
J
J
Jelle Licht wrote on 31 Mar 21:57 +0200
Re: [bug#62375] Continue the npm-binary importer
8734s6gooc.fsf@fsfe.org
Hi Pablo,

Pablo Zamora <pablo.zamora@outlook.fr> writes:

Toggle quote (6 lines)
> Dear Guix,
>
> I am stuck while trying to define the 'bitwarden' package definition on my personal channel here https://github.com/Pablo12345678901/guix-custom-channel/blob/master/dev/my-bitwarden.scm . I am facing issues due to the npm dependencies.
>
> One of the potential solution would be this importer. I would like to finish writing it and perform the tests.

That is nice to hear! I just sent v2 of this patch addressing most of
the shortcomings that were identified earlier.

Toggle quote (2 lines)
> On which branch should I send git pull request to continue what has been done so far ? This would be my first commit with guix.

I see two ways forward, depending on how much work you foresee and/or
are willing to do:

1) This patch gets through another round of review, and afterwards you
can send patches based on the 'master' branch.

2) You work on improving this patch while it is still under review, and
send a V{3,4,5} patch as a reply to this very thread.

Kind regards,
Jelle Licht
J
J
jlicht wrote on 31 Mar 22:37 +0200
[PATCH v3] import: Add binary npm importer.
(address . 62375@debbugs.gnu.org)
7148a74f04e1598a18b58fbd0b1b27a2397312b2.1711917305.git.jlicht@fsfe.org
From: Jelle Licht <jlicht@fsfe.org>

* guix/scripts/import.scm: (importers): Add "npm-binary".
* guix/import/npm-binary.scm: New file.
* guix/scripts/import/npm-binary.scm: New file.
* Makefile.am: Add them.

Co-authored-by: Timothy Sample <samplet@ngyro.com>
Co-authored-by: Lars-Dominik Braun <lars@6xq.net>

Change-Id: I98a45068cf5b9c42790664cc743feaa7ac76f807
---

Changes in v3:
- Ensure that package bindings generated during recursive import match the
package bindings used in the list of inputs, as this was a regression introduced in the V2 series.

Changes in v2:
- Change *SOME-VAR* to %SOME-VAR
- Removed unused http-error-code
- Rebase on master
- Refactor hash-url to use port-sha256 helper
- use explicit record accessors instead of order-sensitive destructuring
- address line-width styling issues
- added basic documentation
- added some basic tests (using simple mocks instead of with-http-server)
- simplify import script entrypoint

Makefile.am | 3 +
doc/guix.texi | 33 ++++
guix/import/npm-binary.scm | 277 +++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/npm-binary.scm | 114 ++++++++++++
tests/npm-binary.scm | 146 +++++++++++++++
6 files changed, 574 insertions(+), 1 deletion(-)
create mode 100644 guix/import/npm-binary.scm
create mode 100644 guix/scripts/import/npm-binary.scm
create mode 100755 tests/npm-binary.scm

Toggle diff (496 lines)
diff --git a/Makefile.am b/Makefile.am
index 1c5688ac13..459c47a954 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -305,6 +305,7 @@ MODULES = \
guix/import/kde.scm \
guix/import/launchpad.scm \
guix/import/minetest.scm \
+ guix/import/npm-binary.scm \
guix/import/opam.scm \
guix/import/print.scm \
guix/import/pypi.scm \
@@ -359,6 +360,7 @@ MODULES = \
guix/scripts/import/hexpm.scm \
guix/scripts/import/json.scm \
guix/scripts/import/minetest.scm \
+ guix/scripts/import/npm-binary.scm \
guix/scripts/import/opam.scm \
guix/scripts/import/pypi.scm \
guix/scripts/import/stackage.scm \
@@ -557,6 +559,7 @@ SCM_TESTS = \
tests/modules.scm \
tests/monads.scm \
tests/nar.scm \
+ tests/npm-binary.scm \
tests/networking.scm \
tests/opam.scm \
tests/openpgp.scm \
diff --git a/doc/guix.texi b/doc/guix.texi
index 69a904473c..566af6e849 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -14641,6 +14641,39 @@ Invoking guix import
in Guix.
@end table
+@item npm-binary
+@cindex npm
+@cindex Node.js
+Import metadata from the @uref{https://registry.npmjs.org, npm
+Registry}, as in this example:
+
+@example
+guix import npm-binary buffer-crc32
+@end example
+
+The npm-binary importer also allows you to specify a version string:
+
+@example
+guix import npm-binary buffer-crc32 1.0.0
+@end example
+
+@quotation Note
+Generated package expressions skip the build step of the
+@code{node-build-system}. As such, generated package expressions often
+refer to transpiled or generated files, instead of being built from
+source.
+@end quotation
+
+Additional options include:
+
+@table @code
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
+@end table
+
@item opam
@cindex OPAM
@cindex OCaml
diff --git a/guix/import/npm-binary.scm b/guix/import/npm-binary.scm
new file mode 100644
index 0000000000..57c985baf2
--- /dev/null
+++ b/guix/import/npm-binary.scm
@@ -0,0 +1,277 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2019, 2020 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2020, 2023, 2024 Jelle Licht <jlicht@fsfe.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import npm-binary)
+ #:use-module ((gnu services configuration) #:select (alist?))
+ #:use-module (gcrypt hash)
+ #:use-module (gnu packages)
+ #:use-module (guix base32)
+ #:use-module (guix http-client)
+ #:use-module (guix import json)
+ #:use-module (guix import utils)
+ #:use-module (guix memoization)
+ #:use-module (guix utils)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 receive)
+ #:use-module (ice-9 regex)
+ #:use-module (json)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-41)
+ #:use-module (srfi srfi-9)
+ #:use-module (web client)
+ #:use-module (web response)
+ #:use-module (web uri)
+ #:export (npm-binary-recursive-import
+ npm-binary->guix-package
+ make-versioned-package
+ name+version->symbol))
+
+;; Autoload Guile-Semver so we only have a soft dependency.
+(module-autoload! (current-module)
+ '(semver)
+ '(string->semver semver? semver->string semver=? semver>?))
+(module-autoload! (current-module)
+ '(semver ranges)
+ '(*semver-range-any* string->semver-range semver-range-contains?))
+
+;; Dist-tags
+(define-json-mapping <dist-tags> make-dist-tags dist-tags?
+ json->dist-tags
+ (latest dist-tags-latest "latest" string->semver))
+
+(define-record-type <versioned-package>
+ (make-versioned-package name version)
+ versioned-package?
+ (name versioned-package-name) ;string
+ (version versioned-package-version)) ;string
+
+(define (dependencies->versioned-packages entries)
+ (match entries
+ (((names . versions) ...)
+ (map make-versioned-package names versions))
+ (_ '())))
+
+(define (extract-license license-string)
+ (if (unspecified? license-string)
+ 'unspecified!
+ (spdx-string->license license-string)))
+
+(define-json-mapping <dist> make-dist dist?
+ json->dist
+ (tarball dist-tarball))
+
+(define (empty-or-string s)
+ (if (string? s) s ""))
+
+(define-json-mapping <package-revision> make-package-revision package-revision?
+ json->package-revision
+ (name package-revision-name)
+ (version package-revision-version "version" ;semver
+ string->semver)
+ (home-page package-revision-home-page "homepage") ;string
+ (dependencies package-revision-dependencies ;list of versioned-package
+ "dependencies"
+ dependencies->versioned-packages)
+ (dev-dependencies package-revision-dev-dependencies ;list of versioned-package
+ "devDependencies" dependencies->versioned-packages)
+ (peer-dependencies package-revision-peer-dependencies ;list of versioned-package
+ "peerDependencies" dependencies->versioned-packages)
+ (license package-revision-license "license" ;license | #f
+ (match-lambda
+ ((? unspecified?) #f)
+ ((? string? str) (spdx-string->license str))
+ ((? alist? alist)
+ (match (assoc "type" alist)
+ ((_ . (? string? type))
+ (spdx-string->license type))
+ (_ #f)))))
+ (description package-revision-description ;string
+ "description" empty-or-string)
+ (dist package-revision-dist "dist" json->dist)) ;dist
+
+(define (versions->package-revisions versions)
+ (match versions
+ (((version . package-spec) ...)
+ (map json->package-revision package-spec))
+ (_ '())))
+
+(define (versions->package-versions versions)
+ (match versions
+ (((version . package-spec) ...)
+ (map string->semver versions))
+ (_ '())))
+
+(define-json-mapping <meta-package> make-meta-package meta-package?
+ json->meta-package
+ (name meta-package-name) ;string
+ (description meta-package-description) ;string
+ (dist-tags meta-package-dist-tags "dist-tags" json->dist-tags) ;dist-tags
+ (revisions meta-package-revisions "versions" versions->package-revisions))
+
+;; TODO: Support other registries
+(define %registry "https://registry.npmjs.org")
+(define %default-page "https://www.npmjs.com/package")
+
+(define (lookup-meta-package name)
+ (let ((json (json-fetch (string-append %registry "/" (uri-encode name)))))
+ (and=> json json->meta-package)))
+
+(define lookup-meta-package* (memoize lookup-meta-package))
+
+(define (meta-package-versions meta)
+ (map package-revision-version
+ (meta-package-revisions meta)))
+
+(define (meta-package-latest meta)
+ (and=> (meta-package-dist-tags meta) dist-tags-latest))
+
+(define* (meta-package-package meta #:optional
+ (version (meta-package-latest meta)))
+ (match version
+ ((? semver?) (find (lambda (revision)
+ (semver=? version (package-revision-version revision)))
+ (meta-package-revisions meta)))
+ ((? string?) (meta-package-package meta (string->semver version)))
+ (_ #f)))
+
+(define* (semver-latest svs #:optional (svr *semver-range-any*))
+ (find (cut semver-range-contains? svr <>)
+ (sort svs semver>?)))
+
+(define* (resolve-package name #:optional (svr *semver-range-any*))
+ (let ((meta (lookup-meta-package* name)))
+ (and meta
+ (let* ((version (semver-latest (or (meta-package-versions meta) '()) svr))
+ (pkg (meta-package-package meta version)))
+ pkg))))
+
+
+;;;
+;;; Converting packages
+;;;
+
+(define (hash-url url)
+ "Downloads the resource at URL and computes the base32 hash for it."
+ (bytevector->nix-base32-string (port-sha256 (http-fetch url))))
+
+(define (npm-name->name npm-name)
+ "Return a Guix package name for the npm package with name NPM-NAME."
+ (define (clean name)
+ (string-map (lambda (chr) (if (char=? chr #\/) #\- chr))
+ (string-filter (negate (cut char=? <> #\@)) name)))
+ (guix-name "node-" (clean npm-name)))
+
+(define (name+version->symbol name version)
+ (string->symbol (string-append name "-" version)))
+
+(define (package-revision->symbol package)
+ (let* ((npm-name (package-revision-name package))
+ (version (semver->string (package-revision-version package)))
+ (name (npm-name->name npm-name)))
+ (name+version->symbol name version)))
+
+(define (npm-package->package-sexp npm-package)
+ "Return the `package' s-expression for an NPM-PACKAGE."
+ (define resolve-spec
+ (match-lambda
+ (($ <versioned-package> name version)
+ (resolve-package name (string->semver-range version)))))
+
+ (if (package-revision? npm-package)
+ (let ((name (package-revision-name npm-package))
+ (version (package-revision-version npm-package))
+ (home-page (package-revision-home-page npm-package))
+ (dependencies (package-revision-dependencies npm-package))
+ (dev-dependencies (package-revision-dev-dependencies npm-package))
+ (peer-dependencies (package-revision-peer-dependencies npm-package))
+ (license (package-revision-license npm-package))
+ (description (package-revision-description npm-package))
+ (dist (package-revision-dist npm-package)))
+ (let* ((name (npm-name->name name))
+ (url (dist-tarball dist))
+ (home-page (if (string? home-page)
+ home-page
+ (string-append %default-page "/" (uri-encode name))))
+ (synopsis description)
+ (resolved-deps (map resolve-spec
+ (append dependencies peer-dependencies)))
+ (peer-names (map versioned-package-name peer-dependencies))
+ ;; lset-difference for treating peer-dependencies as dependencies,
+ ;; which leads to dependency cycles. lset-union for treating them as
+ ;; (ignored) dev-dependencies, which leads to broken packages.
+ (dev-names
+ (lset-union string=
+ (map versioned-package-name dev-dependencies)
+ peer-names))
+ (extra-phases
+ (match dev-names
+ (() '())
+ ((dev-names ...)
+ `((add-after 'patch-dependencies 'delete-dev-dependencies
+ (lambda _
+ (delete-dependencies '(,@(reverse dev-names))))))))))
+ (values
+ `(package
+ (name ,name)
+ (version ,(semver->string (package-revision-version npm-package)))
+ (source (origin
+ (method url-fetch)
+ (uri ,url)
+ (sha256 (base32 ,(hash-url url)))))
+ (build-system node-build-system)
+ (arguments
+ (list
+ #:tests? #f
+ #:phases
+ #~(modify-phases %standard-phases
+ (delete 'build)
+ ,@extra-phases)))
+ ,@(match dependencies
+ (() '())
+ ((dependencies ...)
+ `((inputs
+ (list ,@(map package-revision->symbol resolved-deps))))))
+ (home-page ,home-page)
+ (synopsis ,synopsis)
+ (description ,description)
+ (license ,license))
+ (map (match-lambda (($ <package-revision> name version)
+ (list name (semver->string version))))
+ resolved-deps))))
+ (values #f '())))
+
+
+;;;
+;;; Interface
+;;;
+
+(define npm-binary->guix-package
+ (lambda* (name #:key (version *semver-range-any*) #:allow-other-keys)
+ (let* ((svr (match version
+ ((? string?) (string->semver-range version))
+ (_ version)))
+ (pkg (resolve-package name svr)))
+ (npm-package->package-sexp pkg))))
+
+(define* (npm-binary-recursive-import package-name #:key version)
+ (recursive-import package-name
+ #:repo->guix-package (memoize npm-binary->guix-package)
+ #:version version
+ #:guix-name npm-name->name))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 1f34cab088..d724f2bca3 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -49,7 +49,7 @@ (define %standard-import-options '())
(define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
"gem" "go" "cran" "crate" "texlive" "json" "opam"
- "minetest" "elm" "hexpm" "composer"))
+ "minetest" "elm" "hexpm" "composer" "npm-binary"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/npm-binary.scm b/guix/scripts/import/npm-binary.scm
new file mode 100644
index 0000000000..4a4daa7945
--- /dev/null
+++ b/guix/scripts/import/npm-binary.scm
@@ -0,0 +1,114 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2014 David Thompson <davet@gnu.org>
+;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2019 Timothy Sample <samplet@ngyro.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import npm-binary)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import npm-binary)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (srfi srfi-41)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:export (guix-import-npm-binary))
+
+
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import npm-binary PACKAGE-NAME [VERSION]
+Import and convert the npm package PACKAGE-NAME using the
+`node-build-system' (but without building the package from source)."))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -r, --recursive import packages recursively"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import npm-binary")))
+ (option '(#\r "recursive") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'recursive #t result)))
+ %standard-import-options))
+
+
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-npm-binary . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (args-fold* args %options
+ (lambda (opt name arg result)
+ (leave (G_ "~A: unrecognized option~%") name))
+ (lambda (arg result)
+ (alist-cons 'argument arg result))
+ %default-options))
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (let loop ((args args))
+ (match args
+ ((package-name version)
+ (match (if (assoc-ref opts 'recursive)
+ ;; Recursive import
+ (npm-binary-recursive-import package-name #:version version)
+ ;; Single import
+ (npm-binary->guix-package package-name #:version version))
+ ((or #f '())
+ (leave (G_ "failed to download meta-data for package '~a@~a'~%")
+ package-name version))
+ ((? list? sexps)
+ (map (match-lambda
+ ((and ('package ('name name) ('version version) . rest) pkg)
+ `(define-public ,(name+version->symbol name version)
+ ,pkg))
+ (_ #f))
+ sexps))
+ (sexp (list sexp))))
+ ((package-name)
+ (loop (list package-name "*")))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%")))))))
diff --git a/tests/npm-binary.scm b/tests/npm-binary.scm
new file mode 100755
index 0000000000..cf85e572b3
--- /dev/null
+++ b/tests/npm-binary.scm
@@ -0,0 +1,146 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright
This message was truncated. Download the full message here.
P
P
Pablo Zamora wrote on 1 Apr 00:03 +0200
RE: [bug#62375] Continue the npm-binary importer
(name . 62375@debbugs.gnu.org)(address . 62375@debbugs.gnu.org)(name . Jelle Licht)(address . jlicht@fsfe.org)
AS8P189MB20245595664C8E10AA652B7EFD382@AS8P189MB2024.EURP189.PROD.OUTLOOK.COM
Dear Jelle,

Thank you very much for your help.

I will shortly (this week) take a thorough look on your patches so I think of choosing '2)', even temporarily in order to become accustomed on how to develop a Guix importer.

Kind regards,

Pablo Zamora
________________________________
De : Jelle Licht <jlicht@fsfe.org>
Envoyé : dimanche 31 mars 2024 21:57
À : Pablo Zamora <pablo.zamora@outlook.fr>; 62375@debbugs.gnu.org <62375@debbugs.gnu.org>
Objet : Re: [bug#62375] Continue the npm-binary importer


Hi Pablo,

Pablo Zamora <pablo.zamora@outlook.fr> writes:

Toggle quote (6 lines)
> Dear Guix,
>
> I am stuck while trying to define the 'bitwarden' package definition on my personal channel here https://github.com/Pablo12345678901/guix-custom-channel/blob/master/dev/my-bitwarden.scm . I am facing issues due to the npm dependencies.
>
> One of the potential solution would be this importer. I would like to finish writing it and perform the tests.

That is nice to hear! I just sent v2 of this patch addressing most of
the shortcomings that were identified earlier.

Toggle quote (2 lines)
> On which branch should I send git pull request to continue what has been done so far ? This would be my first commit with guix.

I see two ways forward, depending on how much work you foresee and/or
are willing to do:

1) This patch gets through another round of review, and afterwards you
can send patches based on the 'master' branch.

2) You work on improving this patch while it is still under review, and
send a V{3,4,5} patch as a reply to this very thread.

Kind regards,
Jelle Licht
Attachment: file
L
L
Ludovic Courtès wrote on 1 Apr 22:41 +0200
Re: [bug#62375] [PATCH v3] import: Add binary npm importer.
(address . jlicht@fsfe.org)
871q7odde1.fsf@gnu.org
Hi!

jlicht@fsfe.org skribis:

Toggle quote (12 lines)
> From: Jelle Licht <jlicht@fsfe.org>
>
> * guix/scripts/import.scm: (importers): Add "npm-binary".
> * guix/import/npm-binary.scm: New file.
> * guix/scripts/import/npm-binary.scm: New file.
> * Makefile.am: Add them.
>
> Co-authored-by: Timothy Sample <samplet@ngyro.com>
> Co-authored-by: Lars-Dominik Braun <lars@6xq.net>
>
> Change-Id: I98a45068cf5b9c42790664cc743feaa7ac76f807

Yay!

Toggle quote (6 lines)
> +The npm-binary importer also allows you to specify a version string:
> +
> +@example
> +guix import npm-binary buffer-crc32 1.0.0
> +@end example

For consistency with other importers (pypi, gem, cran), could you change
the syntax to:

guix import npm-binary buffer-crc32@1.0.0

?

That’s the last remaining issue for me.

Toggle quote (3 lines)
> +;; TODO: Support other registries
> +(define %registry "https://registry.npmjs.org")

For the purposes of tests, you could make it:

(define %npm-registry
(make-parameter "https://registry.npmjs.org"))

That would allow you to write tests using ‘with-http-server’ and
‘parameterize’ as done in ‘tests/pypi.scm’ and others, which I find
nicer and more robust than ‘mock’.

Not a blocker though.

Thanks!

Ludo’.
J
J
Jonathan Brielmaier wrote on 2 Apr 00:01 +0200
[PATCH 0/1] npm binary importer
(address . 62375@debbugs.gnu.org)
ea36dc83-ad86-4da8-57d7-adf8630daeab@web.de
Hi Jelle,

one rather unimportant remark. The commit message mentions Lars-Dominik
as co-author:

But there is no copyright header for him in any of the added files...

~Jonathan
J
J
Jelle Licht wrote on 2 Apr 16:12 +0200
Re: [bug#62375] [PATCH v3] import: Add binary npm importer.
(name . Ludovic Courtès)(address . ludo@gnu.org)
87r0fng8ge.fsf@fsfe.org
Hi Ludo',

Thanks for the speedy review!

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (34 lines)
> Hi!
>
> jlicht@fsfe.org skribis:
>
>> From: Jelle Licht <jlicht@fsfe.org>
>>
>> * guix/scripts/import.scm: (importers): Add "npm-binary".
>> * guix/import/npm-binary.scm: New file.
>> * guix/scripts/import/npm-binary.scm: New file.
>> * Makefile.am: Add them.
>>
>> Co-authored-by: Timothy Sample <samplet@ngyro.com>
>> Co-authored-by: Lars-Dominik Braun <lars@6xq.net>
>>
>> Change-Id: I98a45068cf5b9c42790664cc743feaa7ac76f807
>
> Yay!
>
>> +The npm-binary importer also allows you to specify a version string:
>> +
>> +@example
>> +guix import npm-binary buffer-crc32 1.0.0
>> +@end example
>
> For consistency with other importers (pypi, gem, cran), could you change
> the syntax to:
>
> guix import npm-binary buffer-crc32@1.0.0
>
> ?
>
> That’s the last remaining issue for me.
>

I needed some custom logic to support the npm scoped packages
("@linthtml/linthtml@1.2.3"), but it should now work.

Toggle quote (8 lines)
>> +;; TODO: Support other registries
>> +(define %registry "https://registry.npmjs.org")
>
> For the purposes of tests, you could make it:
>
> (define %npm-registry
> (make-parameter "https://registry.npmjs.org"))

Easy!

Toggle quote (4 lines)
> That would allow you to write tests using ‘with-http-server’ and
> ‘parameterize’ as done in ‘tests/pypi.scm’ and others, which I find
> nicer and more robust than ‘mock’.

The `with-http-server' construct does not play well with an interactive
REPL workflow due to not cleaning up the used port in all situations,
making that port unavailable for running the test again[1]. `mock',
brittle as it may be, does not to suffer from this drawback, so I'd
vastly prefer using that for now.

Toggle quote (6 lines)
> Not a blocker though.
>
> Thanks!
>
> Ludo’.

I'll send a v4, let it simmer for some days and then merge it if no big
blockers or requested changes show up.

- Jelle
J
J
jlicht wrote on 2 Apr 16:13 +0200
[PATCH v4] import: Add binary npm importer.
(address . 62375@debbugs.gnu.org)
1892fd7b000320e56cbf10958fd4cf88d3c83ced.1712067148.git.jlicht@fsfe.org
From: Jelle Licht <jlicht@fsfe.org>

* guix/scripts/import.scm: (importers): Add "npm-binary".
* guix/import/npm-binary.scm: New file.
* guix/scripts/import/npm-binary.scm: New file.
* Makefile.am: Add them.

Co-authored-by: Timothy Sample <samplet@ngyro.com>
Co-authored-by: Lars-Dominik Braun <lars@6xq.net>

Change-Id: I98a45068cf5b9c42790664cc743feaa7ac76f807
---

Changes in v4:
- Add copyright line for LDB
- Use package-name@version-spec notation on CLI
- Simplify CLI argument handling
- Turn %registry into a parameter named %npm-registry

Changes in v3:
- Ensure that package bindings generated during recursive import match the
package bindings used in the list of inputs

Changes in v2:
- Change *SOME-VAR* to %SOME-VAR
- Removed unused http-error-code
- Rebase on master
- Refactor hash-url to use port-sha256 helper
- use explicit record accessors instead of order-sensitive destructuring
- address line-width styling issues
- added basic documentation
- added some basic tests (using simple mocks instead of with-http-server)
- simplify import script entrypoint

Makefile.am | 3 +
doc/guix.texi | 33 ++++
guix/import/npm-binary.scm | 279 +++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/npm-binary.scm | 121 +++++++++++++
tests/npm-binary.scm | 146 +++++++++++++++
6 files changed, 583 insertions(+), 1 deletion(-)
create mode 100644 guix/import/npm-binary.scm
create mode 100644 guix/scripts/import/npm-binary.scm
create mode 100755 tests/npm-binary.scm

Toggle diff (490 lines)
diff --git a/Makefile.am b/Makefile.am
index 1c5688ac13..459c47a954 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -305,6 +305,7 @@ MODULES = \
guix/import/kde.scm \
guix/import/launchpad.scm \
guix/import/minetest.scm \
+ guix/import/npm-binary.scm \
guix/import/opam.scm \
guix/import/print.scm \
guix/import/pypi.scm \
@@ -359,6 +360,7 @@ MODULES = \
guix/scripts/import/hexpm.scm \
guix/scripts/import/json.scm \
guix/scripts/import/minetest.scm \
+ guix/scripts/import/npm-binary.scm \
guix/scripts/import/opam.scm \
guix/scripts/import/pypi.scm \
guix/scripts/import/stackage.scm \
@@ -557,6 +559,7 @@ SCM_TESTS = \
tests/modules.scm \
tests/monads.scm \
tests/nar.scm \
+ tests/npm-binary.scm \
tests/networking.scm \
tests/opam.scm \
tests/openpgp.scm \
diff --git a/doc/guix.texi b/doc/guix.texi
index 69a904473c..f8f8f7bf3a 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -14641,6 +14641,39 @@ Invoking guix import
in Guix.
@end table
+@item npm-binary
+@cindex npm
+@cindex Node.js
+Import metadata from the @uref{https://registry.npmjs.org, npm
+Registry}, as in this example:
+
+@example
+guix import npm-binary buffer-crc32
+@end example
+
+The npm-binary importer also allows you to specify a version string:
+
+@example
+guix import npm-binary buffer-crc32@@1.0.0
+@end example
+
+@quotation Note
+Generated package expressions skip the build step of the
+@code{node-build-system}. As such, generated package expressions often
+refer to transpiled or generated files, instead of being built from
+source.
+@end quotation
+
+Additional options include:
+
+@table @code
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
+@end table
+
@item opam
@cindex OPAM
@cindex OCaml
diff --git a/guix/import/npm-binary.scm b/guix/import/npm-binary.scm
new file mode 100644
index 0000000000..6dfedc4910
--- /dev/null
+++ b/guix/import/npm-binary.scm
@@ -0,0 +1,279 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2019, 2020 Timothy Sample <samplet@ngyro.com>
+;;; Copyright © 2021 Lars-Dominik Braun <lars@6xq.net>
+;;; Copyright © 2020, 2023, 2024 Jelle Licht <jlicht@fsfe.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import npm-binary)
+ #:use-module ((gnu services configuration) #:select (alist?))
+ #:use-module (gcrypt hash)
+ #:use-module (gnu packages)
+ #:use-module (guix base32)
+ #:use-module (guix http-client)
+ #:use-module (guix import json)
+ #:use-module (guix import utils)
+ #:use-module (guix memoization)
+ #:use-module (guix utils)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 receive)
+ #:use-module (ice-9 regex)
+ #:use-module (json)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-26)
+ #:use-module (srfi srfi-41)
+ #:use-module (srfi srfi-9)
+ #:use-module (web client)
+ #:use-module (web response)
+ #:use-module (web uri)
+ #:export (npm-binary-recursive-import
+ npm-binary->guix-package
+ %npm-registry
+ make-versioned-package
+ name+version->symbol))
+
+;; Autoload Guile-Semver so we only have a soft dependency.
+(module-autoload! (current-module)
+ '(semver)
+ '(string->semver semver? semver->string semver=? semver>?))
+(module-autoload! (current-module)
+ '(semver ranges)
+ '(*semver-range-any* string->semver-range semver-range-contains?))
+
+;; Dist-tags
+(define-json-mapping <dist-tags> make-dist-tags dist-tags?
+ json->dist-tags
+ (latest dist-tags-latest "latest" string->semver))
+
+(define-record-type <versioned-package>
+ (make-versioned-package name version)
+ versioned-package?
+ (name versioned-package-name) ;string
+ (version versioned-package-version)) ;string
+
+(define (dependencies->versioned-packages entries)
+ (match entries
+ (((names . versions) ...)
+ (map make-versioned-package names versions))
+ (_ '())))
+
+(define (extract-license license-string)
+ (if (unspecified? license-string)
+ 'unspecified!
+ (spdx-string->license license-string)))
+
+(define-json-mapping <dist> make-dist dist?
+ json->dist
+ (tarball dist-tarball))
+
+(define (empty-or-string s)
+ (if (string? s) s ""))
+
+(define-json-mapping <package-revision> make-package-revision package-revision?
+ json->package-revision
+ (name package-revision-name)
+ (version package-revision-version "version" ;semver
+ string->semver)
+ (home-page package-revision-home-page "homepage") ;string
+ (dependencies package-revision-dependencies ;list of versioned-package
+ "dependencies"
+ dependencies->versioned-packages)
+ (dev-dependencies package-revision-dev-dependencies ;list of versioned-package
+ "devDependencies" dependencies->versioned-packages)
+ (peer-dependencies package-revision-peer-dependencies ;list of versioned-package
+ "peerDependencies" dependencies->versioned-packages)
+ (license package-revision-license "license" ;license | #f
+ (match-lambda
+ ((? unspecified?) #f)
+ ((? string? str) (spdx-string->license str))
+ ((? alist? alist)
+ (match (assoc "type" alist)
+ ((_ . (? string? type))
+ (spdx-string->license type))
+ (_ #f)))))
+ (description package-revision-description ;string
+ "description" empty-or-string)
+ (dist package-revision-dist "dist" json->dist)) ;dist
+
+(define (versions->package-revisions versions)
+ (match versions
+ (((version . package-spec) ...)
+ (map json->package-revision package-spec))
+ (_ '())))
+
+(define (versions->package-versions versions)
+ (match versions
+ (((version . package-spec) ...)
+ (map string->semver versions))
+ (_ '())))
+
+(define-json-mapping <meta-package> make-meta-package meta-package?
+ json->meta-package
+ (name meta-package-name) ;string
+ (description meta-package-description) ;string
+ (dist-tags meta-package-dist-tags "dist-tags" json->dist-tags) ;dist-tags
+ (revisions meta-package-revisions "versions" versions->package-revisions))
+
+(define %npm-registry
+ (make-parameter "https://registry.npmjs.org"))
+(define %default-page "https://www.npmjs.com/package")
+
+(define (lookup-meta-package name)
+ (let ((json (json-fetch (string-append (%npm-registry) "/" (uri-encode name)))))
+ (and=> json json->meta-package)))
+
+(define lookup-meta-package* (memoize lookup-meta-package))
+
+(define (meta-package-versions meta)
+ (map package-revision-version
+ (meta-package-revisions meta)))
+
+(define (meta-package-latest meta)
+ (and=> (meta-package-dist-tags meta) dist-tags-latest))
+
+(define* (meta-package-package meta #:optional
+ (version (meta-package-latest meta)))
+ (match version
+ ((? semver?) (find (lambda (revision)
+ (semver=? version (package-revision-version revision)))
+ (meta-package-revisions meta)))
+ ((? string?) (meta-package-package meta (string->semver version)))
+ (_ #f)))
+
+(define* (semver-latest svs #:optional (svr *semver-range-any*))
+ (find (cut semver-range-contains? svr <>)
+ (sort svs semver>?)))
+
+(define* (resolve-package name #:optional (svr *semver-range-any*))
+ (let ((meta (lookup-meta-package* name)))
+ (and meta
+ (let* ((version (semver-latest (or (meta-package-versions meta) '()) svr))
+ (pkg (meta-package-package meta version)))
+ pkg))))
+
+
+;;;
+;;; Converting packages
+;;;
+
+(define (hash-url url)
+ "Downloads the resource at URL and computes the base32 hash for it."
+ (bytevector->nix-base32-string (port-sha256 (http-fetch url))))
+
+(define (npm-name->name npm-name)
+ "Return a Guix package name for the npm package with name NPM-NAME."
+ (define (clean name)
+ (string-map (lambda (chr) (if (char=? chr #\/) #\- chr))
+ (string-filter (negate (cut char=? <> #\@)) name)))
+ (guix-name "node-" (clean npm-name)))
+
+(define (name+version->symbol name version)
+ (string->symbol (string-append name "-" version)))
+
+(define (package-revision->symbol package)
+ (let* ((npm-name (package-revision-name package))
+ (version (semver->string (package-revision-version package)))
+ (name (npm-name->name npm-name)))
+ (name+version->symbol name version)))
+
+(define (npm-package->package-sexp npm-package)
+ "Return the `package' s-expression for an NPM-PACKAGE."
+ (define resolve-spec
+ (match-lambda
+ (($ <versioned-package> name version)
+ (resolve-package name (string->semver-range version)))))
+
+ (if (package-revision? npm-package)
+ (let ((name (package-revision-name npm-package))
+ (version (package-revision-version npm-package))
+ (home-page (package-revision-home-page npm-package))
+ (dependencies (package-revision-dependencies npm-package))
+ (dev-dependencies (package-revision-dev-dependencies npm-package))
+ (peer-dependencies (package-revision-peer-dependencies npm-package))
+ (license (package-revision-license npm-package))
+ (description (package-revision-description npm-package))
+ (dist (package-revision-dist npm-package)))
+ (let* ((name (npm-name->name name))
+ (url (dist-tarball dist))
+ (home-page (if (string? home-page)
+ home-page
+ (string-append %default-page "/" (uri-encode name))))
+ (synopsis description)
+ (resolved-deps (map resolve-spec
+ (append dependencies peer-dependencies)))
+ (peer-names (map versioned-package-name peer-dependencies))
+ ;; lset-difference for treating peer-dependencies as dependencies,
+ ;; which leads to dependency cycles. lset-union for treating them as
+ ;; (ignored) dev-dependencies, which leads to broken packages.
+ (dev-names
+ (lset-union string=
+ (map versioned-package-name dev-dependencies)
+ peer-names))
+ (extra-phases
+ (match dev-names
+ (() '())
+ ((dev-names ...)
+ `((add-after 'patch-dependencies 'delete-dev-dependencies
+ (lambda _
+ (delete-dependencies '(,@(reverse dev-names))))))))))
+ (values
+ `(package
+ (name ,name)
+ (version ,(semver->string (package-revision-version npm-package)))
+ (source (origin
+ (method url-fetch)
+ (uri ,url)
+ (sha256 (base32 ,(hash-url url)))))
+ (build-system node-build-system)
+ (arguments
+ (list
+ #:tests? #f
+ #:phases
+ #~(modify-phases %standard-phases
+ (delete 'build)
+ ,@extra-phases)))
+ ,@(match dependencies
+ (() '())
+ ((dependencies ...)
+ `((inputs
+ (list ,@(map package-revision->symbol resolved-deps))))))
+ (home-page ,home-page)
+ (synopsis ,synopsis)
+ (description ,description)
+ (license ,license))
+ (map (match-lambda (($ <package-revision> name version)
+ (list name (semver->string version))))
+ resolved-deps))))
+ (values #f '())))
+
+
+;;;
+;;; Interface
+;;;
+
+(define npm-binary->guix-package
+ (lambda* (name #:key (version *semver-range-any*) #:allow-other-keys)
+ (let* ((svr (match version
+ ((? string?) (string->semver-range version))
+ (_ version)))
+ (pkg (resolve-package name svr)))
+ (npm-package->package-sexp pkg))))
+
+(define* (npm-binary-recursive-import package-name #:key version)
+ (recursive-import package-name
+ #:repo->guix-package (memoize npm-binary->guix-package)
+ #:version version
+ #:guix-name npm-name->name))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 1f34cab088..d724f2bca3 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -49,7 +49,7 @@ (define %standard-import-options '())
(define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
"gem" "go" "cran" "crate" "texlive" "json" "opam"
- "minetest" "elm" "hexpm" "composer"))
+ "minetest" "elm" "hexpm" "composer" "npm-binary"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/npm-binary.scm b/guix/scripts/import/npm-binary.scm
new file mode 100644
index 0000000000..3403a69bcc
--- /dev/null
+++ b/guix/scripts/import/npm-binary.scm
@@ -0,0 +1,121 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2014 David Thompson <davet@gnu.org>
+;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2019 Timothy Sample <samplet@ngyro.com>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import npm-binary)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import npm-binary)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (srfi srfi-41)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:export (guix-import-npm-binary))
+
+
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import npm-binary PACKAGE-NAME [VERSION]
+Import and convert the npm package PACKAGE-NAME using the
+`node-build-system' (but without building the package from source)."))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -r, --recursive import packages recursively"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import npm-binary")))
+ (option '(#\r "recursive") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'recursive #t result)))
+ %standard-import-options))
+
+(define* (package-name->name+version* spec)
+ "Given SPEC, a package name like \"@scope/pac@^0.9.1\", return two values:
+\"@scope/pac\" and \"^0.9.1\". When the version part is unavailable, SPEC and \"*\"
+are returned. The first part may start with '@', the latter part must not contain
+contain '@'."
+ (match (string-rindex spec delimiter)
+ (#f (values spec "*"))
+ (0 (values spec "*"))
+ (idx (values (substring spec 0 idx)
+ (substring spec (1+ idx))))))
+
+
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-npm-binary . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (parse-command-line args %options (list %default-options)
+ #:build-options? #f))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((spec)
+ (define-values (package-name version)
+ (package-name->name+version* spec))
+ (match (if (assoc-ref opts 'recursive)
+ ;; Recursive import
+ (npm-binary-recursive-import package-name #:version version)
+ ;; Single import
+ (npm-binary->guix-package package-name #:version version))
+ ((or #f '())
+ (leave (G_ "failed to download meta-data for package '~a@~a'~%")
+ package-name version))
+ (('package etc ...) `(package ,@etc))
+ ((? list? sexps)
+ (map (match-lambda
+ ((and ('package ('name name) ('version version) . rest) pkg)
+ `
This message was truncated. Download the full message here.
J
J
Jelle Licht wrote on 2 Apr 16:16 +0200
Re: [bug#62375] [PATCH 0/1] npm binary importer
87o7arg88a.fsf@fsfe.org
Jonathan Brielmaier via Guix-patches via <guix-patches@gnu.org> writes:

Toggle quote (8 lines)
> Hi Jelle,
>
> one rather unimportant remark. The commit message mentions Lars-Dominik
> as co-author:
> https://issues.guix.gnu.org/62375#12-lineno8
>
> But there is no copyright header for him in any of the added files...

Thanks for the reminder; it was a contribution that they shared in 2021,
so I've now added the copyright header in V4.

- Jelle
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 62375@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 62375
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch