[PATCH 0/4] Update Conda

  • Done
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Pierre Neidhardt
  • Ricardo Wurmus
Owner
unassigned
Submitted by
Pierre Neidhardt
Severity
normal
P
P
Pierre Neidhardt wrote on 31 May 2019 15:57
(address . guix-patches@gnu.org)
20190531135729.890-1-mail@ambrevar.xyz
Recent conda needs a separate python-conda-package-handling library.
This library requires a libarchive built with zstd support.
So far so good, except that python-libarchive-c does not seem to support zstd in its source code.

I've reported the issue upstream
according to them python-libarchive-c is fine.

I don't have enough Python knowledge to understand whether upstream is right
or if there is indeed something fishy with python-libarchive-c.
Pierre Neidhardt (4):
gnu: libarchive: Add zstd support.
gnu: python-libarchive-c: Explicitly rely on latest libarchive.
gnu: Add python-conda-package-handling.
gnu: python-conda: Update to 4.7.0.

gnu/packages/backup.scm | 10 ++--
gnu/packages/package-management.scm | 81 ++++++++++++++++-------------
gnu/packages/python-xyz.scm | 2 +-
3 files changed, 54 insertions(+), 39 deletions(-)

--
2.21.0
P
P
Pierre Neidhardt wrote on 31 May 2019 16:04
[PATCH 1/4] gnu: libarchive: Add zstd support.
(address . 36026@debbugs.gnu.org)
20190531140444.3810-1-mail@ambrevar.xyz
* gnu/packages/backup.scm (libarchive)[inputs]: Include zstd.
[arguments]: Build against zstd.
---
gnu/packages/backup.scm | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)

Toggle diff (37 lines)
diff --git a/gnu/packages/backup.scm b/gnu/packages/backup.scm
index b15c15ad46..da128a0901 100644
--- a/gnu/packages/backup.scm
+++ b/gnu/packages/backup.scm
@@ -216,7 +216,8 @@ backups (called chunks) to allow easy burning to CD/DVD.")
("lzo" ,lzo)
("bzip2" ,bzip2)
("libxml2" ,libxml2)
- ("xz" ,xz)))
+ ("xz" ,xz)
+ ("zstd" ,zstd)))
(arguments
`(#:phases
(modify-phases %standard-phases
@@ -245,7 +246,8 @@ backups (called chunks) to allow easy burning to CD/DVD.")
(libxml2 (assoc-ref inputs "libxml2"))
(xz (assoc-ref inputs "xz"))
(zlib (assoc-ref inputs "zlib"))
- (bzip2 (assoc-ref inputs "bzip2")))
+ (bzip2 (assoc-ref inputs "bzip2"))
+ (zstd (assoc-ref inputs "zstd")))
(substitute* (string-append lib "/pkgconfig/libarchive.pc")
(("-lnettle")
(string-append "-L" nettle "/lib -lnettle"))
@@ -256,7 +258,9 @@ backups (called chunks) to allow easy burning to CD/DVD.")
(("-lz")
(string-append "-L" zlib "/lib -lz"))
(("-lbz2")
- (string-append "-L" bzip2 "/lib -lbz2")))
+ (string-append "-L" bzip2 "/lib -lbz2"))
+ (("-lzstd")
+ (string-append "-L" zstd "/lib -lzstd")))
#t))))
;; libarchive/test/test_write_format_gnutar_filenames.c needs to be
--
2.21.0
P
P
Pierre Neidhardt wrote on 31 May 2019 16:04
[PATCH 2/4] gnu: python-libarchive-c: Explicitly rely on latest libarchive.
(address . 36026@debbugs.gnu.org)
20190531140444.3810-2-mail@ambrevar.xyz
* gnu/packages/python-xyz.scm (python-libarchive-c)[inputs]: Do it.
---
gnu/packages/python-xyz.scm | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

Toggle diff (15 lines)
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 5322b70759..9c9da1d678 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -7191,7 +7191,7 @@ a hash value.")
`(("python-mock" ,python-mock)
("python-pytest" ,python-pytest)))
(inputs
- `(("libarchive" ,libarchive)))
+ `(("libarchive" ,libarchive-3.3.3)))
(home-page "https://github.com/Changaco/python-libarchive-c")
(synopsis "Python interface to libarchive")
(description
--
2.21.0
P
P
Pierre Neidhardt wrote on 31 May 2019 16:04
[PATCH 3/4] gnu: Add python-conda-package-handling.
(address . 36026@debbugs.gnu.org)
20190531140444.3810-3-mail@ambrevar.xyz
* gnu/packages/package-management.scm (python-conda-package-handling): New variable.
---
gnu/packages/package-management.scm | 36 +++++++++++++++++++++++++++++
1 file changed, 36 insertions(+)

Toggle diff (49 lines)
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index b4e1e077fe..28e23fd1d3 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -716,6 +716,42 @@ environments.")
(define-public python2-anaconda-client
(package-with-python2 python-anaconda-client))
+(define-public python-conda-package-handling
+ (package
+ (name "python-conda-package-handling")
+ (version "1.1.1")
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/conda/conda-package-handling/")
+ (commit version)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ "0qjraqb10bnnr329n68s97lkxh8mxchd6dmsynd0n6i9h49hhfgq"))))
+ (build-system python-build-system)
+ (inputs
+ `(("python-six" ,python-six)
+ ("python-libarchive-c" ,python-libarchive-c)
+ ("python-tqdm" ,python-tqdm)))
+ (native-inputs
+ `(("python-pytest" ,python-pytest)
+ ("python-pytest-cov" ,python-pytest-cov)))
+ (arguments
+ `(#:phases
+ (modify-phases %standard-phases
+ (replace 'check
+ (lambda _
+ ;; TODO: Tests fail.
+ (invoke "pytest" "-vv" "tests"))))))
+ (home-page "https://conda.io")
+ (synopsis "Create and extract conda packages of various formats")
+ (description
+ "This library is an abstraction of conda package handling and a tool for
+extracting, creating, and converting between formats.")
+ (license license:bsd-3)))
+
(define-public python-conda
(package
(name "python-conda")
--
2.21.0
P
P
Pierre Neidhardt wrote on 31 May 2019 16:04
[PATCH 4/4] gnu: python-conda: Update to 4.7.0.
(address . 36026@debbugs.gnu.org)
20190531140444.3810-4-mail@ambrevar.xyz
* gnu/packages/package-management.scm (python-conda): Update to 4.7.0.
---
gnu/packages/package-management.scm | 45 +++++++----------------------
1 file changed, 10 insertions(+), 35 deletions(-)

Toggle diff (76 lines)
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 28e23fd1d3..fb84f9abfc 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -755,16 +755,17 @@ extracting, creating, and converting between formats.")
(define-public python-conda
(package
(name "python-conda")
- (version "4.3.16")
+ (version "4.7.0")
(source
(origin
- (method url-fetch)
- (uri (string-append "https://github.com/conda/conda/archive/"
- version ".tar.gz"))
- (file-name (string-append name "-" version ".tar.gz"))
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://github.com/conda/conda/")
+ (commit version)))
+ (file-name (git-file-name name version))
(sha256
(base32
- "1jq8hyrc5npb5sf4vw6s6by4602yj8f79vzpbwdfgpkn02nfk1dv"))))
+ "0dzplykxrlri9a5c65bc48sdjpg2i50qzjf1qf9448qkghvmnmpr"))))
(build-system python-build-system)
(arguments
`(#:phases
@@ -774,33 +775,6 @@ extracting, creating, and converting between formats.")
(with-output-to-file "conda/.version"
(lambda () (display ,version)))
#t))
- (add-before 'check 'remove-failing-tests
- (lambda _
- ;; These tests require internet/network access
- (let ((network-tests '("test_cli.py"
- "test_create.py"
- "test_export.py"
- "test_fetch.py"
- "test_history.py"
- "test_info.py"
- "test_install.py"
- "test_priority.py"
- "conda_env/test_cli.py"
- "conda_env/test_create.py"
- "conda_env/specs/test_notebook.py"
- "conda_env/utils/test_notebooks.py"
- "core/test_index.py"
- "core/test_repodata.py")))
- (with-directory-excursion "tests"
- (for-each delete-file network-tests)
-
- ;; FIXME: This test creates a file, then deletes it and tests
- ;; that the file was deleted. For some reason it fails when
- ;; building with guix, but does not when you run it in the
- ;; directory left when you build with the --keep-failed
- ;; option
- (delete-file "gateways/disk/test_delete.py")
- #t))))
(replace 'check
(lambda _
(setenv "HOME" "/tmp")
@@ -813,8 +787,9 @@ extracting, creating, and converting between formats.")
("python-pytest" ,python-pytest)
("python-responses" ,python-responses)
("python-pyyaml" ,python-pyyaml)
- ("python-anaconda-client" ,python-anaconda-client)))
- (home-page "https://github.com/conda/conda")
+ ("python-anaconda-client" ,python-anaconda-client)
+ ("python-conda-package-handling" ,python-conda-package-handling)))
+ (home-page "https://conda.io")
(synopsis "Cross-platform, OS-agnostic, system-level binary package manager")
(description
"Conda is a cross-platform, Python-agnostic binary package manager. It
--
2.21.0
L
L
Ludovic Courtès wrote on 13 Jun 2019 10:45
Re: [bug#36026] [PATCH 0/4] Update Conda
(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 36026@debbugs.gnu.org)
87o9316fhs.fsf@gnu.org
Pierre Neidhardt <mail@ambrevar.xyz> skribis:

Toggle quote (5 lines)
> gnu: libarchive: Add zstd support.
> gnu: python-libarchive-c: Explicitly rely on latest libarchive.
> gnu: Add python-conda-package-handling.
> gnu: python-conda: Update to 4.7.0.

Apart from the rebuild issue with patch #1 that needs to be addressed,
it LGTM!

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 13 Jun 2019 10:44
Re: [bug#36026] [PATCH 1/4] gnu: libarchive: Add zstd support.
(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 36026@debbugs.gnu.org)
87sgsd6fjc.fsf@gnu.org
Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

Toggle quote (3 lines)
> * gnu/packages/backup.scm (libarchive)[inputs]: Include zstd.
> [arguments]: Build against zstd.

Here you’re modifying ‘libarchive’, which has tons of dependents.

If you instead modify ‘libarchive-3.3.3’, you’ll be able to apply it
directly on master, and follow up with a patch to apply the changes on
‘libarchive’ itself in ‘core-updates’.

WDYT?

Thanks,
Ludo’.
P
P
Pierre Neidhardt wrote on 13 Jun 2019 11:08
Re: [bug#36026] [PATCH 0/4] Update Conda
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 36026@debbugs.gnu.org)
87o9313l9y.fsf@ambrevar.xyz
Thanks for the clarifications around the libarchive business.

The problem is that this patch still does not work because
python-libarchive-c does not support zstd, which
python-conda-package-handling needs. I don't understand how upstream
does this. I've reported it upstream:


My knowledge of python is not enough to understand if upstream is wrong
or if I'm using python-libarchive-c incorrectly.

--
Pierre Neidhardt
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl0CEokACgkQm9z0l6S7
zH+nzQf+KpFqMHYiOogD2IZLIZ7zx122CKEFHeHGCsETtSB95n6CfdozqxsONjvu
h9BCkfZgSRMWJTW/KDU2mGqFH612rKrLLuo39XeywFTw8Ydqn+CTym32iDRdnBil
FZwPeCY+Ps6oXB2W8zI8JiSE06DSJRDR32Tq5eYHCRyL39PvlM8zBDKfGe4DB1fu
XlO/OK7ly4YYA9HQHYpP1+q7WeUwYWGFrMbAhETLlUPfdQ/tJ2S5cpeJyl1TsuNn
EFMV6R6Y4VmRVZFqOpCS7G/O9PVcjnDzdsvH1daFKxHwHH4H0Z3K5AIHwjc0b+sK
tqOBk/NiJeAqvdDGi8GuFGjmmdCBkg==
=IddL
-----END PGP SIGNATURE-----

R
R
Ricardo Wurmus wrote on 13 Jun 2019 20:57
(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 36026@debbugs.gnu.org)
87pnnhqpo2.fsf@elephly.net
Hi Pierre,

Toggle quote (16 lines)
> Recent conda needs a separate python-conda-package-handling library.
> This library requires a libarchive built with zstd support.
> So far so good, except that python-libarchive-c does not seem to support zstd in its source code.
>
> I've reported the issue upstream
> (https://github.com/conda/conda-package-handling/issues/14) but
> according to them python-libarchive-c is fine.
>
> I don't have enough Python knowledge to understand whether upstream is right
> or if there is indeed something fishy with python-libarchive-c.
> Pierre Neidhardt (4):
> gnu: libarchive: Add zstd support.
> gnu: python-libarchive-c: Explicitly rely on latest libarchive.
> gnu: Add python-conda-package-handling.
> gnu: python-conda: Update to 4.7.0.

Does this actually work? If it does: congratulations!

I tried to upgrade conda a while ago, but I got stuck:


How did you overcome these problems? I had to patch a lot of things and
even then couldn’t make it work. Did I miss something obvious?

I’m glad you took care of this and I didn’t have to become the Conda
guy… :)

--
Ricardo
R
R
Ricardo Wurmus wrote on 13 Jun 2019 20:59
Re: [bug#36026] [PATCH 4/4] gnu: python-conda: Update to 4.7.0.
(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)(address . 36026@debbugs.gnu.org)
87o931qpks.fsf@elephly.net
Pierre Neidhardt <mail@ambrevar.xyz> writes:

Toggle quote (1 lines)
> * gnu/packages/package-management.scm (python-conda): Update to 4.7.0.
[…]
Toggle quote (28 lines)
> - (add-before 'check 'remove-failing-tests
> - (lambda _
> - ;; These tests require internet/network access
> - (let ((network-tests '("test_cli.py"
> - "test_create.py"
> - "test_export.py"
> - "test_fetch.py"
> - "test_history.py"
> - "test_info.py"
> - "test_install.py"
> - "test_priority.py"
> - "conda_env/test_cli.py"
> - "conda_env/test_create.py"
> - "conda_env/specs/test_notebook.py"
> - "conda_env/utils/test_notebooks.py"
> - "core/test_index.py"
> - "core/test_repodata.py")))
> - (with-directory-excursion "tests"
> - (for-each delete-file network-tests)
> -
> - ;; FIXME: This test creates a file, then deletes it and tests
> - ;; that the file was deleted. For some reason it fails when
> - ;; building with guix, but does not when you run it in the
> - ;; directory left when you build with the --keep-failed
> - ;; option
> - (delete-file "gateways/disk/test_delete.py")
> - #t))))

Do none of the tests now require network access? Are the tests actually
still run?

--
Ricardo
P
P
Pierre Neidhardt wrote on 13 Jun 2019 21:06
Re: [bug#36026] [PATCH 0/4] Update Conda
(name . Ricardo Wurmus)(address . rekado@elephly.net)(address . 36026@debbugs.gnu.org)
87sgsd9uey.fsf@ambrevar.xyz
Hi Ricardo,

Sorry to kill your enthusiasm, but nope, it does not work :/
The is an issue building the conda-package-handling dependency. See the
prior discussion for the details.

So I haven't even reached to the issues you've hinted at on GitHub.
Don't know if they still apply with my patch, we can only figure it out
one step at a time :p

--
Pierre Neidhardt
-----BEGIN PGP SIGNATURE-----

iQEzBAEBCAAdFiEEUPM+LlsMPZAEJKvom9z0l6S7zH8FAl0CnsUACgkQm9z0l6S7
zH8qswf+M7djthSKbJWJeOjxlDv6Rz2VngQDYTkQ0RNGThstQdijFwB97sn1tAwp
yWnxwL+YGJlmspSCjJ3zcAyI/TyWVPZ0bUCmYgKW2404miYL9c/VGnNMP2WpY8bi
GWHtVvSPaJQCaO0ePAOwlFF3pSnWvuCklUmrmI3imkWoleQqCdMScmWQ0RcMYSnj
UcPm/xVYksH1dVYeOSpqIDNjESSW2yDC03IMyPZmCoksFv4D9fIHlL4IPjdnAvbH
mHjrNOt7iEvSnhxx18o/wtf90g6IgrZdVonBG5BMtCytVnFGkqZs4u9WQvNL8o8S
F7941dzmg3DQcvCmsjA/P4sIMqnDaw==
=8LUa
-----END PGP SIGNATURE-----

R
R
Ricardo Wurmus wrote on 13 Aug 2020 00:07
Update Conda
(address . 36026-done@debbugs.gnu.org)(name . Pierre Neidhardt)(address . mail@ambrevar.xyz)
875z9n4ixq.fsf@elephly.net
Commit 638ef1e81d upgrades Conda to 4.8.3. This makes these patches
obsolete.

What a nightmare this was!

--
Ricardo
Closed
?