[PATCH 0/6] Update Scrapy.

  • Done
  • quality assurance status badge
Details
2 participants
  • Christopher Baines
  • Vinicius Monego
Owner
unassigned
Submitted by
Vinicius Monego
Severity
normal
V
V
Vinicius Monego wrote on 6 Aug 2023 17:31
(address . guix-patches@gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153159.285234-1-monego@posteo.net
Vinicius Monego (6):
gnu: python-w3lib: Update to 2.1.2.
gnu: python-cssselect: Update to 1.2.0.
gnu: python-parsel: Update to 1.8.1.
gnu: python-itemadapter: Update to 0.8.0.
gnu: python-itemloaders: Update to 1.1.0.
gnu: python-scrapy: Update to 2.10.0.

gnu/local.mk | 1 -
.../python-w3lib-fix-test-failure.patch | 60 --------------
gnu/packages/python-web.scm | 80 +++++++------------
gnu/packages/python-xyz.scm | 13 +--
4 files changed, 38 insertions(+), 116 deletions(-)
delete mode 100644 gnu/packages/patches/python-w3lib-fix-test-failure.patch

--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 1/6] gnu: python-w3lib: Update to 2.1.2.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-1-monego@posteo.net
* gnu/packages/python-web.scm (python-w3lib): Update to 2.1.2.
[source]: Remove patch.
[build-system]: Use pyproject-build-system.
[arguments]: Do not override the check phase.
[native-inputs]: Remove python-six.
* gnu/packages/patches/python-w3lib-fix-test-failure.patch: Delete file.
* gnu/local.mk (dist_patch_DATA): Remove it.
---
gnu/local.mk | 1 -
.../python-w3lib-fix-test-failure.patch | 60 -------------------
gnu/packages/python-web.scm | 16 ++---
3 files changed, 4 insertions(+), 73 deletions(-)
delete mode 100644 gnu/packages/patches/python-w3lib-fix-test-failure.patch

Toggle diff (60 lines)
diff --git a/gnu/local.mk b/gnu/local.mk
index 77707127a7..e091b310c2 100644
--- a/gnu/local.mk
+++ b/gnu/local.mk
@@ -1730,7 +1730,6 @@ dist_patch_DATA = \
%D%/packages/patches/python-shiboken-2-compat.patch \
%D%/packages/patches/python-telingo-fix-comparison.patch \
%D%/packages/patches/python-typeguard-python3.10.patch \
- %D%/packages/patches/python-w3lib-fix-test-failure.patch \
%D%/packages/patches/python-wxwidgets-type-errors.patch \
%D%/packages/patches/quodlibet-fix-invalid-glob.patch \
%D%/packages/patches/quodlibet-fix-mtime-tests.patch \
diff --git a/gnu/packages/patches/python-w3lib-fix-test-failure.patch b/gnu/packages/patches/python-w3lib-fix-test-failure.patch
deleted file mode 100644
index f38fc749df..0000000000
--- a/gnu/packages/patches/python-w3lib-fix-test-failure.patch
+++ /dev/null
@@ -1,60 +0,0 @@
-From fae6cc40e112cd13697cb0e8d79976f32c72491d Mon Sep 17 00:00:00 2001
-From: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
-Date: Wed, 10 Mar 2021 12:31:05 -0300
-Subject: [PATCH] [CI] Mark single add_or_replace_parameter test as xfail
-
----
-This patch is based on upstream commit
-fae6cc40e112cd13697cb0e8d79976f32c72491d which is part of
-https://github.com/scrapy/w3lib/pull/166.
-
- .gitignore | 3 ++-
- tests/test_url.py | 25 ++++++++++++++-----------
- 2 files changed, 16 insertions(+), 12 deletions(-)
-
-diff --git a/tests/test_url.py b/tests/test_url.py
-index 8b07c00..0f7458e 100644
---- a/tests/test_url.py
-+++ b/tests/test_url.py
-@@ -1,12 +1,14 @@
--# -*- coding: utf-8 -*-
- from __future__ import absolute_import
- import os
- import unittest
-+
-+import pytest
-+from six.moves.urllib.parse import urlparse
-+
- from w3lib.url import (is_url, safe_url_string, safe_download_url,
- url_query_parameter, add_or_replace_parameter, url_query_cleaner,
- file_uri_to_path, parse_data_uri, path_to_file_uri, any_to_uri,
- urljoin_rfc, canonicalize_url, parse_url, add_or_replace_parameters)
--from six.moves.urllib.parse import urlparse
-
-
- class UrlTests(unittest.TestCase):
-@@ -310,10 +311,6 @@ def test_add_or_replace_parameter(self):
- self.assertEqual(add_or_replace_parameter(url, 'arg3', 'nv3'),
- 'http://domain/test?arg1=v1&arg2=v2&arg3=nv3')
-
-- url = 'http://domain/test?arg1=v1;arg2=v2'
-- self.assertEqual(add_or_replace_parameter(url, 'arg1', 'v3'),
-- 'http://domain/test?arg1=v3&arg2=v2')
--
- self.assertEqual(add_or_replace_parameter("http://domain/moreInfo.asp?prodID=",'prodID', '20'),
-@@ -338,6 +335,13 @@ def test_add_or_replace_parameter(self):
- self.assertEqual(add_or_replace_parameter(url, 'arg1', 'v3'),
-
-+ @pytest.mark.xfail(reason="https://github.com/scrapy/w3lib/issues/164")
-+ def test_add_or_replace_parameter_fail(self):
-+ self.assertEqual(
-+ add_or_replace_parameter('http://domain/test?arg1=v1;arg2=v2','arg1', 'v3'),
-+ )
-+
- def test_add_or_replace_parameters(self):
- self.assertEqual(add_or_replace_parameters(url, {'arg': 'v'}),
Toggle diff (36 lines)
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 17c9563cfd..24071a8553 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -7752,25 +7752,17 @@ (define-public python-smart-open
(define-public python-w3lib
(package
(name "python-w3lib")
- (version "1.22.0")
+ (version "2.1.2")
(source
(origin
(method url-fetch)
(uri (pypi-uri "w3lib" version))
- (patches (search-patches "python-w3lib-fix-test-failure.patch"))
(sha256
(base32
- "1pv02lvvmgz2qb61vz1jkjc04fgm4hpfvaj5zm4i3mjp64hd1mha"))))
- (build-system python-build-system)
- (arguments
- `(#:phases
- (modify-phases %standard-phases
- (replace 'check
- (lambda* (#:key tests? #:allow-other-keys)
- (when tests?
- (invoke "pytest")))))))
+ "1cd4b3w5g3pfccsg79kjj27fwi216ip927rjq7isp8pfjzlp8nzd"))))
+ (build-system pyproject-build-system)
(native-inputs
- (list python-pytest python-six))
+ (list python-pytest))
(home-page "https://github.com/scrapy/w3lib")
(synopsis "Python library of web-related functions")
(description
--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 2/6] gnu: python-cssselect: Update to 1.2.0.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-2-monego@posteo.net
* gnu/packages/python-web.scm (python-cssselect): Update to 1.2.0.
[build-system]: Use pyproject-build-system.
[arguments]: Do not override the check phase.
---
gnu/packages/python-web.scm | 11 +++--------
1 file changed, 3 insertions(+), 8 deletions(-)

Toggle diff (31 lines)
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 24071a8553..9ecd9f35a3 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -2095,7 +2095,7 @@ (define-public python-css-parser
(define-public python-cssselect
(package
(name "python-cssselect")
- (version "1.1.0")
+ (version "1.2.0")
(source (origin
;; The PyPI release does not contain tests.
(method git-fetch)
@@ -2105,13 +2105,8 @@ (define-public python-cssselect
(file-name (git-file-name name version))
(sha256
(base32
- "0xslrnhbrmgakp4xg6k26qffay3kqffp3a2z2sk27c65rwxa79kc"))))
- (build-system python-build-system)
- (arguments
- `(#:phases (modify-phases %standard-phases
- (replace 'check
- (lambda _
- (invoke "pytest" "-vv"))))))
+ "1x4nrvb1p1byi1whmspik7lbh303akdlh762dayfxam3hycsh5kk"))))
+ (build-system pyproject-build-system)
(native-inputs
(list python-lxml python-pytest))
(home-page "https://github.com/scrapy/cssselect")
--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 3/6] gnu: python-parsel: Update to 1.8.1.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-3-monego@posteo.net
* gnu/packages/python-web.scm (python-parsel): Update to 1.8.1.
[build-system]: Use pyproject-build-system.
[propagated-inputs]: Add python-jmespath, python-typing-extensions. Remove
python-six.
[native-inputs]: Add python-psutil. Remove python-pytest-runner.
---
gnu/packages/python-web.scm | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)

Toggle diff (34 lines)
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 9ecd9f35a3..70a1ab082f 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -8039,21 +8039,22 @@ (define-public python-protego
(define-public python-parsel
(package
(name "python-parsel")
- (version "1.6.0")
+ (version "1.8.1")
(source
(origin
(method url-fetch)
(uri (pypi-uri "parsel" version))
(sha256
- (base32 "0yawf9r3r863lwxj0n89i7h3n8xjbsl5b7n6xg76r68scl5yzvvh"))))
- (build-system python-build-system)
+ (base32 "0f8yh30y3961a7kqwcnp4j3s7044ilakykiavc0skwdkr5l8xwmg"))))
+ (build-system pyproject-build-system)
(propagated-inputs
(list python-cssselect
+ python-jmespath
python-lxml
- python-six
+ python-typing-extensions
python-w3lib))
(native-inputs
- (list python-pytest python-pytest-runner))
+ (list python-psutil python-pytest))
(home-page "https://github.com/scrapy/parsel")
(synopsis "Extract data from HTML and XML using XPath and CSS selectors")
(description "Parsel is a library to extract and remove data from
--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 4/6] gnu: python-itemadapter: Update to 0.8.0.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-4-monego@posteo.net
* gnu/packages/python-xyz.scm (python-itemadapter): Update to 0.8.0.
---
gnu/packages/python-xyz.scm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

Toggle diff (22 lines)
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index 884c37e975..e6cbe7d333 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -32504,13 +32504,13 @@ (define-public python-posix-ipc
(define-public python-itemadapter
(package
(name "python-itemadapter")
- (version "0.5.0")
+ (version "0.8.0")
(source
(origin
(method url-fetch)
(uri (pypi-uri "itemadapter" version))
(sha256
- (base32 "083wp3h2brh8x19jbdr8rz3biqwp3jlqd0rfzcyrjyhssffsgdh5"))))
+ (base32 "1aa898gjgwy3axxfrgsh4kdvhp6n6wz3ccdishq0gh8azf2q8xbp"))))
(build-system python-build-system)
(home-page "https://github.com/scrapy/itemadapter")
(synopsis "Common interface for data container classes")
--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 5/6] gnu: python-itemloaders: Update to 1.1.0.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-5-monego@posteo.net
* gnu/packages/python-xyz.scm (python-itemloaders): Update to 1.1.0.
[build-system]: Use pyproject-build-system.
[native-inputs]: Add python-pytest.
---
gnu/packages/python-xyz.scm | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)

Toggle diff (28 lines)
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index e6cbe7d333..c7bce1b135 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -32534,16 +32534,17 @@ (define-public python-itemadapter
(define-public python-itemloaders
(package
(name "python-itemloaders")
- (version "1.0.4")
+ (version "1.1.0")
(source
(origin
(method url-fetch)
(uri (pypi-uri "itemloaders" version))
(sha256
- (base32 "15hc78h90qhwass1bga1c3xar2dd6j8sxg61zg6jvh74lf6csxqj"))))
- (build-system python-build-system)
+ (base32 "0j2aw4ipalj208594x80blpgkh1i63gqqa4nb67b823av9hirn11"))))
+ (build-system pyproject-build-system)
+ (native-inputs (list python-pytest))
(propagated-inputs
- (list python-itemadapter python-jmespath python-parsel python-w3lib))
+ (list python-itemadapter python-jmespath python-parsel python-w3lib))
(home-page "https://github.com/scrapy/itemloaders")
(synopsis "Base library for scrapy's ItemLoader")
(description "Itemloaders is a library that helps you collect data
--
2.34.1
V
V
Vinicius Monego wrote on 6 Aug 2023 17:33
[PATCH 6/6] gnu: python-scrapy: Update to 2.10.0.
(address . 65114@debbugs.gnu.org)(name . Vinicius Monego)(address . monego@posteo.net)
20230806153317.285639-6-monego@posteo.net
* gnu/packages/python-web.scm (python-scrapy): Update to 2.10.0.
[build-system]: Use pyproject-build-system.
[arguments]: Use G-Expressions. Do not override the check phase, skip tests
in #:test-flags instead. Run tests sequentially and skip fewer tests.
---
gnu/packages/python-web.scm | 42 ++++++++++++++++---------------------
1 file changed, 18 insertions(+), 24 deletions(-)

Toggle diff (61 lines)
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index 70a1ab082f..c12740b587 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -8065,36 +8065,30 @@ (define-public python-parsel
(define-public python-scrapy
(package
(name "python-scrapy")
- (version "2.7.1")
+ (version "2.10.0")
(source
(origin
(method url-fetch)
(uri (pypi-uri "Scrapy" version))
(sha256
- (base32 "0kpi3hg2ycs6s8cg41r2zc1axd0rpnps8bnzg7wisjyjaf1l1yih"))))
- (build-system python-build-system)
+ (base32 "0fdf1kx39zy7b0v5gkwwqvm58ww63z5bvm78yjymhfn0r0zs65jf"))))
+ (build-system pyproject-build-system)
(arguments
- `(#:phases
- (modify-phases %standard-phases
- (replace 'check
- (lambda* (#:key tests? #:allow-other-keys)
- (when tests?
- (invoke "pytest"
- "-n" (number->string (parallel-job-count))
- ;; These tests fail when run in parallel (see:
- ;; https://github.com/scrapy/scrapy/issues/5502).
- "--ignore" "tests/test_engine.py"
- "--ignore" "tests/test_engine_stop_download_bytes.py"
- "--ignore" "tests/test_engine_stop_download_headers.py"
- ;; This test require network access.
- "--ignore" "tests/test_command_check.py"
- "-k"
- (string-append
- ;; The followin tests fail for unknown reasons.
- "not test_server_set_cookie_domain_suffix_public_private"
- " and not test_user_set_cookie_domain_suffix_public_private"
- " and not test_pformat")
- "tests")))))))
+ (list #:test-flags
+ ;; Tests fail with DNS lookup or need a display.
+ #~(list "-k" (string-append
+ "not test_SCRAPY_CHECK_set"
+ " and not "
+ (string-join
+ (list "test_check_all_default_contracts"
+ "test_check_cb_kwargs_contract"
+ "test_check_returns_items_contract"
+ "test_check_returns_requests_contract"
+ "test_check_scrapes_contract"
+ "test_pformat"
+ "test_pformat_old_windows"
+ "test_pformat_windows")
+ " and not ")))))
(propagated-inputs
(list python-botocore ; Optional: For S3FeedStorage class.
python-cryptography
--
2.34.1
C
C
Christopher Baines wrote on 27 Aug 2023 20:14
Re: [bug#65114] [PATCH 0/6] Update Scrapy.
(name . Vinicius Monego)(address . monego@posteo.net)
87h6okz70v.fsf@cbaines.net
Vinicius Monego <monego@posteo.net> writes:

Toggle quote (15 lines)
> Vinicius Monego (6):
> gnu: python-w3lib: Update to 2.1.2.
> gnu: python-cssselect: Update to 1.2.0.
> gnu: python-parsel: Update to 1.8.1.
> gnu: python-itemadapter: Update to 0.8.0.
> gnu: python-itemloaders: Update to 1.1.0.
> gnu: python-scrapy: Update to 2.10.0.
>
> gnu/local.mk | 1 -
> .../python-w3lib-fix-test-failure.patch | 60 --------------
> gnu/packages/python-web.scm | 80 +++++++------------
> gnu/packages/python-xyz.scm | 13 +--
> 4 files changed, 38 insertions(+), 116 deletions(-)
> delete mode 100644 gnu/packages/patches/python-w3lib-fix-test-failure.patch

These changes look good to me +1

Thanks,

Chris
-----BEGIN PGP SIGNATURE-----

iQKlBAEBCgCPFiEEPonu50WOcg2XVOCyXiijOwuE9XcFAmTrkrBfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldDNF
ODlFRUU3NDU4RTcyMEQ5NzU0RTBCMjVFMjhBMzNCMEI4NEY1NzcRHG1haWxAY2Jh
aW5lcy5uZXQACgkQXiijOwuE9XdYKQ/7B6MDvufDf29Ihn44D3wUgz6R/SN97/a4
cosAFyyW+9ZQ++P3BvgjceIG2b0y5KMeR1fH1TEOJfN/gF/ZCwlWuY9KVBVWtBIl
whyhAjHysxlA4ys79FOY152OEnQUvSlg0prCBLShVfgbyp0xmV/DpXZ9DW4OEOCH
XVjfiYvtJfP0ykeZi3ugkwATJQxWOF0pO4FN0hI55Oemz90TI3/WRwE02QgHCfKk
6tGCO/8flpQ2NtVD8owYM+5QKWkn5a/t5vJZRHFEgheYSFXUDsP1dCLXDFO2H2Yt
r9Cxsx+41Vo7WGKtsSrRVRPr4WsxnjwdD7wo5W35L5qPW8DpXmzOvm8Wlc0SnVmw
Jx5Cx40SzIfcFcRNDRfcuSVIO0VdkjduEF3ezPmA6HJ/e6dbmoyAdN7mnmsR2rwK
iaViiyhvjeHi6/GjPVEtGmOHF9pxd8z6DInELiJNu0vJ2FbAuPV9VPjKmzoQfEsA
HBW6lLPhkp7XAjdhODrQ7qj6MDEkpA7YgUPmmClVKLIIPA0tcO0sD/ktlsqvHPOq
pfrzVOyD77cKiy3GNTCEx0A0qXv+JlfoxbJiswLOnm975P9iXTMmkT6NE4KLEu12
cFtFIKM8i/Q54HxB9esl2DOph6D5kqryzi5Old1bR+QwWRq4DrNCzX1o4osLr3+l
pFT/GEN/IaQ=
=k50B
-----END PGP SIGNATURE-----

V
V
Vinicius Monego wrote on 2 Sep 2023 19:59
(name . Christopher Baines)(address . mail@cbaines.net)(address . 65114-done@debbugs.gnu.org)
8f624fb52caf9169f71db63a228555a87304ceda.camel@posteo.net
Em dom, 2023-08-27 às 19:14 +0100, Christopher Baines escreveu:


[...]

Toggle quote (3 lines)
> These changes look good to me +1
>

[...]

Applied with a minor update to Scrapy, thanks.

Vinicius
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 65114@debbugs.gnu.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 65114
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch