[PATCH core-updates 0/1] Allow patch-and-repack to work with plain files.

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
normal
Merged with
M
M
Maxim Cournoyer wrote on 10 Jan 2021 21:02
(address . guix-patches@gnu.org)(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
20210110200231.24214-1-maxim.cournoyer@gmail.com
While attempting to reproduce the now closed issue
http://issues.guix.gnu.org/30116, I stumbled upon another annoyance, which
is that the patch-and-repack procedure used as part of an origin derivation
didn't support single files, so the following failed, for example:

<#part type="text/plain" filename="/home/maxim/src/guix-core-updates/repro.scm" disposition=inline description="Script exhibiting problem">
<#/part>

The following patch fixes that, allowing to use plain files with any of the
already supported compressors, or non-tarball archived that would include a
single directory (the same convention as used for our tarballs).

Maxim Cournoyer (1):
guix: packages: Allow patch-and-repack to work with plain files.

guix/packages.scm | 96 +++++++++++++++++++++++++++++++---------------
tests/packages.scm | 87 +++++++++++++++++++++++++++++++++++++++--
2 files changed, 149 insertions(+), 34 deletions(-)

--
2.29.2
M
M
Maxim Cournoyer wrote on 10 Jan 2021 23:00
control message for bug #45774
(address . control@debbugs.gnu.org)
87sg784fmn.fsf@gmail.com
forcemerge 45774 45773
quit
M
M
Maxim Cournoyer wrote on 10 Jan 2021 23:07
Re: bug#45773: [PATCH core-updates 0/1] Allow patch-and-repack to work with plain files.
(address . 45773@debbugs.gnu.org)
87o8hw4f9s.fsf@gmail.com
Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (8 lines)
> While attempting to reproduce the now closed issue
> <http://issues.guix.gnu.org/30116>, I stumbled upon another annoyance, which
> is that the patch-and-repack procedure used as part of an origin derivation
> didn't support single files, so the following failed, for example:
>
> <#part type="text/plain" filename="/home/maxim/src/guix-core-updates/repro.scm" disposition=inline description="Script exhibiting problem">
> <#/part>

That didn't work out well :-).

Here's the attachment that I meant to send along the cover letter.
;;; Run script with: ./pre-inst-env guile -e main -s repro.scm (use-modules (gnu packages base) (gnu packages bootstrap) (guix build utils) (guix derivations) (guix download) (guix packages) (guix store)) (define %test-file-uri "https://raw.githubusercontent.com/realgud/realgud/master/realgud/common/bp-image-data.el") (define (main _) (let ((source (origin (method url-fetch) (uri %test-file-uri) (modules '((guix build utils))) (patch-inputs `(("tar" ,%bootstrap-coreutils&co) ("xz" ,%bootstrap-coreutils&co) ("locales" ,glibc-utf8-locales))) (snippet '(begin (with-fluids ((%default-port-encoding "ISO-8859-1") (%default-port-conversion-strategy 'error)) (substitute* "bp-image-data.el" (("something") "something else"))))) (sha256 (base32 "1qpn2zhh2qw579bhgjyxvf670r4kibaxls589hkm2yhwfvsjvs68"))))) (with-store store (build-derivations store (list (run-with-store store (origin->derivation source)))))))
Thanks,

Maxim
L
L
Ludovic Courtès wrote on 14 Jan 2021 18:48
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)
877dof4dgr.fsf_-_@gnu.org
Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (29 lines)
> Before this change, only plain directories, tar or zip archives were supported
> as the source of a package for the GNU build system; anything else would cause
> the unpack phase to fail. Origins relying on snippets would suffer from the
> same problem.
>
> This change adds the support to use files of the following extensions: .gz,
> .Z, .bz2, .lz, and .xz, even when they are not tarballs. Files of unknown
> extensions are treated as uncompressed files and supported as well.
>
> * guix/packages.scm (patch-and-repack): Only add the compressor utility to the
> PATH when the file is compressed. Bind more inputs in the mlet, and use them
> for decompressing single files. Adjust decompression and compression routines.
> [decompression-type]: Return #f when no known compression extension is used.
> [tarball?]: New nested procedure.
> * guix/build/utils.scm (compressor, tarball?): New procedures. Move
> %xz-parallel-args to the new 'compression helpers' section.
> * tests/packages.scm: Add tests. Add missing copyright year for Jan.
> * guix/build/gnu-build-system.scm (first-subdirectory): Return #f when no
> sub-directory was found.
> (unpack): Support more file types, including uncompressed plain files.
> ---
> guix/build/gnu-build-system.scm | 24 ++++++--
> guix/build/utils.scm | 47 ++++++++++-----
> guix/packages.scm | 100 +++++++++++++++++---------------
> guix/tests.scm | 40 ++++++++++++-
> tests/builders.scm | 40 ++++++++++++-
> tests/packages.scm | 69 +++++++++++++++++++++-
> 6 files changed, 247 insertions(+), 73 deletions(-)

How frequent is it for an origin to be a regular file other than an
archive? The underlying question for me is: will this generalization
and increased complexity pay off? WDYT?

There are aspects of the patch that I find welcome regardless, such as
the improved handling of compression helpers.

Thanks,
Ludo’.
M
M
Maxim Cournoyer wrote on 14 Jan 2021 19:43
(name . Ludovic Courtès)(address . ludo@gnu.org)
87k0sfl5pc.fsf@gmail.com
Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (37 lines)
> Hi!
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> Before this change, only plain directories, tar or zip archives were supported
>> as the source of a package for the GNU build system; anything else would cause
>> the unpack phase to fail. Origins relying on snippets would suffer from the
>> same problem.
>>
>> This change adds the support to use files of the following extensions: .gz,
>> .Z, .bz2, .lz, and .xz, even when they are not tarballs. Files of unknown
>> extensions are treated as uncompressed files and supported as well.
>>
>> * guix/packages.scm (patch-and-repack): Only add the compressor utility to the
>> PATH when the file is compressed. Bind more inputs in the mlet, and use them
>> for decompressing single files. Adjust decompression and compression routines.
>> [decompression-type]: Return #f when no known compression extension is used.
>> [tarball?]: New nested procedure.
>> * guix/build/utils.scm (compressor, tarball?): New procedures. Move
>> %xz-parallel-args to the new 'compression helpers' section.
>> * tests/packages.scm: Add tests. Add missing copyright year for Jan.
>> * guix/build/gnu-build-system.scm (first-subdirectory): Return #f when no
>> sub-directory was found.
>> (unpack): Support more file types, including uncompressed plain files.
>> ---
>> guix/build/gnu-build-system.scm | 24 ++++++--
>> guix/build/utils.scm | 47 ++++++++++-----
>> guix/packages.scm | 100 +++++++++++++++++---------------
>> guix/tests.scm | 40 ++++++++++++-
>> tests/builders.scm | 40 ++++++++++++-
>> tests/packages.scm | 69 +++++++++++++++++++++-
>> 6 files changed, 247 insertions(+), 73 deletions(-)
>
> How frequent is it for an origin to be a regular file other than an
> archive? The underlying question for me is: will this generalization
> and increased complexity pay off? WDYT?

I think consistency is the main driver here. The url-fetch method
supports single file sources; it makes sense that the other components
handling sources support it as well. It's hard to judge of the
popularity of such a feature when it's never been available; but some
use cases come to mind such as single Emacs package file. I've made use
of such feature for the new texlive-updmap.cfg definition in

Toggle quote (3 lines)
> There are aspects of the patch that I find welcome regardless, such as
> the improved handling of compression helpers.

Great! Thanks for looking at it.

Maxim
M
M
Maxim Cournoyer wrote on 25 Jan 2021 15:20
control message for bug #45959
(address . control@debbugs.gnu.org)
87ft2p3xpi.fsf@gmail.com
forcemerge 45959 45773
quit
M
M
Maxim Cournoyer wrote on 27 Jan 2021 04:57
Re: bug#45773: [PATCH core-updates 0/1] Allow patch-and-repack to work with plain files.
(name . Ludovic Courtès)(address . ludo@gnu.org)
87wnvzyqu5.fsf_-_@gmail.com
Hi Ludo,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

Toggle quote (49 lines)
> Hi Ludovic,
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Hi!
>>
>> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>>
>>> Before this change, only plain directories, tar or zip archives were supported
>>> as the source of a package for the GNU build system; anything else would cause
>>> the unpack phase to fail. Origins relying on snippets would suffer from the
>>> same problem.
>>>
>>> This change adds the support to use files of the following extensions: .gz,
>>> .Z, .bz2, .lz, and .xz, even when they are not tarballs. Files of unknown
>>> extensions are treated as uncompressed files and supported as well.
>>>
>>> * guix/packages.scm (patch-and-repack): Only add the compressor utility to the
>>> PATH when the file is compressed. Bind more inputs in the mlet, and use them
>>> for decompressing single files. Adjust decompression and compression routines.
>>> [decompression-type]: Return #f when no known compression extension is used.
>>> [tarball?]: New nested procedure.
>>> * guix/build/utils.scm (compressor, tarball?): New procedures. Move
>>> %xz-parallel-args to the new 'compression helpers' section.
>>> * tests/packages.scm: Add tests. Add missing copyright year for Jan.
>>> * guix/build/gnu-build-system.scm (first-subdirectory): Return #f when no
>>> sub-directory was found.
>>> (unpack): Support more file types, including uncompressed plain files.
>>> ---
>>> guix/build/gnu-build-system.scm | 24 ++++++--
>>> guix/build/utils.scm | 47 ++++++++++-----
>>> guix/packages.scm | 100 +++++++++++++++++---------------
>>> guix/tests.scm | 40 ++++++++++++-
>>> tests/builders.scm | 40 ++++++++++++-
>>> tests/packages.scm | 69 +++++++++++++++++++++-
>>> 6 files changed, 247 insertions(+), 73 deletions(-)
>>
>> How frequent is it for an origin to be a regular file other than an
>> archive? The underlying question for me is: will this generalization
>> and increased complexity pay off? WDYT?
>
> I think consistency is the main driver here. The url-fetch method
> supports single file sources; it makes sense that the other components
> handling sources support it as well. It's hard to judge of the
> popularity of such a feature when it's never been available; but some
> use cases come to mind such as single Emacs package file. I've made use
> of such feature for the new texlive-updmap.cfg definition in
> <http://issues.guix.gnu.org/45870>.

I've been building a sizable part of core-updates on top of this change
now for nearly two weeks, and in this time it's already come handy
twice.

I've made sure the tests ran fine and pushed to core-updates as commit
cfcead2e515c0dae02127e5a76496463898be6b6.

Let me know if anything breaks :-).

Maxim
Closed
?