RAID installation script with ‘mdadm’ no longer works

DoneSubmitted by Ludovic Courtès.
Details
4 participants
  • Gábor Boskovits
  • Ludovic Courtès
  • Tobias Geerinckx-Rice
  • Vagrant Cascadian
Owner
unassigned
Severity
important
L
L
Ludovic Courtès wrote on 6 Nov 2019 11:13
(address . bug-Guix@gnu.org)
87sgn18g92.fsf@inria.fr
Hello,
Looks like our RAID installation method no longer works, as can be seenat https://ci.guix.gnu.org/build/1906208/details:
Toggle snippet (19 lines)+ guix --versionguix (GNU Guix) c4de60ac3c6aa5b46519011af89988215c347e9eCopyright (C) 2019 the Guix authorsLicense GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>This is free software: you are free to change and redistribute it.There is NO WARRANTY, to the extent permitted by law.+ export GUIX_BUILD_OPTIONS=--no-grafts+ GUIX_BUILD_OPTIONS=--no-grafts+ parted --script /dev/vdb mklabel gpt mkpart primary ext2 1M 3M mkpart primary ext2 3M 600M mkpart primary ext2 600M 1200M set 1 boot on set 1 bios_grub on+ mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2 /dev/vdb2 /dev/vdb3mdadm: chunk size defaults to 512Kmdadm: Defaulting to version 1.2 metadata[ 13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with default_layout setting[ 13.894691] md/raid0: please set raid0.default_layout to 1 or 2[ 13.896000] md: pers->run() failed ...mdadm: RUN_ARRAY failed: Unknown error 524[ 13.901603] md: md0 stopped.
Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?
We should then update (gnu tests install) and the manual accordingly.
Thanks,Ludo’.
G
G
Gábor Boskovits wrote on 6 Nov 2019 12:07
Re: bug#38086: RAID installation script with ‘mdad m’ no longer works
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . bug-Guix@gnu.org)
CAE4v=pizjYjTMAYCXFYmacK7wiFZ5DzqZ86ArHn0LNfr2Di8Tw@mail.gmail.com
Hello Ludo,

Ludovic Courtès <ludo@gnu.org> ezt írta (időpont: 2019. nov. 6., Sze,11:14):
Toggle quote (33 lines)> Hello,>> Looks like our RAID installation method no longer works, as can be seen> at <https://ci.guix.gnu.org/build/1906208/details>:>> --8<---------------cut here---------------start------------->8---> + guix --version> guix (GNU Guix) c4de60ac3c6aa5b46519011af89988215c347e9e> Copyright (C) 2019 the Guix authors> License GPLv3+: GNU GPL version 3 or later <> http://gnu.org/licenses/gpl.html>> This is free software: you are free to change and redistribute it.> There is NO WARRANTY, to the extent permitted by law.> + export GUIX_BUILD_OPTIONS=--no-grafts> + GUIX_BUILD_OPTIONS=--no-grafts> + parted --script /dev/vdb mklabel gpt mkpart primary ext2 1M 3M mkpart> primary ext2 3M 600M mkpart primary ext2 600M 1200M set 1 boot on set 1> bios_grub on> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2> /dev/vdb2 /dev/vdb3> mdadm: chunk size defaults to 512K> mdadm: Defaulting to version 1.2 metadata> [ 13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with> default_layout setting> [ 13.894691] md/raid0: please set raid0.default_layout to 1 or 2> [ 13.896000] md: pers->run() failed ...> mdadm: RUN_ARRAY failed: Unknown error 524> [ 13.901603] md: md0 stopped.> --8<---------------cut here---------------end--------------->8--->> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?>
On kernel 5.3.4 and above theraid0.default_layout=2 kernel boot paramter should be set. We shouldgenerate our grub configuration accordingly.
See this for reference:https://blog.icod.de/2019/10/10/caution-kernel-5-3-4-and-raid0-default_layout/


Toggle quote (5 lines)> We should then update (gnu tests install) and the manual accordingly.>> Thanks,> Ludo’.>
Best regards,g_bor
-- OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21
Attachment: file
L
L
Ludovic Courtès wrote on 12 Nov 2019 00:28
Re: bug#38086: RAID installation script with ‘mda dm’ no longer works
(name . Gábor Boskovits)(address . boskovits@gmail.com)(address . 38086@debbugs.gnu.org)
877e46m1qd.fsf@gnu.org
Hi Gábor,
Gábor Boskovits <boskovits@gmail.com> skribis:
Toggle quote (19 lines)>> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2>> /dev/vdb2 /dev/vdb3>> mdadm: chunk size defaults to 512K>> mdadm: Defaulting to version 1.2 metadata>> [ 13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with>> default_layout setting>> [ 13.894691] md/raid0: please set raid0.default_layout to 1 or 2>> [ 13.896000] md: pers->run() failed ...>> mdadm: RUN_ARRAY failed: Unknown error 524>> [ 13.901603] md: md0 stopped.>> --8<---------------cut here---------------end--------------->8--->>>> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?>>>> On kernel 5.3.4 and above the> raid0.default_layout=2 kernel boot paramter should be set. We should> generate our grub configuration accordingly.
That’s part of the solution, thank you!
With the patch below, the “raid-root-os” test successfully installs thesystem to a RAID0 device, but then that system fails to boot with:
Toggle snippet (8 lines)Booting from Hard Disk...GRUB loading.Welcome to GRUB!
error: invalid arch-independent ELF magic.Entering rescue mode...
(It sits there forever.)
Are we missing something in ‘grub.cfg’? If so, I wonder if that problemarose with the upgrade in commit069ab3bbfde704760acaca20dff8a29d167c6be5.
Thoughts?
Ludo’.
Toggle diff (27 lines)diff --git a/gnu/tests/install.scm b/gnu/tests/install.scmindex 22c9554705..5e421f7c54 100644--- a/gnu/tests/install.scm+++ b/gnu/tests/install.scm@@ -543,7 +543,8 @@ where /gnu lives on a separate partition.") (bootloader (bootloader-configuration (bootloader grub-bootloader) (target "/dev/vdb")))- (kernel-arguments '("console=ttyS0"))+ (kernel-arguments '("console=ttyS0"+ "raid0.default_layout=2")) ;; Add a kernel module for RAID-0 (aka. "stripe"). (initrd-modules (cons "raid0" %base-initrd-modules))@@ -578,9 +579,11 @@ export GUIX_BUILD_OPTIONS=--no-grafts parted --script /dev/vdb mklabel gpt \\ mkpart primary ext2 1M 3M \\ mkpart primary ext2 3M 600M \\- mkpart primary ext2 600M 1200M \\+ mkpart primary ext2 600M 1.4G \\ set 1 boot on \\ set 1 bios_grub on+modprobe raid0+echo 1 > /sys/module/raid0/parameters/default_layout mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2 \\ /dev/vdb2 /dev/vdb3 mkfs.ext4 -L root-fs /dev/md0
L
L
Ludovic Courtès wrote on 12 Nov 2019 00:29
control message for bug #38086
(address . control@debbugs.gnu.org)
875zjqm1q1.fsf@gnu.org
severity 38086 importantquit
L
L
Ludovic Courtès wrote on 22 Nov 2019 19:30
Re: bug#38086: RAID installation script with ‘mda dm’ no longer works
(name . Gábor Boskovits)(address . boskovits@gmail.com)(address . 38086@debbugs.gnu.org)
87zhgn7oh8.fsf@gnu.org
An update: this is the last known good test:
https://berlin.guixsd.org/build/1793057/details
and this is the first known-bad (‘mdadm’ failing with “cannot assemblemulti-zone RAID0 without default_layout setting”):
https://berlin.guixsd.org/build/1795351/details
We have to resort to an ungly hack to get the evaluation number andcorresponding commit of each build because they aren’t accessible overHTTP (which is unfortunate!):
Toggle snippet (10 lines)sqlite> select * from builds where rowid = 1793057;/gnu/store/618hm2w0clcrxz16yww846mgqdc1l4s0-raid-root-os.drv|7863|test.raid-root-os.i686-linux|i686-linux|raid-root-os||0|1570439988|1570459635|1570459744sqlite> select * from checkouts where evaluation = 7863;guix-master|7b6a8e23b0de18262a42e44432f955517d71d796|7863|guix|/gnu/store/7sd2lwj83n6kyn66p9bdgs5yvzqnl539-guix-7b6a8e2sqlite> select * from builds where rowid = 1795351;/gnu/store/qskl45gw9y9hd8qp7s5451d53pvpc60q-raid-root-os.drv|7867|test.raid-root-os.i686-linux|i686-linux|raid-root-os||2|1570440409|0|1570457622sqlite> select * from checkouts where evaluation = 7867;guix-master|7d82e920717f08bceb42bb570d786dff233171e1|7867|guix|/gnu/store/b2cq9zhdsz4qri2xkg3rgwmyri0wyxxb-guix-7d82e92
So the commit that introduced the change of behavior of ‘mdadm’ is:
Toggle snippet (6 lines)commit 7d82e920717f08bceb42bb570d786dff233171e1Date: Sun Oct 6 06:07:15 2019 +0000
gnu: linux-libre: Update to 5.3.4.
And indeed that brings us back to:
https://blog.icod.de/2019/10/10/caution-kernel-5-3-4-and-raid0-default_layout/
Hmm alright, nothing new here. Oh well!
Ludo’.
V
V
Vagrant Cascadian wrote on 17 Jan 2020 23:42
(address . 38086@debbugs.gnu.org)
87r1zx902a.fsf@yucca
On 2019-11-12, Ludovic Courtès wrote:
Toggle quote (21 lines)> Gábor Boskovits <boskovits@gmail.com> skribis:>>>> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2>>> /dev/vdb2 /dev/vdb3>>> mdadm: chunk size defaults to 512K>>> mdadm: Defaulting to version 1.2 metadata>>> [ 13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with>>> default_layout setting>>> [ 13.894691] md/raid0: please set raid0.default_layout to 1 or 2>>> [ 13.896000] md: pers->run() failed ...>>> mdadm: RUN_ARRAY failed: Unknown error 524>>> [ 13.901603] md: md0 stopped.>>> --8<---------------cut here---------------end--------------->8--->>>>>> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?>>>>>>> On kernel 5.3.4 and above the>> raid0.default_layout=2 kernel boot paramter should be set. We should>> generate our grub configuration accordingly.
So, this might be sort of a tangent, but I'm wondering why you'retesting raid0 (striping, for performance+capacity at risk of data loss)instead of raid1 (mirroring, for redundancy, fast reads, slow writes,half capacity of storage), or another raid level with more disks (raid5,raid6, raid10). raid1 would be the simplest to switch the code to, sinceit uses only two disks.

The issue triggering this bug might be a non-issue on other raid levelsthat in my mind might make more sense for rootfs. Or maybe people haveuse-casese for rootfs on raid0 that I'm too uncreative to think of? :)

live well, vagrant
-----BEGIN PGP SIGNATURE-----
iHUEARYKAB0WIQRlgHNhO/zFx+LkXUXcUY/If5cWqgUCXiI4bQAKCRDcUY/If5cWqg6tAQDnN10yIpC58DKd2sEfmoNP7PAa65iyqNRVlZY6Kns59gD+NAGhQ31f2ohLa0EAa2MrsGqlFocle4vjk1sW9T88IQA==bcKG-----END PGP SIGNATURE-----
G
G
Gábor Boskovits wrote on 18 Jan 2020 14:29
Re: bug#38086: RAID installation script with ‘mdad m’ no longer works
(name . Vagrant Cascadian)(address . vagrant@debian.org)
CAE4v=piMdWV6-aG7CDULJwjjkj=tbBvEjaOF4k8Gm6vaznh-fw@mail.gmail.com
Vagrant Cascadian <vagrant@debian.org> ezt írta (időpont: 2020. jan. 17.,Pén 23:42):
Toggle quote (35 lines)> On 2019-11-12, Ludovic Courtès wrote:> > Gábor Boskovits <boskovits@gmail.com> skribis:> >> >>> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2> >>> /dev/vdb2 /dev/vdb3> >>> mdadm: chunk size defaults to 512K> >>> mdadm: Defaulting to version 1.2 metadata> >>> [ 13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with> >>> default_layout setting> >>> [ 13.894691] md/raid0: please set raid0.default_layout to 1 or 2> >>> [ 13.896000] md: pers->run() failed ...> >>> mdadm: RUN_ARRAY failed: Unknown error 524> >>> [ 13.901603] md: md0 stopped.> >>> --8<---------------cut here---------------end--------------->8---> >>>> >>> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?> >>>> >>> >> On kernel 5.3.4 and above the> >> raid0.default_layout=2 kernel boot paramter should be set. We should> >> generate our grub configuration accordingly.>> So, this might be sort of a tangent, but I'm wondering why you're> testing raid0 (striping, for performance+capacity at risk of data loss)> instead of raid1 (mirroring, for redundancy, fast reads, slow writes,> half capacity of storage), or another raid level with more disks (raid5,> raid6, raid10). raid1 would be the simplest to switch the code to, since> it uses only two disks.>>> The issue triggering this bug might be a non-issue on other raid levels> that in my mind might make more sense for rootfs. Or maybe people have> use-casese for rootfs on raid0 that I'm too uncreative to think of? :)>
I often see raid 10 as root. I believe it might make sense to test thatsetup.
Toggle quote (5 lines)>>> live well,> vagrant>
Attachment: file
L
L
Ludovic Courtès wrote on 18 Jan 2020 22:46
Re: bug#38086: RAID installation script with ‘mda dm’ no longer works
(name . Vagrant Cascadian)(address . vagrant@debian.org)
87lfq4v3nb.fsf@gnu.org
Hi!
Vagrant Cascadian <vagrant@debian.org> skribis:
Toggle quote (7 lines)> So, this might be sort of a tangent, but I'm wondering why you're> testing raid0 (striping, for performance+capacity at risk of data loss)> instead of raid1 (mirroring, for redundancy, fast reads, slow writes,> half capacity of storage), or another raid level with more disks (raid5,> raid6, raid10). raid1 would be the simplest to switch the code to, since> it uses only two disks.
Good point! I guess it would make sense to test RAID1, indeed.
I gave it a shot with the patch below. Problem is that installationseemingly hangs here:
Toggle snippet (11 lines)+ parted --script /dev/vdb mklabel gpt mkpart primary ext2 1M 3M mkpart primary ext2 3M 1.4G mkpart primary ext2 1.4G 2.8G set 1 boot on set 1 bios_grub on+ mdadm --create /dev/md0 --verbose --level=mirror --raid-devices=2 /dev/vdb2 /dev/vdb3mdadm: Note: this array has metadata at the start and may not be suitable as a boot device. If you plan to store '/boot' on this device please ensure that your boot-loader understands md/v1.x metadata, or use --metadata=0.90mdadm: size set to 1361920Kmdadm: largest drive (/dev/vdb3) exceeds size (1361920K) by more than 1%
As you can see, it’s attempting to make a RAID1 device out of twopartitions (not two disks), which makes no sense in the real world, butis easier to handle here. So I wonder if this is what’s causing it tohang…
Thoughts?
Ludo’.
Toggle diff (39 lines)diff --git a/gnu/tests/install.scm b/gnu/tests/install.scmindex 8842d48df8..12e6eb26df 100644--- a/gnu/tests/install.scm+++ b/gnu/tests/install.scm@@ -546,8 +546,8 @@ where /gnu lives on a separate partition.") (target "/dev/vdb"))) (kernel-arguments '("console=ttyS0")) - ;; Add a kernel module for RAID-0 (aka. "stripe").- (initrd-modules (cons "raid0" %base-initrd-modules))+ ;; Add a kernel module for RAID-1 (aka. "mirror").+ (initrd-modules (cons "raid1" %base-initrd-modules)) (mapped-devices (list (mapped-device (source (list "/dev/vda2" "/dev/vda3"))@@ -578,11 +578,11 @@ guix --version export GUIX_BUILD_OPTIONS=--no-grafts parted --script /dev/vdb mklabel gpt \\ mkpart primary ext2 1M 3M \\- mkpart primary ext2 3M 600M \\- mkpart primary ext2 600M 1200M \\+ mkpart primary ext2 3M 1.4G \\+ mkpart primary ext2 1.4G 2.8G \\ set 1 boot on \\ set 1 bios_grub on-mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2 \\+mdadm --create /dev/md0 --verbose --level=mirror --raid-devices=2 \\ /dev/vdb2 /dev/vdb3 mkfs.ext4 -L root-fs /dev/md0 mount /dev/md0 /mnt@@ -605,7 +605,7 @@ by 'mdadm'.") %raid-root-os-source #:script %raid-root-installation-script- #:target-size (* 1300 MiB)))+ #:target-size (* 2800 MiB))) (command (qemu-command/writable-image image))) (run-basic-test %raid-root-os `(,@command) "raid-root-os")))))
T
T
Tobias Geerinckx-Rice wrote on 18 Jan 2020 23:03
(name . Ludovic Courtès)(address . ludo@gnu.org)
87h80sbexq.fsf@nckx
Ludovic Courtès 写道:
Toggle quote (8 lines)> As you can see, it’s attempting to make a RAID1 device out of > two> partitions (not two disks), which makes no sense in the real > world, but> is easier to handle here. So I wonder if this is what’s causing > it to> hang…
It's just waiting for input:
$ # dd & losetup magic, where loop0 is 20% larger than loop1 $ sudo mdadm --create /dev/md0 --verbose --level=mirror --raid-devices=2 /dev/loop{0,1} mdadm: Note: this array has metadata at the start and may not be suitable as a boot device. If you plan to store '/boot' on this device please ensure that your boot-loader understands md/v1.x metadata, or use --metadata=0.90 mdadm: size set to 101376K mdadm: largest drive (/dev/loop1) exceeds size (101376K) by more than 1% Continue creating array?
Adding --force does not avoid this.
I recommend tweaking the partition table to make both members equal, but a ‘yes|’ also works if you're in a hurry ;-)
Kind regards,
T G-R
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEfo+u0AlEeO9y5k0W2Imw8BjFSTwFAl4jgKEACgkQ2Imw8BjFSTyvYw/+M3VniY/3EnWypbLcneJqx+II8GDA3HwmhvI9q+4PWqpK6ZAMzxTEt1CsQvCiXgmsX3EnjyRTtEaGil/o16QsBJdDeF4HGK//MStM2OEQY7Tix273X0DGJvKKkB1RhOhZjsXEUtnndOfGpNCDQClW4UD6Ht4WaxWstSWzdk5Cx4Gpy5+B+wdf2zYpgxDzCAHb6yrvk43y7MP0KjxowmJlIEKWr0Js2j7sZUr34Fmj1jSVQF4tUVezY6rX6WtjEdPc0Flx6smiq6uN7YDup+ntGmDL/3lqHtUL+t2hwQ8hmjAR4b8A5hFhL8CxlTZxzEp2a43nkR7mN9+OF0qZYSVkrJ57/y0GQAkt4FkSAYYHXPkyax4S9v/v9Xyf7OV06DTBLDOjY26EweViKcGLqUqQ5lJ0jY1qK7oI/uV3r+viq7XohA9Xa43SzosdiSegtOuyVtBk9Nc5RaBoOTjB2lws022rtucKunrKeM/WZKRHihZOOLgKrg3OgvBofMK6FPNArvm5YK8+fDdGcOJXKCictWIYPWMqZtS8lAnameN2GvyulzO+4oOFhtUo0ZohpXeg/Zhb2SwhddzsRzXeQ8O2h3XcwAXj3kO2zDD4E2CZ+bGq5koNtk3VkvMAmUB8BUk0dSrGDtyoripoje+bl0SmmX2ZORepCNG/2I5jrPl8xEg==x/c4-----END PGP SIGNATURE-----
L
L
Ludovic Courtès wrote on 19 Jan 2020 23:13
(name . Tobias Geerinckx-Rice)(address . me@tobias.gr)
87a76jqelv.fsf@gnu.org
Hi Tobias!
Tobias Geerinckx-Rice <me@tobias.gr> skribis:
Toggle quote (15 lines)> It's just waiting for input:>> $ # dd & losetup magic, where loop0 is 20% larger than loop1> $ sudo mdadm --create /dev/md0 --verbose --level=mirror> --raid-devices=2 /dev/loop{0,1}> mdadm: Note: this array has metadata at the start and> may not be suitable as a boot device. If you plan to> store '/boot' on this device please ensure that> your boot-loader understands md/v1.x metadata, or use> --metadata=0.90> mdadm: size set to 101376K> mdadm: largest drive (/dev/loop1) exceeds size (101376K) by more than> 1%> Continue creating array?
D’oh, I hadn’t seen that message.
Toggle quote (5 lines)> Adding --force does not avoid this.>> I recommend tweaking the partition table to make both members equal,> but a ‘yes|’ also works if you're in a hurry ;-)
“yes|” works like a charm, I went this that.
Pushed in commit 3adf320e44e54017a67f219ce9667a379c393dad, thank you!
Ludo’.
Closed
T
T
Tobias Geerinckx-Rice wrote on 19 Jan 2020 23:31
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 38086@debbugs.gnu.org)
8736cbaxiq.fsf@nckx
Ludo',
Ludovic Courtès 写道:
Toggle quote (4 lines)>> Continue creating array?>> D’oh, I hadn’t seen that message.
I doubt it was there for you to see in Guix's output. Things not ending with a newline tend to get lost easily due to line-buffering. Probably not worth worrying about.
Toggle quote (2 lines)> “yes|” works like a charm, I went this that.
‘Beautiful’,
T G-R
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEfo+u0AlEeO9y5k0W2Imw8BjFSTwFAl4k2M0ACgkQ2Imw8BjFSTweMRAAjp0HGqwVygVdx9JWuoS4lI3hoPZDOq1HSstInOaMbY+J8TAl7qccqOhy15ziTZfOiYjYsLI0de8hDEA71LyDCtVvRQPCJrh0MCxrlkMmMTflua0NRh9sfMqE4S58W9ftl2cBg+zI3BY5j31pdQkOfi1soYD3VZNII/a8JAt5z2mpLr6NyvyjxVR6ZhsbzKVYEyfVDnGhCqbv9IguNdjnDCOfy18T37tJ/S+FCR6C1nZnB7HvJTyu59CKNWb5bFP5oj/IlGdBr1OxJCrUX2ks+VOjnUeTFX1WZxdRji96phHfDHR05dcfx502oOt4JEVCQ9sfYJebHSHxEHRoIgFIq1CX3L114wRaURY0m2EwuksOZofJExeNtY3o6weXO1R5e3ZBzTN2DCpQ8gALttj4KnPy24DGXL2JW7Vq7wuaoraeRLuj6eP1xkgl1ik7gq46DDs2BH1AI0blkb79Kz1vfrbEuzpxQzOYiQpW8bgWO9A3VNaqAVEerLfNS6Dkx6XH91/MSTkZSD81jTOAH4Tbz/pDzUWtCrS7BMta9IfUrvNs4raEMyNk0gOisfeebdAwQWCT//RjY4lIQ2r982Bkv/reC6xu8nuPDVg+qe0o7Ub78vgo1f247+2qIUmv003X3DBQI07Yuc7pRfpzcExcko0PhBkxhdXMLeb0ZdaRFgo==jCNI-----END PGP SIGNATURE-----
?
Your comment

This issue is archived.

To comment on this conversation send email to 38086@debbugs.gnu.org