RAID installation script with ‘mdadm’ no longer works

  • Done
  • quality assurance status badge
Details
4 participants
  • Gábor Boskovits
  • Ludovic Courtès
  • Tobias Geerinckx-Rice
  • Vagrant Cascadian
Owner
unassigned
Submitted by
Ludovic Courtès
Severity
important
L
L
Ludovic Courtès wrote on 6 Nov 2019 11:13
(address . bug-Guix@gnu.org)
87sgn18g92.fsf@inria.fr
Hello,

Looks like our RAID installation method no longer works, as can be seen

Toggle snippet (19 lines)
+ guix --version
guix (GNU Guix) c4de60ac3c6aa5b46519011af89988215c347e9e
Copyright (C) 2019 the Guix authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ export GUIX_BUILD_OPTIONS=--no-grafts
+ GUIX_BUILD_OPTIONS=--no-grafts
+ parted --script /dev/vdb mklabel gpt mkpart primary ext2 1M 3M mkpart primary ext2 3M 600M mkpart primary ext2 600M 1200M set 1 boot on set 1 bios_grub on
+ mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2 /dev/vdb2 /dev/vdb3
mdadm: chunk size defaults to 512K
mdadm: Defaulting to version 1.2 metadata
[ 13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with default_layout setting
[ 13.894691] md/raid0: please set raid0.default_layout to 1 or 2
[ 13.896000] md: pers->run() failed ...
mdadm: RUN_ARRAY failed: Unknown error 524
[ 13.901603] md: md0 stopped.

Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?

We should then update (gnu tests install) and the manual accordingly.

Thanks,
Ludo’.
G
G
Gábor Boskovits wrote on 6 Nov 2019 12:07
Re: bug#38086: RAID installation script with ‘mdad m’ no longer works
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . bug-Guix@gnu.org)
CAE4v=pizjYjTMAYCXFYmacK7wiFZ5DzqZ86ArHn0LNfr2Di8Tw@mail.gmail.com
Hello Ludo,


Ludovic Courtès <ludo@gnu.org> ezt írta (id?pont: 2019. nov. 6., Sze,
11:14):

Toggle quote (33 lines)
> Hello,
>
> Looks like our RAID installation method no longer works, as can be seen
> at <https://ci.guix.gnu.org/build/1906208/details>:
>
> --8<---------------cut here---------------start------------->8---
> + guix --version
> guix (GNU Guix) c4de60ac3c6aa5b46519011af89988215c347e9e
> Copyright (C) 2019 the Guix authors
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> + export GUIX_BUILD_OPTIONS=--no-grafts
> + GUIX_BUILD_OPTIONS=--no-grafts
> + parted --script /dev/vdb mklabel gpt mkpart primary ext2 1M 3M mkpart
> primary ext2 3M 600M mkpart primary ext2 600M 1200M set 1 boot on set 1
> bios_grub on
> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2
> /dev/vdb2 /dev/vdb3
> mdadm: chunk size defaults to 512K
> mdadm: Defaulting to version 1.2 metadata
> [ 13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with
> default_layout setting
> [ 13.894691] md/raid0: please set raid0.default_layout to 1 or 2
> [ 13.896000] md: pers->run() failed ...
> mdadm: RUN_ARRAY failed: Unknown error 524
> [ 13.901603] md: md0 stopped.
> --8<---------------cut here---------------end--------------->8---
>
> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?
>

On kernel 5.3.4 and above the
raid0.default_layout=2 kernel boot paramter should be set. We should
generate our grub configuration accordingly.

See this for reference:



Toggle quote (5 lines)
> We should then update (gnu tests install) and the manual accordingly.
>
> Thanks,
> Ludo’.
>
Best regards,
g_bor

--
OpenPGP Key Fingerprint: 7988:3B9F:7D6A:4DBF:3719:0367:2506:A96C:CF63:0B21
Attachment: file
L
L
Ludovic Courtès wrote on 12 Nov 2019 00:28
Re: bug#38086: RAID installation script with ‘mda dm’ no longer works
(name . Gábor Boskovits)(address . boskovits@gmail.com)(address . 38086@debbugs.gnu.org)
877e46m1qd.fsf@gnu.org
Hi Gábor,

Gábor Boskovits <boskovits@gmail.com> skribis:

Toggle quote (19 lines)
>> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2
>> /dev/vdb2 /dev/vdb3
>> mdadm: chunk size defaults to 512K
>> mdadm: Defaulting to version 1.2 metadata
>> [ 13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with
>> default_layout setting
>> [ 13.894691] md/raid0: please set raid0.default_layout to 1 or 2
>> [ 13.896000] md: pers->run() failed ...
>> mdadm: RUN_ARRAY failed: Unknown error 524
>> [ 13.901603] md: md0 stopped.
>> --8<---------------cut here---------------end--------------->8---
>>
>> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?
>>
>
> On kernel 5.3.4 and above the
> raid0.default_layout=2 kernel boot paramter should be set. We should
> generate our grub configuration accordingly.

That’s part of the solution, thank you!

With the patch below, the “raid-root-os” test successfully installs the
system to a RAID0 device, but then that system fails to boot with:

Toggle snippet (8 lines)
Booting from Hard Disk...
GRUB loading.
Welcome to GRUB!

error: invalid arch-independent ELF magic.
Entering rescue mode...

(It sits there forever.)

Are we missing something in ‘grub.cfg’? If so, I wonder if that problem
arose with the upgrade in commit
069ab3bbfde704760acaca20dff8a29d167c6be5.

Thoughts?

Ludo’.
Toggle diff (27 lines)
diff --git a/gnu/tests/install.scm b/gnu/tests/install.scm
index 22c9554705..5e421f7c54 100644
--- a/gnu/tests/install.scm
+++ b/gnu/tests/install.scm
@@ -543,7 +543,8 @@ where /gnu lives on a separate partition.")
(bootloader (bootloader-configuration
(bootloader grub-bootloader)
(target "/dev/vdb")))
- (kernel-arguments '("console=ttyS0"))
+ (kernel-arguments '("console=ttyS0"
+ "raid0.default_layout=2"))
;; Add a kernel module for RAID-0 (aka. "stripe").
(initrd-modules (cons "raid0" %base-initrd-modules))
@@ -578,9 +579,11 @@ export GUIX_BUILD_OPTIONS=--no-grafts
parted --script /dev/vdb mklabel gpt \\
mkpart primary ext2 1M 3M \\
mkpart primary ext2 3M 600M \\
- mkpart primary ext2 600M 1200M \\
+ mkpart primary ext2 600M 1.4G \\
set 1 boot on \\
set 1 bios_grub on
+modprobe raid0
+echo 1 > /sys/module/raid0/parameters/default_layout
mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2 \\
/dev/vdb2 /dev/vdb3
mkfs.ext4 -L root-fs /dev/md0
L
L
Ludovic Courtès wrote on 12 Nov 2019 00:29
control message for bug #38086
(address . control@debbugs.gnu.org)
875zjqm1q1.fsf@gnu.org
severity 38086 important
quit
L
L
Ludovic Courtès wrote on 22 Nov 2019 19:30
Re: bug#38086: RAID installation script with ‘mda dm’ no longer works
(name . Gábor Boskovits)(address . boskovits@gmail.com)(address . 38086@debbugs.gnu.org)
87zhgn7oh8.fsf@gnu.org
An update: this is the last known good test:


and this is the first known-bad (‘mdadm’ failing with “cannot assemble
multi-zone RAID0 without default_layout setting”):


We have to resort to an ungly hack to get the evaluation number and
corresponding commit of each build because they aren’t accessible over
HTTP (which is unfortunate!):

Toggle snippet (10 lines)
sqlite> select * from builds where rowid = 1793057;
/gnu/store/618hm2w0clcrxz16yww846mgqdc1l4s0-raid-root-os.drv|7863|test.raid-root-os.i686-linux|i686-linux|raid-root-os||0|1570439988|1570459635|1570459744
sqlite> select * from checkouts where evaluation = 7863;
guix-master|7b6a8e23b0de18262a42e44432f955517d71d796|7863|guix|/gnu/store/7sd2lwj83n6kyn66p9bdgs5yvzqnl539-guix-7b6a8e2
sqlite> select * from builds where rowid = 1795351;
/gnu/store/qskl45gw9y9hd8qp7s5451d53pvpc60q-raid-root-os.drv|7867|test.raid-root-os.i686-linux|i686-linux|raid-root-os||2|1570440409|0|1570457622
sqlite> select * from checkouts where evaluation = 7867;
guix-master|7d82e920717f08bceb42bb570d786dff233171e1|7867|guix|/gnu/store/b2cq9zhdsz4qri2xkg3rgwmyri0wyxxb-guix-7d82e92

So the commit that introduced the change of behavior of ‘mdadm’ is:

Toggle snippet (6 lines)
commit 7d82e920717f08bceb42bb570d786dff233171e1
Date: Sun Oct 6 06:07:15 2019 +0000

gnu: linux-libre: Update to 5.3.4.

And indeed that brings us back to:


Hmm alright, nothing new here. Oh well!

Ludo’.
V
V
Vagrant Cascadian wrote on 17 Jan 2020 23:42
(address . 38086@debbugs.gnu.org)
87r1zx902a.fsf@yucca
On 2019-11-12, Ludovic Courtès wrote:
Toggle quote (21 lines)
> Gábor Boskovits <boskovits@gmail.com> skribis:
>
>>> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2
>>> /dev/vdb2 /dev/vdb3
>>> mdadm: chunk size defaults to 512K
>>> mdadm: Defaulting to version 1.2 metadata
>>> [ 13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with
>>> default_layout setting
>>> [ 13.894691] md/raid0: please set raid0.default_layout to 1 or 2
>>> [ 13.896000] md: pers->run() failed ...
>>> mdadm: RUN_ARRAY failed: Unknown error 524
>>> [ 13.901603] md: md0 stopped.
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?
>>>
>>
>> On kernel 5.3.4 and above the
>> raid0.default_layout=2 kernel boot paramter should be set. We should
>> generate our grub configuration accordingly.

So, this might be sort of a tangent, but I'm wondering why you're
testing raid0 (striping, for performance+capacity at risk of data loss)
instead of raid1 (mirroring, for redundancy, fast reads, slow writes,
half capacity of storage), or another raid level with more disks (raid5,
raid6, raid10). raid1 would be the simplest to switch the code to, since
it uses only two disks.


The issue triggering this bug might be a non-issue on other raid levels
that in my mind might make more sense for rootfs. Or maybe people have
use-casese for rootfs on raid0 that I'm too uncreative to think of? :)


live well,
vagrant
-----BEGIN PGP SIGNATURE-----

iHUEARYKAB0WIQRlgHNhO/zFx+LkXUXcUY/If5cWqgUCXiI4bQAKCRDcUY/If5cW
qg6tAQDnN10yIpC58DKd2sEfmoNP7PAa65iyqNRVlZY6Kns59gD+NAGhQ31f2ohL
a0EAa2MrsGqlFocle4vjk1sW9T88IQA=
=bcKG
-----END PGP SIGNATURE-----

G
G
Gábor Boskovits wrote on 18 Jan 2020 14:29
Re: bug#38086: RAID installation script with ‘mdad m’ no longer works
(name . Vagrant Cascadian)(address . vagrant@debian.org)
CAE4v=piMdWV6-aG7CDULJwjjkj=tbBvEjaOF4k8Gm6vaznh-fw@mail.gmail.com
Vagrant Cascadian <vagrant@debian.org> ezt írta (id?pont: 2020. jan. 17.,
Pén 23:42):

Toggle quote (35 lines)
> On 2019-11-12, Ludovic Courtès wrote:
> > Gábor Boskovits <boskovits@gmail.com> skribis:
> >
> >>> + mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2
> >>> /dev/vdb2 /dev/vdb3
> >>> mdadm: chunk size defaults to 512K
> >>> mdadm: Defaulting to version 1.2 metadata
> >>> [ 13.890586] md/raid0:md0: cannot assemble multi-zone RAID0 with
> >>> default_layout setting
> >>> [ 13.894691] md/raid0: please set raid0.default_layout to 1 or 2
> >>> [ 13.896000] md: pers->run() failed ...
> >>> mdadm: RUN_ARRAY failed: Unknown error 524
> >>> [ 13.901603] md: md0 stopped.
> >>> --8<---------------cut here---------------end--------------->8---
> >>>
> >>> Anyone knows what it takes to “set raid0.default_layout to 1 or 2”?
> >>>
> >>
> >> On kernel 5.3.4 and above the
> >> raid0.default_layout=2 kernel boot paramter should be set. We should
> >> generate our grub configuration accordingly.
>
> So, this might be sort of a tangent, but I'm wondering why you're
> testing raid0 (striping, for performance+capacity at risk of data loss)
> instead of raid1 (mirroring, for redundancy, fast reads, slow writes,
> half capacity of storage), or another raid level with more disks (raid5,
> raid6, raid10). raid1 would be the simplest to switch the code to, since
> it uses only two disks.
>
>
> The issue triggering this bug might be a non-issue on other raid levels
> that in my mind might make more sense for rootfs. Or maybe people have
> use-casese for rootfs on raid0 that I'm too uncreative to think of? :)
>

I often see raid 10 as root. I believe it might make sense to test that
setup.

Toggle quote (5 lines)
>
>
> live well,
> vagrant
>
Attachment: file
L
L
Ludovic Courtès wrote on 18 Jan 2020 22:46
Re: bug#38086: RAID installation script with ‘mda dm’ no longer works
(name . Vagrant Cascadian)(address . vagrant@debian.org)
87lfq4v3nb.fsf@gnu.org
Hi!

Vagrant Cascadian <vagrant@debian.org> skribis:

Toggle quote (7 lines)
> So, this might be sort of a tangent, but I'm wondering why you're
> testing raid0 (striping, for performance+capacity at risk of data loss)
> instead of raid1 (mirroring, for redundancy, fast reads, slow writes,
> half capacity of storage), or another raid level with more disks (raid5,
> raid6, raid10). raid1 would be the simplest to switch the code to, since
> it uses only two disks.

Good point! I guess it would make sense to test RAID1, indeed.

I gave it a shot with the patch below. Problem is that installation
seemingly hangs here:

Toggle snippet (11 lines)
+ parted --script /dev/vdb mklabel gpt mkpart primary ext2 1M 3M mkpart primary ext2 3M 1.4G mkpart primary ext2 1.4G 2.8G set 1 boot on set 1 bios_grub on
+ mdadm --create /dev/md0 --verbose --level=mirror --raid-devices=2 /dev/vdb2 /dev/vdb3
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
mdadm: size set to 1361920K
mdadm: largest drive (/dev/vdb3) exceeds size (1361920K) by more than 1%

As you can see, it’s attempting to make a RAID1 device out of two
partitions (not two disks), which makes no sense in the real world, but
is easier to handle here. So I wonder if this is what’s causing it to
hang…

Thoughts?

Ludo’.
Toggle diff (39 lines)
diff --git a/gnu/tests/install.scm b/gnu/tests/install.scm
index 8842d48df8..12e6eb26df 100644
--- a/gnu/tests/install.scm
+++ b/gnu/tests/install.scm
@@ -546,8 +546,8 @@ where /gnu lives on a separate partition.")
(target "/dev/vdb")))
(kernel-arguments '("console=ttyS0"))
- ;; Add a kernel module for RAID-0 (aka. "stripe").
- (initrd-modules (cons "raid0" %base-initrd-modules))
+ ;; Add a kernel module for RAID-1 (aka. "mirror").
+ (initrd-modules (cons "raid1" %base-initrd-modules))
(mapped-devices (list (mapped-device
(source (list "/dev/vda2" "/dev/vda3"))
@@ -578,11 +578,11 @@ guix --version
export GUIX_BUILD_OPTIONS=--no-grafts
parted --script /dev/vdb mklabel gpt \\
mkpart primary ext2 1M 3M \\
- mkpart primary ext2 3M 600M \\
- mkpart primary ext2 600M 1200M \\
+ mkpart primary ext2 3M 1.4G \\
+ mkpart primary ext2 1.4G 2.8G \\
set 1 boot on \\
set 1 bios_grub on
-mdadm --create /dev/md0 --verbose --level=stripe --raid-devices=2 \\
+mdadm --create /dev/md0 --verbose --level=mirror --raid-devices=2 \\
/dev/vdb2 /dev/vdb3
mkfs.ext4 -L root-fs /dev/md0
mount /dev/md0 /mnt
@@ -605,7 +605,7 @@ by 'mdadm'.")
%raid-root-os-source
#:script
%raid-root-installation-script
- #:target-size (* 1300 MiB)))
+ #:target-size (* 2800 MiB)))
(command (qemu-command/writable-image image)))
(run-basic-test %raid-root-os
`(,@command) "raid-root-os")))))
T
T
Tobias Geerinckx-Rice wrote on 18 Jan 2020 23:03
(name . Ludovic Courtès)(address . ludo@gnu.org)
87h80sbexq.fsf@nckx
Ludovic Courtès ???
Toggle quote (8 lines)
> As you can see, it’s attempting to make a RAID1 device out of
> two
> partitions (not two disks), which makes no sense in the real
> world, but
> is easier to handle here. So I wonder if this is what’s causing
> it to
> hang…

It's just waiting for input:

$ # dd & losetup magic, where loop0 is 20% larger than loop1
$ sudo mdadm --create /dev/md0 --verbose --level=mirror
--raid-devices=2 /dev/loop{0,1}
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
mdadm: size set to 101376K
mdadm: largest drive (/dev/loop1) exceeds size (101376K) by more
than 1%
Continue creating array?

Adding --force does not avoid this.

I recommend tweaking the partition table to make both members
equal, but a ‘yes|’ also works if you're in a hurry ;-)

Kind regards,

T G-R
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEfo+u0AlEeO9y5k0W2Imw8BjFSTwFAl4jgKEACgkQ2Imw8BjF
STyvYw/+M3VniY/3EnWypbLcneJqx+II8GDA3HwmhvI9q+4PWqpK6ZAMzxTEt1Cs
QvCiXgmsX3EnjyRTtEaGil/o16QsBJdDeF4HGK//MStM2OEQY7Tix273X0DGJvKK
kB1RhOhZjsXEUtnndOfGpNCDQClW4UD6Ht4WaxWstSWzdk5Cx4Gpy5+B+wdf2zYp
gxDzCAHb6yrvk43y7MP0KjxowmJlIEKWr0Js2j7sZUr34Fmj1jSVQF4tUVezY6rX
6WtjEdPc0Flx6smiq6uN7YDup+ntGmDL/3lqHtUL+t2hwQ8hmjAR4b8A5hFhL8Cx
lTZxzEp2a43nkR7mN9+OF0qZYSVkrJ57/y0GQAkt4FkSAYYHXPkyax4S9v/v9Xyf
7OV06DTBLDOjY26EweViKcGLqUqQ5lJ0jY1qK7oI/uV3r+viq7XohA9Xa43Szosd
iSegtOuyVtBk9Nc5RaBoOTjB2lws022rtucKunrKeM/WZKRHihZOOLgKrg3OgvBo
fMK6FPNArvm5YK8+fDdGcOJXKCictWIYPWMqZtS8lAnameN2GvyulzO+4oOFhtUo
0ZohpXeg/Zhb2SwhddzsRzXeQ8O2h3XcwAXj3kO2zDD4E2CZ+bGq5koNtk3VkvMA
mUB8BUk0dSrGDtyoripoje+bl0SmmX2ZORepCNG/2I5jrPl8xEg=
=x/c4
-----END PGP SIGNATURE-----

L
L
Ludovic Courtès wrote on 19 Jan 2020 23:13
(name . Tobias Geerinckx-Rice)(address . me@tobias.gr)
87a76jqelv.fsf@gnu.org
Hi Tobias!

Tobias Geerinckx-Rice <me@tobias.gr> skribis:

Toggle quote (15 lines)
> It's just waiting for input:
>
> $ # dd & losetup magic, where loop0 is 20% larger than loop1
> $ sudo mdadm --create /dev/md0 --verbose --level=mirror
> --raid-devices=2 /dev/loop{0,1}
> mdadm: Note: this array has metadata at the start and
> may not be suitable as a boot device. If you plan to
> store '/boot' on this device please ensure that
> your boot-loader understands md/v1.x metadata, or use
> --metadata=0.90
> mdadm: size set to 101376K
> mdadm: largest drive (/dev/loop1) exceeds size (101376K) by more than
> 1%
> Continue creating array?

D’oh, I hadn’t seen that message.

Toggle quote (5 lines)
> Adding --force does not avoid this.
>
> I recommend tweaking the partition table to make both members equal,
> but a ‘yes|’ also works if you're in a hurry ;-)

“yes|” works like a charm, I went this that.

Pushed in commit 3adf320e44e54017a67f219ce9667a379c393dad, thank you!

Ludo’.
Closed
T
T
Tobias Geerinckx-Rice wrote on 19 Jan 2020 23:31
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 38086@debbugs.gnu.org)
8736cbaxiq.fsf@nckx
Ludo',

Ludovic Courtès ???
Toggle quote (4 lines)
>> Continue creating array?
>
> D’oh, I hadn’t seen that message.

I doubt it was there for you to see in Guix's output. Things not
ending with a newline tend to get lost easily due to
line-buffering. Probably not worth worrying about.

Toggle quote (2 lines)
> “yes|” works like a charm, I went this that.

‘Beautiful’,

T G-R
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEfo+u0AlEeO9y5k0W2Imw8BjFSTwFAl4k2M0ACgkQ2Imw8BjF
STweMRAAjp0HGqwVygVdx9JWuoS4lI3hoPZDOq1HSstInOaMbY+J8TAl7qccqOhy
15ziTZfOiYjYsLI0de8hDEA71LyDCtVvRQPCJrh0MCxrlkMmMTflua0NRh9sfMqE
4S58W9ftl2cBg+zI3BY5j31pdQkOfi1soYD3VZNII/a8JAt5z2mpLr6NyvyjxVR6
ZhsbzKVYEyfVDnGhCqbv9IguNdjnDCOfy18T37tJ/S+FCR6C1nZnB7HvJTyu59CK
NWb5bFP5oj/IlGdBr1OxJCrUX2ks+VOjnUeTFX1WZxdRji96phHfDHR05dcfx502
oOt4JEVCQ9sfYJebHSHxEHRoIgFIq1CX3L114wRaURY0m2EwuksOZofJExeNtY3o
6weXO1R5e3ZBzTN2DCpQ8gALttj4KnPy24DGXL2JW7Vq7wuaoraeRLuj6eP1xkgl
1ik7gq46DDs2BH1AI0blkb79Kz1vfrbEuzpxQzOYiQpW8bgWO9A3VNaqAVEerLfN
S6Dkx6XH91/MSTkZSD81jTOAH4Tbz/pDzUWtCrS7BMta9IfUrvNs4raEMyNk0gOi
sfeebdAwQWCT//RjY4lIQ2r982Bkv/reC6xu8nuPDVg+qe0o7Ub78vgo1f247+2q
IUmv003X3DBQI07Yuc7pRfpzcExcko0PhBkxhdXMLeb0ZdaRFgo=
=jCNI
-----END PGP SIGNATURE-----

?