[PATCH] bootloader: extlinux: support for optional FDTDIR

There are situations that u-boot doesn't have to load from the device

tree. some provide the device tree using a vendor bootloader (like what

raspberry-pi does) or with an external bootloader that chainloads the

u-boot (what Asahi does for m1n1 bootloader).

Unfortunately we couldn't find any reliable document to enforce u-boot

to pass the device tree via `extlinux.conf`, however during our tests,

we found that removing the `FDTDIR` line from the `extlinux.conf` tend

us to do so.

There is also no reliable way to guess if u-boot bootloader should load

device tree or not on a specific hardware. in addition, there are

hardware that can be booted with both firmware device tree on some

kernels and with special device tree on other (modified) kernels.

so we propose the following patch to define an optional parameter in

`bootloader` record, called `ignore-fdtdir?` which by default is set to

`#f` to keep the current behavior unchanged. if this paramter is set to

`#t`, the `FDTDIR` line will be discarded from the `extlinux.conf` and

u-boot doesn't load the device tree automatically.

This patch was tested on a Raspberry PI and in a VM. It is proven to

preserve the old behavior. We’d love to hear your feedback and hope we

can support more arm hardware in the future.

Regards,

Reza

--

Reza Alizadeh Majd

PantherX Team

---

gnu/bootloader.scm | 5 ++++-

gnu/bootloader/extlinux.scm | 12 ++++++++++--

2 files changed, 14 insertions(+), 3 deletions(-)

On 09-08-2022 12:27, Reza Alizadeh Majd wrote:

PantherX claiming it's the upstream of Guix (see tooltip of 'operating

system' on the home page), but that's incorrect, it's the other way around.

It is also implies that most of it is developed by PantherX people, but

IIUC most of it is based on Guix System.

The website also hides well that it's mostly Guix (e.g.: on the home

page it's only mentioned in the tooltip).

The About page mentions both "Less waste" and "other tech should follow

the foot-steps [of Bitcoin]", this seems contradictory.

It also claims privacy, but the website includes stuff from an external

service (dkkma.com), and apparently it's a tracker.

Greetings,

Maxime.

On 09-08-2022 12:27, Reza Alizadeh Majd wrote:

The VM thing sounds like a good system test, add it to gnu/tests/ so we

can verify it and avoid breaking things with future changes.

Greetings,

Maxime.

Maxime, how is your information about pantherx related to the patch? This patch in question makes guix bootloader config more flexible with the perspective of supporting additional hardware. It doesn’t rely on any component of pantherx, thus I am not sure it’s the right place to discuss that.

As for the tests, I ran the image on a non-guix arm machine and I doubt it’s possible or makes sense to run a VM during tests phase. That would require much effort and probably hardware support for virtualization (that’s not present on many relevant machines). I believe a simple test like checking if the line is present in a newly generated extlinux.conf would have more sense.

Sincerely,

Pavel

P. S. Just to clarify: patch was written by Reza, I was the one who took part in testing it.

On 10-08-2022 16:37, Pavel Shlyak wrote:

The patch appears to be by PantherX (see: the @panterx.org mail address,

the use of "we" and the corporate phrasing). The mail asks for feedback,

so I give feedback. It appears that PantherX will interact with Guix in

the future; PantherX making false or misleading claims makes me less

interested in reviewing patches, so this seems important feedback to give.

I could of course out-of-the-blue send a mail to whatever PantherX main

e-mail address is, but I'm not a customer of PantherX, so that would

seem to be a pointless endeavor to me -- if it's anything like other

companies, it will just be moved into the trash.  I believe that

directly responding to the apparent employee of PantherX (and at the

same time writing some more technical comments) has higher chances of

success.

Summarised: this seems a good place to make such comments and I'm not

aware of any better places.

Greetings,

Maxime.

Hello, Maxime

Firstly, we always welcome feedback and we’re glad to answer your questions if you contact us directly. Here are the options: https://www.pantherx.org/contact/ https://www.pantherx.org/contact/ . That might be a better place for it to avoid making other devs who’re subscribed to the mailing list read about things they’re not really into.

Secondly, we’ve already taken some steps to correct the issues you’ve mentioned on our website. I hope it will be clear and polished by the time we publish the first PantherX release.

Finally, I’m personally glad and thankful that you’ve found time to inspect our website. Even I, as a member of the team, haven’t looked at it that closely :)

P. S. Reza’s working on the tests now. It will just take some time before he sends the following patch here.

Sincerely,

Pavel

Hello,

The bootloader record is supposed to be bootloader agnostic. This fdtdir

naming thing seems to be extlinux specific.

Should we maybe rename this field "device-tree-support?"

Thanks,

Mathieu

Maxime Devos 写道：

For the record: they did when Maxime posted this, but at least

that tooltip has since been corrected and now reads:

‘PantherX is a downstream distribution of GNU Guix and

uses the Linux Kernel’

Keep on auditing everything,

T G-R

Sorry for the late response, I was a little busy to apply the suggested

changes.

On Wed, 10 Aug 2022 11:32:38 +0200

Maxime Devos <maximedevos@telenet.be> wrote:

I implemented the `gnu/tests/bootloader.scm` tests related to the

changes we are proposing.

also about other remarks you mentioned me directly, I applied them on

my recent patch.

No, that's not possible. As I mentioned in the initial message, some

hardware may or may not require it depending on the kernel.

On Mon, 15 Aug 2022 11:27:19 +0200

Mathieu Othacehe <othacehe@gnu.org> wrote:

in the recent patch I renamed the `ignore-fdtdir?` parameter to the

`device-tree-support?`

Regards,

Reza

--

Reza Alizadeh Majd

PantherX Team

It seems that I forgot to update some parts of the commit message, I

just submitted the edited one.

--

Reza Alizadeh Majd

PantherX Team

On 16-08-2022 19:08, Reza Alizadeh Majd wrote:

My question has a part 'depending on the hardware', so possibly the

relevant code could check what the hardware is.  Likewise, the code

could check the kernel version.  More generally, when something can be

decided manually, it can often be detected automatically with some

work.  I'm not seeing any impossibility here.

Also, again, why are you submitting this work-around when it appears to

be simply a kernel bug that needs a kernel package to be updated and

maybe a devicetree fix to be backported? As written in a previous response:

Greetings,

Maxime.

Dear Maxime,

I do not think I can agree that «a relevant code could check what the hardware is» as it’s not hardware-defined, but also user-defined. It’s not just about detection, it’s also about choice. The same system definition I have on RPI4, for example, can boot both with kernel-provided FDT (loaded with uboot) and bootloader-provided FDT (loaded with RPI bootloader). There are also a lot of different devices out there on the market and there’s no common way to know how OS definition is written for it. If you have ideas - you’re welcome to propose your way to do it.

It does not appear to «be simply a kernel bug» to me. Kernel does not pass configuration to the bootloader and we’re configuring a bootloader entry here. It’s simple as that: bootloader entry may include or not include that line and that’s user-defined.

Of course, you’re always welcome to suggest your ideas how the automation you like to have could be implemented. Also, please, keep in mind this change doesn’t change anything on current boards/systems and most users won’t even notice this change is merged.

Sincerely,

Pavel Shlyak

On 20-08-2022 12:15, Pavel Shlyak wrote:

If the user really wants to choose a different DT, they can customise

their kernel by overriding the sourcce.

If the bootloader DT is more precise than the kernel DT (or the kernel

DT is missing), why not submit the bootloader DT to the kernel? Then

everyone can benefit, not only people using the 'RPI bootloader.'

Between an inferior and a superior DT, I do not see the point of

providing an option in Guix for selecting the inferior one.

Likewise, if they are equivalent, I don't see the point either.

You write that the system definition can both boot with the

kernel-provided FDT and bootloader FDT, then why are you writing this

patch if things work?

The kernel has multiple DTs. I assume that, somehow, the kernel can

figure out which one.

If you must go for this work-around, you could try porting the logic

that the kernel uses to figure out the right DT, and extend it for the

device that required the patch.

AFAIK, device tree information is used by the kernel, not the

bootloader. AFAIK, at most the bootloader passes a DT to the kernel.  We

could just not support overriding the DT in Guix in the bootloader

entry? I don't see the point if updating the DT in the kernel appears to

be sufficient.

Greetings,

Maxime.

Yes, unless it’s generated by bootloader.

Because it passes board-specific parameters. We cannot submit DTs for all board revisions, memory sizes etc.

They are not

It can boot on RPI4b, but not on RPI3b+ or Compute Module 4

DTs are loaded by the bootloader. Kernel cannot figure anything out.

No, kernel does not include this logic

Uboot uses DT on some platforms

I hope dynamic DT with some data that only bootloader can know is sufficient for you. Again, this is how things work on Raspberry and some other boards on any distro. We don’t support that - we don’t support these devices. I personally don’t loose much as we can apply this patch directly on pantherx channel, making pantherx richer in device support. However, I do not quite like the idea of me answering «Install PantherX» to the people who cannot get GUIX on their devices.

I would be also happy if someone more competent on the topic joined this discussion.

On 22-08-2022 12:52, Pavel Shlyak wrote:

Could you point me at the documentation or code that claims or does

that? I am not finding any evidence that device trees are generated at boot.

If the bootloader can, surely the kernel can.

I believe the kernel folks will appreciate a patch fixing the DT for

RPI3b+ and Compute Module 4.

DTs are a kernel thing, e.g. the Linux documentation

https://www.kernel.org/doc/html/latest/devicetree/usage-model.html

mentions DT, also, Linux.  I could not find any information on

bootloaders loading DTs.

The page

https://www.kernel.org/doc/html/latest/devicetree/usage-model.html

mentions various properties for specifying the model in the DT info, it

has a section '2.2 Platform Identification' on how Linux decides which

one is the right one. Plenty of logic there.

OK. Is the board you are trying to support one of them, and does for

that case the pre-patch behaviour suffice there (Linux will load its own

DT later anyway?).

This response makes me wonder where the boot failed -- did it fail in

the bootloader, or in the kernel startup?

It is neither sufficient nor insufficient for me -- it is you that is

adding support for some boards, not me, GRUB+x86_64 works just nice here.

Besides, the bootloader/kernel distinction is just a matter of

convention, bootloaders don't magically have access to more information

than kernels. Anything a bootloader can determine, a kernel can as well,

and vice-versa, they are both just software running on a CPU and various

associated hardware.

That's what I thought the patch was for -- adding support for some

devices, turning the "it's not supported" into an "it's supported".

Moving support from the bootloader to the kernel would accomplish that

as well. Also, ad populum.

If you don't want to support new platforms, that's fine, but why are you

sending a patch then?

My point is that supporting more devices would be nice, but this patch

isn't the way to do it.

Additionally, the proper capitalisation is Guix, GUIX is another thing.

Greetings,

Maxime.

Google «device tree raspberry bootloader», 1st result https://forums.raspberrypi.com/viewtopic.php?t=329799

Bootloader runs on GPU on Raspberry. It cannot run kernel.

Also, check m1n1 on Apple. It has docs.

And for other devices that behave the same way? You’re literally promoting making GUIX not bootable on all devices alike.

Because you didn’t search for it. Google «device tree raspberry bootloader», the first link is about bootloader forming the device tree https://forums.raspberrypi.com/viewtopic.php?t=329799.Google «uboot device tree» https://u-boot.readthedocs.io/en/latest/usage/fdt_overlays.html to know how uboot manipulates them.

Moreover, Raspberry PI uboot uses DTB to boot on the board as in https://patchwork.ozlabs.org/project/uboot/patch/20191106144104.28177-1-matthias.bgg@kernel.org/(Instead of using the embedded DTB as done in RPi3 we use the devicetree provided by the firmware.)

They do, if they are run on a separate core on the SOC that linux or arm core has no access to. Check https://github.com/christinaa/rpi-open-firmware

Well, there is no other way to support devices that require DTB not to be loaded with uboot. The solutions you suggest are not possible.

Moreover, keep in mind FDTDIR is not in the http://www.freedesktop.org/wiki/Specifications/BootLoaderSpec/specification and making is permanent we basically violate it.

Since we’ve not come to any understanding here, I kindly invite Vagrant and Tobias to join the discussion. They seem to be familiar with the relevant parts of GUIX.

Sincerely,

Pavel

On 22-08-2022 21:19, Pavel Shlyak wrote:

That web page does not claim that anywhere.

I have never claimed that the GPU can run the kernel.

What does m1n1 have to do with anything here? m1n1 isn't extlinux and

isn't packaged in Guix.

Bootloaders running on the GPU is something I'm not used to at all, it's

not something I had expected, see later.

I literally never wrote such a thing. In what sentences did I promote that?

Even if you meant 'implied' instead of 'literally', then that still

doesn't make sense to me; I'm running Guix System, it's in my own

interest to keep it bootable on my device.

I did search for it, figuring out an _appropriate_ query and finding

relevant results is another matter.

No. It has monopoly and privacy problems.

Generating the DT is a different matter from loading the DT.  It's also

about firmware, not the bootloader.

These overlays look rather manual, to be done by the user for individual

models, I don't see the relevancy.

Going by the mention of 'defconfig' and 'arch/arm' and 'configs', this

appears to be a patch to Linux, not uboot. As such, it appears that the

device tree information is used by Linux here, there is no information

there on whether it is used by U-Boot.

That's a setup I would not have expected.

AFAIK nothing is stopping Linux from sending some code to the separate

core to figure out the relevant information and sending it back to

Linux. But given the unusual setup, I would consider it plausible that

Linux people want to delegate such responsibility to the bootloader.

Was this (i.e. "they are run on a separate core on the SOC ...") the

case for the Raspberry device you are trying to support?

If so, figuring out the appropriate options to let U-boot pass the

device tree to Linux seems reasonable to me.

And is this a bad thing, and if so, perhaps FDTDIR could be added to the

specification? Guix being in violation of some specification is not in

itself a bug, for example the store model of Guix is not 'Linux Standard

Base'.

It might be a bad thing, but there's a step missing in your argument.

See the 'If so, figuring out the appropriate options ...' above.

Also, again, it's Guix, not GUIX. GUIX is a Microsoft thing.

Greetings,

Maxime

Here's a rough list of the DT changes performed by the firmware, besides merging the obvious dtoverlays and dtparams:

It chainloads uboot

I cannot agree.

https://github.com/u-boot/u-boot/blob/master/arch/arm/mach-bcm283x/Kconfig https://github.com/u-boot/u-boot/blob/master/arch/arm/mach-bcm283x/Kconfig

https://github.com/u-boot/u-boot/blob/master/configs/A10-OLinuXino-Lime_defconfig https://github.com/u-boot/u-boot/blob/master/configs/A10-OLinuXino-Lime_defconfig

I don’t think so. Keep in mind the source code in Guix is named extlinux.scm and the file is named extlinux.conf and extlinux doesn’t support FDTDIR

Of course it passes device tree. The problem in question is the source of that device tree: in my case, it should not be loaded from a file.

On 2022-08-22, Pavel Shlyak wrote:

I don't know exactly where in the code off the top of my head, but

u-boot definitely has support for modifying the device-tree it passes to

the kernel, and in some cases (e.g. memory layout, framebuffer)

*requires* modifying the device-tree that is passed to the kernel. FWIW,

I say this as someone who's maintained u-boot in debian (and to some

degree guix) for years...

Also, for some virtual platforms (qemu-riscv64?) the device-tree is

created on-the-fly when bootstrapping the virtual machine; in these

cases you have to rely on the device-tree passed via the boot "firmware"

as you shouldn't need to recompile the kernel just to pass a different

set of qemu arguments!

There are cases where the kernel cannot possibly track the combinatorial

explosion of potential add-on hardware modules (rpi hats, beagleboard

capes, etc.) that need to be represented in the device-tree in order to

work, sometimes before the kernel is even booted. There is some support

for device-tree overlays both in-kernel and in u-boot (and plausibly

other bootloaders) kernel to handle this, but in many cases, a simpler

and more reliable method is to provide a single custom device-tree.

Yes and no. Device-trees are used in, by and for both bootloaders and

kernels alike. There is long-standing debate around device-tree should

being treated as a boot firmware thing rather than a kernel thing. There

are some pretty simple questions:

You don't see ACPI tables for every supported x86 board in the kernel,

why should boards using device-tree be any different?

Why is the linux kernel the source of device-trees for, say, the Hurd,

*BSD, etc.?

So yes, it is de-facto where most device-trees come from at the end of

the day, but ... that doesn't necessarily mean it *should* be.

Well, that's what the FDTDIR and FDT settings for extlinux do; it tells

the bootloader (e.g. u-boot) to load a device-tree. Grub also has a

devicetree option, though no correlary to FDTDIR. If grub isn't passed a

devicetree argument, it passes whatever devicetree that was passed from

the boot firmware, as I understand it.

Hrm, that's unfortunate. I daresay supporting FDTDIR is a good thing, as

it allows you to use the same boot media to boot multiple different

devices, presuming they all have a .dtb present for the relevent boards.

Booting arm systems have come a long way from you need a specifically

compiled kernel for every different board, but it is still a horrible

mess.

Sometimes things are implemented in kernel, sometimes in the bootloader,

sometimes in the boot firmware, with possibly 3-4 different boot

firmware layers... and exactly where may be specific to a particular

board. There are arguments that any particular thing might be better

implemented at any of those layers, and maybe there is a convincing

reason to move something from one layer to another...

At the end of the day, the reality is that board support for some

platforms (e.g. arm* and possibly riscv*) is a mess of inconsistancy and

the usually you need to support multiple different ways of doing

seemingly the same things.

I think this tends to be hidden from view for x86 as quirks are just

worked around in the kernel, whereas with some platforms where more of

the boot firmware is free software it is possible to fix it in a more

appropriate layer... or whatever layer the someone just happens to get

it to work at first!

If adding an option to drop the FDTDIR extlinux configuration allows

booting more platforms, I see no fundamental reason why it is wrong, as

long as it doesn't break existing platforms... the implementation

details, I'll leave to people more savvy with scheme. :)

live well,

vagrant

Hey,

OK, so I read the long thread. There's still something unclear to me. If

FDTDIR is not passed, where is located the device tree that is loaded?

Directly in the /boot partition?

It looks like the corresponding code is in the label_boot function of

the u-boot/boot/pxe_utils.c file. I quickly read this part hoping to

find a way to define a priority between device trees located in the

FDTDIR directory and device trees installed elsewhere.

You need to document this new field in the "Bootloader Configuration"

documentation section.

This will test for regressions on an x86_64-linux machine that will

probably never use this FDTDIR thing. As those tests are expensive to

run an maintain we can probably remove the test.

Thanks,

Mathieu

Hello, Mathieu

Thanks for joining us here!

Firstly, I kindly ask everyone in this thread to CC me as emails from this thread are not delivered to me for some reason I don’t really understand. I get emails from all other threads but this one.

Secondly, to reply you question "where is located the device tree that is loaded?", I would like to refer uboot source code comments, as I have not seen this point documented anywhere else. Check boot/pxe_utils.c

/*

* fdt usage is optional:

* It handles the following scenarios.

*

* Scenario 1: If fdt_addr_r specified and "fdt" or "fdtdir" label is

* defined in pxe file, retrieve fdt blob from server. Pass fdt_addr_r to

* bootm, and adjust argc appropriately.

*

* If retrieve fails and no exact fdt blob is specified in pxe file with

* "fdt" label, try Scenario 2.

*

* Scenario 2: If there is an fdt_addr specified, pass it along to

* bootm, and adjust argc appropriately.

*

* Scenario 3: If there is an fdtcontroladdr specified, pass it along to

* bootm, and adjust argc appropriately.

*

* Scenario 4: fdt blob is not available.

*/

In other words, if there’s no fdtdir, uboot just passes the fdt_addr to bootm command, that means it passes the pointer to device tree loaded by firmware (or previous bootloader) to the kernel. If fdt_addr is also missing, it passes fdtcontroladdr to the kernel. If all of the previous scenarios don’t work, it passes no FDT and kernel boots without device tree, that is absolutely normal on some devices, check x86.

As for the tests and documentation, I think I should leave these questions to Reza, as they have more experience and deeper understanding.

Sincerely,

Pavel

Hi Mathieu,

On Thu, 25 Aug 2022 19:35:45 +0200

Mathieu Othacehe <othacehe@gnu.org> wrote:

I just checked the "Bootloader Configuration" section. it describes the

"bootloader-configuration" record itself, but the proposed patch adds

the "device-tree-supports?" field to the "bootloader" record.

unfortunately I couldn't find the section describing the "bootloader"

record fields. so I added the documentations as a note for the

"bootloader" field of "bootloader-configuration" record.

OK, I removed the test from recent patch.

Best,

Reza

--

Reza Alizadeh Majd

PantherX Team

Hey Reza,

Thanks for the updated version!

About that, any reason not to have this "device-tree-supports?" field in

the <bootloader-configuration> record?

The <bootloader> record is about how to install the bootloader while

<bootloader-configuration> is about its configuration. So maybe it would

be a better fit?

Good. Let me know what you think about the proposal and we should be

good to proceed.

Mathieu

Hi Mathieu,

On Sun, 28 Aug 2022 17:49:36 +0200

Mathieu Othacehe <othacehe@gnu.org> wrote:

I wanted to limit my patch to affect as minimum sections as possible.

so I added the field to the <bootloader> record.

I'm agree with your proposal, since the removal of FDTDIR is more of a

configuration for an existing bootloader.

I moved this option to the <bootloader-configuration> and submit a new

patch. it would be great to have your feedback whenever you had time.

Regards,

Reza

--

Reza Alizadeh Majd

PantherX Team

Hey Pavel,

Thanks for the clarification,

Mathieu

Hello Reza,

I pushed this patch with a few modifications.

Thanks,

Mathieu