Issue with booting about linux-on-litex-rocket

[matsbror]

So booting from SDcard still doesn't work. It seems not to recognise the boot image.
When I try to boot from serial, it starts but fails with this message:

sL5DdSMmkekro
[LITEX-TERM] Received firmware download request from the device.
[LITEX-TERM] Uploading boot.bin to 0x40000000 (17920552 bytes)...
[LITEX-TERM] Upload calibration... (inter-frame: 10.00us, length: 64)
[LITEX-TERM] Got unexpected response from device 'b'E''

I use this command:

litex_term /dev/ttyUSB1 --serial-boot --kernel boot.bin

In Litex BIOS command line, I can detect the SD-card, but when I try to init it it fails:

litex> sdcard_init
Initialize SDCard... Failed.

This was with a SoC generated with --with-sdcard I am now trying with --with-spi-sdcard

[gsomlo]

On Wed, Dec 14, 2022 at 07:16:42AM -0800, Mats Brorsson wrote: I wonder how the bios worked as it goes through the uart as well.

The bios uses its own internal hard-coded MMIO addresses to interact with the various peripherals. Linux comes with its own drivers, and uses whatever addresses are provided to it via the DTB. On typical hardware, the DTB itself is hard-coded inside the bios and will always provide correct register addresses to Linux. That is not (yet) the case with LiteX. There's a `litex/tools/litex_json2dts_linux.py` helper script that should generate the DTS automatically, but it hasn't yet been adapted to work with Rocket CPUs. The current status-quo is that one would have to double-check if the MMIO addresses generated during bitstream build (`-csr-csv foo.csv`) match those in the DTS file, and update them if necessary. Patching `json2dts` to teach it about Rocket is an item on my to-do list... :)

[gsomlo]

For some reason, your kernel crashes while attempting to service a UART interrupt. I've recently added IRQ support for LiteUART to https://github.com/litex-hub/linux/tree/litex-rebase, and it's been working *better* for me than in polling mode (which used to be the only supported mode of operation). If it's the latest litex-hub/litex-rebase kernel sources you're using, you might try dropping the top-most commit ("serial: liteuart: add irq support"), or commenting out the IRQ configuration from the uart's DTS node: https://github.com/litex-hub/linux-on-litex-rocket/blob/master/conf/nexys4ddr_fpu.dts#L110-L111 to force the uart back into polling-mode.

[gsomlo]

On Mon, Dec 05, 2022 at 08:35:22AM -0800, Mats Brorsson wrote: How can I debug this? I am suspecting a mismatch between the boot image and the core somehow. I have attached the entire boot sequence from power on.

With BBL-based LiteX+Rocket, you're expected to build the initramfs cpio archive into the kernel, rather than load it in as a separate file. The initramfs goes inside vmlinuz, which goes inside bbl, resulting in a single `boot.bin` blob that gets loaded by the LiteX bios (either using tftp, or from the sdcard).

I have set the jumpers for booting from USB/SD correctly. It worked with another litex system with a 32-bit processor.

The jumpers will only help you get the *bitstream* loaded from sdcard. Once that's done (i.e., the fpga becomes a "computer"), the jumpers should no longer matter, it's down to the LiteX built-in bios from that point forward.

[matsbror]

Thanks, so the loading of the bitstream works then. But apparently the BIOS cannot read from the SD/USB. This is the process I would like to debug. Is there a way to do that? I realise now that even though I can load the FPGA bitstream from the USB-stick, the BIOS can probably not read from the USB (but I got the same problem when I bit the boot.bin on the SDcard.

I will try to send the boot.bin over the serial port instead.

[matsbror]

I just found this that you wrote on your web page "Booting from a µSD card (in SPI mode) is also currently supported.". I did not build the system with --with-spi-sdcard so I will try that.

[gsomlo]

On Mon, Dec 05, 2022 at 12:10:30PM -0800, Mats Brorsson wrote: I just found this that you wrote on your web page "Booting from a µSD card (in SPI mode) is also currently supported.". I did not build the system with --with-spi-sdcard so I will try that.

I missed the part where you said "usb stick"... And you're right, once the bitstream is loaded, LiteX bios doesn't know how to handle a usb stick (not sure there's support to boot from a usb drive optionally available). I would recommend `--with-sdcard` as it's a bit faster in Linux (but `spi-sdcard` *should* also work). Don't forget to build a single unified `boot.bin` bundling bbl, linux, and the built-in initrd in a single file. I'm switching to `boot.json` and individually loading opensbi, initrd, and kernel blobs in newer/future iterations, but the currently published rocket/linux/litex instructions using bbl require a single blob. Good luck!

[matsbror]

Ok, I got progress with --with-spi-sdcard

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
             Timeout
Booting from SDCard in SPI-Mode...
Booting from boot.json...
boot.json file not found.
Booting from boot.bin...
Copying boot.bin to 0x80000000 (17920552 bytes)...
[########################################]
Executing booted program at 0x80000000

--============= Liftoff! ===============--

But it is still tuck at this stage and I will rebuild busybox, the kernel and BBL to see if it helps.

[gsomlo]

On Mon, Dec 05, 2022 at 11:42:43PM -0800, Mats Brorsson wrote: sL5DdSMmkekro [LITEX-TERM] Received firmware download request from the device. [LITEX-TERM] Uploading boot.bin to 0x40000000 (17920552 bytes)... [LITEX-TERM] Upload calibration... (inter-frame: 10.00us, length: 64) [LITEX-TERM] Got unexpected response from device 'b'E'' I use this command: litex_term /dev/ttyUSB1 --serial-boot --kernel boot.bin

I don't have any experience loading things over serial. But loading boot.bin to 0x40000000 is wrong, on Rocket it should be 0x80000000.

[gsomlo]

On Tue, Dec 06, 2022 at 12:20:35AM -0800, Mats Brorsson wrote: Ok, I got progress with --with-spi-sdcard [...] Booting from boot.json... boot.json file not found. Booting from boot.bin... Copying boot.bin to 0x80000000 (17920552 bytes)... [########################################] Executing booted program at 0x80000000 --============= Liftoff! ===============-- But it is still tuck at this stage and I will rebuild busybox, the kernel and BBL to see if it helps.

That looks more promising. My guess is that there's a mismatch between the "computer" and whatever specific parameters were used in building the `boot.bin` image (wrong details in .dts/.dtb, missing config options in the kernel, etc.) There are some pre-built binaries in #1 Maybe try one of them and see if it works for you -- that might help narrow down which part of your own process needs more tuning...

[matsbror]

The pre-built images did not work booting from an SD-card (because the SoC was not built for SPI, I presume). But thanks to your comment that I must load the boot image at 0x800000, I could specify that to litex_term and I am now loading your boot image serially which is horribly slow. I don't know if it will work.

[gsomlo]

On Tue, Dec 06, 2022 at 06:57:49AM -0800, Mats Brorsson wrote: The pre-built images did not work booting from an SD-card (because the SoC was not built for SPI, I presume). But thanks to your comment that I must load the boot image at 0x800000, I could specify that to litex_term and I am now loading your boot image serially which is horribly slow. I don't know if it will work.

It's been a while, but IIRC the nexys4ddr bitstream was built using `--with-ethernet --with-sdcard` (note: *not* spi-sdcard!).

[matsbror]

It's been a while, but IIRC the nexys4ddr bitstream was built using --with-ethernet --with-sdcard (note: not spi-sdcard!).

Yes, this was the issue I had to start boot at all. The board did not recognize that. Switching to spi-sdcard made it start booting but it's still stuck at Liftoff.

[matsbror]

I have come as far as to figure out that the boot loader gets stuck right after calling the uart_sync() in boot.c.

I have put a printout right before and right after, and this is the printout:

--============= Liftoff! ===============--
Before uart sync
Af

It should have printed After uart_sync instead of just Af .

As far as I can see the uart is set up properly and it can load a boot binary over the serial interface just right (with the same behaviour as when loading the file from the SD card.

Changing to define UART_POLLING got the boot process a little further but it still hangs.

It should be noted that I can boot with the prebuilt bitstream and I am using the prebuilt boot.bin. So there's something wrong in how the SoC is built.

Any pointers, please?

Also, is there a way to change the BIOS without re-building the entire SoC?

[gsomlo]

On Tue, Dec 13, 2022 at 11:51:10PM -0800, Mats Brorsson wrote: Any pointers, please?

Sounds like you might have the wrong MMIO register address for the UART somewhere in your DTS (check `bootargs` as well as the actual `liteuart` node itself).

Also, is there a way to change the BIOS without re-building the entire SoC?

I think there might be a way, but I've never figured it out myself. It probably involves loading the bios over the serial link with litex_term, or something like that. @enjoy-digital might be able to provide us with a better starting point...

[matsbror]

But the BIOS works well. I can interact with it.

[matsbror]

In the generated documentation the UART has address: 0x12004800 while in the .dts file (in the conf directory) it is 0x12006800.
I wonder how the bios worked as it goes through the uart as well. I will try to change the dts.

[matsbror]

Thanks! That brought me one step further. The output now ends with:

bbl loader
              vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
                  vvvvvvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrr       vvvvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrr      vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr    vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr    vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr    vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrr      vvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrr       vvvvvvvvvvvvvvvvvvvvvv
rr                vvvvvvvvvvvvvvvvvvvvvv
rr            vvvvvvvvvvvvvvvvvvvvvvvv      rr
rrrr      vvvvvvvvvvvvvvvvvvvvvvvvvv      rrrr
rrrrrr      vvvvvvvvvvvvvvvvvvvvvv      rrrrrr
rrrrrrrr      vvvvvvvvvvvvvvvvvv      rrrrrrrr
rrrrrrrrrr      vvvvvvvvvvvvvv      rrrrrrrrrr
rrrrrrrrrrrr      vvvvvvvvvv      rrrrrrrrrrrr
rrrrrrrrrrrrrr      vvvvvv      rrrrrrrrrrrrrr
rrrrrrrrrrrrrrrr      vv      rrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrr          rrrrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrrrr      rrrrrrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrrrrrr  rrrrrrrrrrrrrrrrrrrrrr

       INSTRUCTION SETS WANT TO BE FREE

Now it remains to see why it's stuck there.

[gsomlo]

On Wed, Dec 14, 2022 at 11:52:54AM -0800, Mats Brorsson wrote: Now it remains to see why it's stuck there.

Means BBL is running, but Linux is still unhappy. Might want to try enabling CONFIG_RISCV_SBI_V01 in your kernel .config, I've had reports that it might make a difference. I haven't used BBL in a while (in fact, I need to update the README file with instructions on how to use opensbi instead). It may be that CONFIG_RISCV_SBI_V01 used to be on by default in riscv defconfig, and was disabled in the interim.

[matsbror]

I enabled that flag, which did not really change anything. However, loading the bitstream from the USB instead of the SD-card made a difference and now it starts to boot Linux but hangs after:

[   15.994106] handlers:
[   15.996340] [<(____ptrval____)>] liteuart_interrupt
[   16.001204] Disabling IRQ #1

I have attached the build log of the bitstream and the full boot log. If you have time to look at it, I would appreciate it very much.
boot.log
build.log

[matsbror]

I will be trying that.

A question: when I set CONFIG_RISCV_SBI_V01=y I get two questions I do not really know what they are:

Early console using RISC-V SBI (SERIAL_EARLYCON_RISCV_SBI) [N/y/?] (NEW)

and

RISC-V SBI console support (HVC_RISCV_SBI) [N/y/?] (NEW)

What is the proper answer to these?

[matsbror]

I answered y to both of the questions before and with the next to last commit for the linux image, I can now boot to prompt.
Thanks for all the help.

[gsomlo]

On Thu, Dec 15, 2022 at 12:31:13PM -0800, Mats Brorsson wrote: Early console using RISC-V SBI (SERIAL_EARLYCON_RISCV_SBI) [N/y/?] (NEW) and RISC-V SBI console support (HVC_RISCV_SBI) [N/y/?] (NEW)

Yes to both won't hurt -- it allows support for using ecalls into machine mode where BBL itself will send/receive characters from the UART. Instead of interacting with LiteUART itself (via MMIO register reads and writes, and IRQs) the linux kernel can trap into machine mode and let the hypervisor (BBL) deal with console i/o. You'd have to modify your DTS (bootargs) accordingly, and probably disable LiteUART driver support from Linux altogether as you wouldn't actually need it anymore. It's a deprecated option, so therefore not future-proof, but could be useful to get something working in a hurry.