Booting into linux on simulation about linux-on-litex-rocket

[gsomlo]

On Sat, Jun 26, 2021 at 06:03:53PM -0700, developandplay wrote: If you are referring to my access strobe fix for blackparrot then I don't think so.

You're right, this is still using the RocketChip -- my mistake :) The "load access fault" appears to happen here: https://github.com/litex-hub/linux/blob/litex-rebase/drivers/soc/litex/litex_soc_ctrl.c#L45 ... with the "bad address" being 0xffffffd004001b04 (presumably of the 32-bit scratch register, but that should be cross-referenced with the csr map of the simulation). This means CSR access is still somehow broken (in the simulator, since it seems to work fine on a number of FPGAs)...

[gsomlo]

Last time I tried this was a very long time ago. Make sure you have libevent-devel and json-c-devel installed. The command line where you load boot.bin over tftp should be:

litex/litex/tools/litex_sim.py \
  --threads 8 --opt-level Ofast \
  --with-ethernet --with-sdram \
  --cpu-type rocket --cpu-variant linux

However, simulating 64-bit rocket running the full tftp download sequence is about an order of magnitude less entertaining than watching paint dry. So, I remember replacing --with-sdram with --ram-init boot.bin, to bypass tftp. Hope it all still works...

[troibe]

Hm I'm still not getting it to start unfortunately.
This is the csr.csv that Litex generates for me:

#--------------------------------------------------------------------------------
# Auto-generated by Migen (3ffd64c) & LiteX (bd146351) on 2021-06-08 21:07:56
#--------------------------------------------------------------------------------
csr_base,ctrl,0x12000000,,
csr_base,ethmac,0x12000800,,
csr_base,ethphy,0x12001000,,
csr_base,identifier_mem,0x12001800,,
csr_base,timer0,0x12002000,,
csr_base,uart,0x12002800,,
csr_register,ctrl_reset,0x12000000,1,rw
csr_register,ctrl_scratch,0x12000004,1,rw
csr_register,ctrl_bus_errors,0x12000008,1,ro
csr_register,ethmac_sram_writer_slot,0x12000800,1,ro
csr_register,ethmac_sram_writer_length,0x12000804,1,ro
csr_register,ethmac_sram_writer_errors,0x12000808,1,ro
csr_register,ethmac_sram_writer_ev_status,0x1200080c,1,ro
csr_register,ethmac_sram_writer_ev_pending,0x12000810,1,ro
csr_register,ethmac_sram_writer_ev_enable,0x12000814,1,rw
csr_register,ethmac_sram_reader_start,0x12000818,1,rw
csr_register,ethmac_sram_reader_ready,0x1200081c,1,ro
csr_register,ethmac_sram_reader_level,0x12000820,1,ro
csr_register,ethmac_sram_reader_slot,0x12000824,1,rw
csr_register,ethmac_sram_reader_length,0x12000828,1,rw
csr_register,ethmac_sram_reader_ev_status,0x1200082c,1,ro
csr_register,ethmac_sram_reader_ev_pending,0x12000830,1,ro
csr_register,ethmac_sram_reader_ev_enable,0x12000834,1,rw
csr_register,ethmac_preamble_crc,0x12000838,1,ro
csr_register,ethphy_crg_reset,0x12001000,1,rw
csr_register,timer0_load,0x12002000,1,rw
csr_register,timer0_reload,0x12002004,1,rw
csr_register,timer0_en,0x12002008,1,rw
csr_register,timer0_update_value,0x1200200c,1,rw
csr_register,timer0_value,0x12002010,1,ro
csr_register,timer0_ev_status,0x12002014,1,ro
csr_register,timer0_ev_pending,0x12002018,1,ro
csr_register,timer0_ev_enable,0x1200201c,1,rw
csr_register,uart_rxtx,0x12002800,1,rw
csr_register,uart_txfull,0x12002804,1,ro
csr_register,uart_rxempty,0x12002808,1,ro
csr_register,uart_ev_status,0x1200280c,1,ro
csr_register,uart_ev_pending,0x12002810,1,ro
csr_register,uart_ev_enable,0x12002814,1,rw
csr_register,uart_txempty,0x12002818,1,ro
csr_register,uart_rxfull,0x1200281c,1,ro
constant,config_clock_frequency,1000000,,
constant,config_cpu_has_interrupt,None,,
constant,config_cpu_reset_addr,268435456,,
constant,config_cpu_type_rocket,None,,
constant,config_cpu_variant_linux,None,,
constant,config_cpu_human_name,rocketrv64[imac],,
constant,config_cpu_nop,nop,,
constant,config_with_build_time,None,,
constant,rom_boot_address,2147483648,,
constant,localip1,192,,
constant,localip2,168,,
constant,localip3,1,,
constant,localip4,50,,
constant,remoteip1,192,,
constant,remoteip2,168,,
constant,remoteip3,1,,
constant,remoteip4,100,,
constant,config_csr_data_width,32,,
constant,config_csr_alignment,32,,
constant,config_bus_standard,wishbone,,
constant,config_bus_data_width,32,,
constant,config_bus_address_width,32,,
constant,config_cpu_has_dma_bus,None,,
constant,ethmac_rx_slots,2,,
constant,ethmac_tx_slots,2,,
constant,ethmac_slot_size,2048,,
constant,ethmac_interrupt,2,,
constant,timer0_interrupt,1,,
constant,uart_interrupt,0,,
memory_region,rom,0x10000000,131072,cached
memory_region,sram,0x11000000,8192,cached
memory_region,main_ram,0x80000000,268435456,cached
memory_region,ethmac,0x30000000,8192,io
memory_region,csr,0x12000000,65536,io

This is the sim.dts that I'm using right now:

/dts-v1/;

/ {
	#address-cells = <1>;
	#size-cells = <1>;
	compatible = "freechips,rocketchip-unknown-dev";
	model = "freechips,rocketchip-unknown";
	chosen {
		bootargs = "earlycon=sbi console=liteuart swiotlb=noforce";
	};
	L13: cpus {
		#address-cells = <1>;
		#size-cells = <0>;
		timebase-frequency = <1000000>;
		L6: cpu@0 {
			clock-frequency = <50000000>;
			compatible = "sifive,rocket0", "riscv";
			d-cache-block-size = <64>;
			d-cache-sets = <64>;
			d-cache-size = <4096>;
			d-tlb-sets = <1>;
			d-tlb-size = <4>;
			device_type = "cpu";
			hardware-exec-breakpoint-count = <1>;
			i-cache-block-size = <64>;
			i-cache-sets = <64>;
			i-cache-size = <4096>;
			i-tlb-sets = <1>;
			i-tlb-size = <4>;
			mmu-type = "riscv,sv39";
			next-level-cache = <&L8>;
			reg = <0x0>;
			/* FIXME: blatant lie! BBL should s/imac/imafdc/;
			   https://github.com/riscv/riscv-pk/issues/166 */
			riscv,isa = "rv64imafdc";
			riscv,pmpregions = <8>;
			status = "okay";
			tlb-split;
			L4: interrupt-controller {
				#interrupt-cells = <1>;
				compatible = "riscv,cpu-intc";
				interrupt-controller;
			};
		};
	};
	L8: memory@80000000 {
		device_type = "memory";
		reg = <0x80000000 0x04000000>; /* 64MB (simulation default) */
	};
	L12: soc {
		#address-cells = <1>;
		#size-cells = <1>;
		compatible = "freechips,rocketchip-unknown-soc", "simple-bus";
		ranges;
		L2: clint@2000000 {
			compatible = "riscv,clint0";
			interrupts-extended = <&L4 3 &L4 7>;
			reg = <0x2000000 0x10000>;
			reg-names = "control";
		};
		L3: debug-controller@0 {
			compatible = "sifive,debug-013", "riscv,debug-013";
			interrupts-extended = <&L4 0x10>;
			reg = <0x0 0x1000>;
			reg-names = "control";
		};
		L0: error-device@3000 {
			compatible = "sifive,error0";
			reg = <0x3000 0x1000>;
		};
		L7: external-interrupts {
			interrupt-parent = <&L1>;
			interrupts = <1 2 3 4>;
		};
		L1: interrupt-controller@c000000 {
			#interrupt-cells = <1>;
			compatible = "riscv,plic0";
			interrupt-controller;
			interrupts-extended = <&L4 11 &L4 9>;
			reg = <0xc000000 0x4000000>;
			reg-names = "control";
			riscv,max-priority = <7>;
			riscv,ndev = <4>;
		};
		L10: rom@10000 {
			compatible = "sifive,rom0";
			reg = <0x10000 0x10000>;
			reg-names = "mem";
		};
		soc_ctrl0: soc_controller@12000000 {
			compatible = "litex,soc-controller";
			reg = <12000000 0xc>;
		};
		liteuart0: serial@12002800 {
			compatible = "litex,liteuart";
			reg = <0x12002800 0x100>;
			interrupt-parent = <&L1>;
			interrupts = <1>;
		};
		mac0: mac@12000800 {
			compatible = "litex,liteeth";
			reg = <0x12000800 0x7c>,
				<0x12001000 0x0a>,
				<0x30000000 0x2000>;
			tx-fifo-depth = <2>;
			rx-fifo-depth = <2>;
			interrupt-parent = <&L1>;
			interrupts = <3>;
		};
	};
};

Console output before things get stuck:

        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2021 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Jun  8 2021 21:07:54
 BIOS CRC passed (e6fbe0f2)

 Migen git sha1: 3ffd64c
 LiteX git sha1: bd146351

--=============== SoC ==================--
CPU:            RocketRV64[imac] @ 1MHz
BUS:            WISHBONE 32-bit @ 4GiB
CSR:            32-bit data
ROM:            128KiB
SRAM:           8KiB
MAIN-RAM:       262144KiB 

--========== Initialization ============--
Ethernet init...

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Executing booted program at 0x80000000

--============= Liftoff! ===============--

Maybe I'm not waiting long enough but for blackparrot and vexriscv at least the BBL loader line gets shown pretty much instantly.

[troibe]

The vmlinux file is the same that I'm using for the arty where I'm now getting some console output.

[troibe]

I revisited this issue and made some progress. Surprisingly I ran into a kernel panic.

lxsim --with-ethernet --with-sdram --cpu-type rocket --cpu-variant linux --csr-csv "csr.csv" --sdram-init boot-sim.bin --threads 8
        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2021 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Jun 23 2021 21:32:11
 BIOS CRC passed (16f2dc15)

 Migen git sha1: 3ffd64c
 LiteX git sha1: adb71bde

--=============== SoC ==================--
CPU:            RocketRV64[imac] @ 1MHz
BUS:            WISHBONE 32-bit @ 4GiB
CSR:            32-bit data
ROM:            128KiB
SRAM:           8KiB
SDRAM:          65536KiB 32-bit @ 1MT/s (CL-2 CWL-2)

--========== Initialization ============--
Ethernet init...
Initializing SDRAM @0x80000000...
Switching SDRAM to software control.
Switching SDRAM to hardware control.

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Cancelled

--============= Console ================--

litex> romboot
Executing booted program at 0x80000000

--============= Liftoff! ===============--
bbl loader
              vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
                  vvvvvvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrr       vvvvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrr      vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr    vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr    vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrrrr    vvvvvvvvvvvvvvvvvvvvvvvv
rrrrrrrrrrrrrrrr      vvvvvvvvvvvvvvvvvvvvvv  
rrrrrrrrrrrrr       vvvvvvvvvvvvvvvvvvvvvv    
rr                vvvvvvvvvvvvvvvvvvvvvv      
rr            vvvvvvvvvvvvvvvvvvvvvvvv      rr
rrrr      vvvvvvvvvvvvvvvvvvvvvvvvvv      rrrr
rrrrrr      vvvvvvvvvvvvvvvvvvvvvv      rrrrrr
rrrrrrrr      vvvvvvvvvvvvvvvvvv      rrrrrrrr
rrrrrrrrrr      vvvvvvvvvvvvvv      rrrrrrrrrr
rrrrrrrrrrrr      vvvvvvvvvv      rrrrrrrrrrrr
rrrrrrrrrrrrrr      vvvvvv      rrrrrrrrrrrrrr
rrrrrrrrrrrrrrrr      vv      rrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrr          rrrrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrrrr      rrrrrrrrrrrrrrrrrrrr
rrrrrrrrrrrrrrrrrrrrrr  rrrrrrrrrrrrrrrrrrrrrr

       INSTRUCTION SETS WANT TO BE FREE
[    0.000000] Linux version 5.13.0-rc3-173670-g9dacadb8c086 (martin@martin-ThinkPad-T14-Gen-1) (riscv64-unknown-linux-gnu-gcc (GCC) 9.2.0, GNU ld (GNU Binutils) 2.34) #3 Wed Jun 9 01:39:48 CEST 2021
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Machine model: freechips,rocketchip-unknown
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080200000-0x0000000083ffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080200000-0x0000000083ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x0000000083ffffff]
[    0.000000] On node 0 totalpages: 15872
[    0.000000]   DMA32 zone: 248 pages used for memmap
[    0.000000]   DMA32 zone: 0 pages reserved
[    0.000000]   DMA32 zone: 15872 pages, LIFO batch:3
[    0.000000] SBI specification v0.1 detected
[    0.000000] riscv: ISA extensions acdfim
[    0.000000] riscv: ELF capabilities acdfim
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 15624
[    0.000000] Kernel command line: earlycon=sbi console=liteuart swiotlb=noforce
[    0.000000] Dentry cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    0.000000] Inode-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
[    0.000000] Sorting __ex_table...
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 47968K/63488K available (3810K kernel code, 4060K rwdata, 2048K rodata, 2808K init, 273K bss, 15520K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] riscv-intc: 64 local interrupts mapped
[    0.000000] plic: interrupt-controller@c000000: mapped 4 interrupts with 1 handlers for 2 contexts.
[    0.000000] random: get_random_bytes called from start_kernel+0x3a8/0x574 with crng_init=0
[    0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0]
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x1d854df40, max_idle_ns: 3526361616960 ns
[    0.000088] sched_clock: 64 bits at 1000kHz, resolution 1000ns, wraps every 2199023255500ns
[    0.006888] Console: colour dummy device 128x32
[    0.009369] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=10000)
[    0.011540] pid_max: default: 32768 minimum: 301
[    0.019933] LSM: Security Framework initializing
[    0.025388] Mount-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.027537] Mountpoint-cache hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.109542] ASID allocator disabled
[    0.116651] EFI services will not be available.
[    0.132434] devtmpfs: initialized
[    0.222549] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.225565] futex hash table entries: 256 (order: 0, 6144 bytes, linear)
[    0.244647] NET: Registered protocol family 16
[    0.695985] clocksource: Switched to clocksource riscv_clocksource
[    1.215104] NET: Registered protocol family 2
[    1.222594] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    1.245086] tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
[    1.248627] TCP established hash table entries: 512 (order: 0, 4096 bytes, linear)
[    1.250868] TCP bind hash table entries: 512 (order: 0, 4096 bytes, linear)
[    1.252940] TCP: Hash tables configured (established 512 bind 512)
[    1.260748] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    1.263292] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    1.273146] NET: Registered protocol family 1
[    1.417378] workingset: timestamp_bits=46 max_order=14 bucket_order=0
[    2.349512] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
[    2.362676] Oops - load access fault [#1]
[    2.364155] CPU: 0 PID: 1 Comm: swapper Not tainted 5.13.0-rc3-173670-g9dacadb8c086 #3
[    2.365874] Hardware name: freechips,rocketchip-unknown (DT)
[    2.367127] epc : litex_soc_ctrl_probe+0x3e/0x106
[    2.369249]  ra : litex_soc_ctrl_probe+0x30/0x106
[    2.371239] epc : ffffffff802420a6 ra : ffffffff80242098 sp : ffffffe00125bbf0
[    2.373053]  gp : ffffffff80cb4df8 tp : ffffffe001250000 t0 : 0000000000000003
[    2.374847]  t1 : 00000000002dc800 t2 : 0000000000000000 s0 : ffffffe00125bc20
[    2.376517]  s1 : ffffffe00133da98 a0 : 0000000004001b00 a1 : ffffffe001205cd8
[    2.378139]  a2 : ffffffe0012b8368 a3 : ffffffe00133dac0 a4 : fffffffffffff000
[    2.379760]  a5 : ffffffd004001b04 a6 : ffffffff80ca7138 a7 : ffffffd004001000
[    2.381379]  s2 : ffffffe0012b8200 s3 : ffffffe0012b8210 s4 : 0000000000000000
[    2.383039]  s5 : ffffffff80c931f0 s6 : ffffffff80cb7018 s7 : ffffffff8060a278
[    2.384676]  s8 : ffffffff80600468 s9 : 0000000000000008 s10: ffffffff804000a8
[    2.386312]  s11: 0000000000000000 t3 : 0000000000001000 t4 : ffffffff80ca71c8
[    2.387921]  t5 : ffffffd004002000 t6 : 0000000000000000
[    2.389271] status: 0000000200000120 badaddr: ffffffd004001b04 cause: 0000000000000005
[    2.390956] Call Trace:
[    2.391978] [<ffffffff802420a6>] litex_soc_ctrl_probe+0x3e/0x106
[    2.394077] [<ffffffff80267f80>] platform_probe+0x3e/0x7c
[    2.395996] [<ffffffff802667ec>] really_probe+0x8e/0x26a
[    2.397861] [<ffffffff80266a00>] driver_probe_device+0x38/0x74
[    2.399806] [<ffffffff80266f70>] device_driver_attach+0x4c/0x50
[    2.401784] [<ffffffff80266fc8>] __driver_attach+0x54/0xc6
[    2.403810] [<ffffffff8026507e>] bus_for_each_dev+0x5a/0x98
[    2.405698] [<ffffffff80267054>] driver_attach+0x1a/0x22
[    2.407555] [<ffffffff8026581e>] bus_add_driver+0x12e/0x190
[    2.409417] [<ffffffff80267572>] driver_register+0x3a/0xd0
[    2.411331] [<ffffffff80268d32>] __platform_driver_register+0x1c/0x24
[    2.413384] [<ffffffff8041044e>] litex_soc_ctrl_driver_init+0x1a/0x22
[    2.415304] [<ffffffff80400cd8>] do_one_initcall+0x6e/0x134
[    2.417214] [<ffffffff80400f78>] kernel_init_freeable+0x172/0x1de
[    2.419164] [<ffffffff803b5018>] kernel_init+0x12/0xf0
[    2.421115] [<ffffffff8000302e>] ret_from_exception+0x0/0xc
[    2.426654] ---[ end trace c52f0c50dc8758d7 ]---
[    2.428148] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    2.429979] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

[gsomlo]

On Thu, Jun 24, 2021 at 04:32:06AM -0700, developandplay wrote: I revisited this issue and made some progress. Surprisingly I ran into a kernel panic. [ 2.362676] Oops - load access fault [#1] [ 2.390956] Call Trace: [ 2.391978] [<ffffffff802420a6>] litex_soc_ctrl_probe+0x3e/0x106 [ 2.394077] [<ffffffff80267f80>] platform_probe+0x3e/0x7c

Likely a CSR access in `litex_soc_ctrl_probe` triggering a cpu fault. Could it be a lingering after-effect of fixing 64/32/16/8 bit access strobes?

[troibe]

If you are referring to my access strobe fix for blackparrot then I don't think so.
I didn't touch anything Litex or Rocket related for that fix just Blackparrot's sytemverilog code.
Therefore Rocket simulation shouldn't be affected.

Unless you also ran into some strobe access problems on Rocket that I'm not aware of.

[troibe]

Is there some sort of (legitimate) address translation happening?
The scratch register should be at 0x12000004 according to the CSR map.

[gsomlo]

On Sun, Jun 27, 2021 at 07:26:53AM -0700, developandplay wrote: Is there some sort of (legitimate) address translation happening? The scratch register should be at 0x12000004 according to the CSR map.

Yeah. On the ecpix5, csr_base is at 0x12000000 physical, but the value for soc_ctl_dev->base returned by devm_platform_ioremap_resource() will be 0xffffffd004016000, here: https://github.com/litex-hub/linux/blob/litex-rebase/drivers/soc/litex/litex_soc_ctrl.c#L102 ***EDIT***: I just noticed that, while we both have the same physical CSR base address (0x12000000), your "virtual" CSR has an extra offset of 0x1b00. Not sure yet whether that's an artefact of platform_ioremap on two different "platforms", or a typo somewhere in e.g. a DTS file...

[troibe]

Thanks for the EDIT now I have something to look into.
Up to this point I wasn't really sure what to look for.