fpgadeveloper / fpga-drive-aximm-pcie

Example designs for FPGA Drive FMC

FPGA Drive FMC Reference Designs

Description

This repo contains the example designs for the Opsero FPGA Drive FMC Gen4 mated with several FPGA and MPSoC evaluation boards.

Important links:

• Datasheet for the FPGA Drive FMC Gen4
• The user guide for these reference designs is hosted here: Ref design for FPGA Drive FMC docs
• To report a bug: Report an issue.
• For technical support: Contact Opsero.

Requirements

This project is designed for version 2022.1 of the Xilinx tools (Vivado/Vitis/PetaLinux). If you are using an older version of the Xilinx tools, then refer to the release tags to find the version of this repository that matches your version of the tools.

In order to test this design on hardware, you will need the following:

• Vivado 2022.1
• Vitis 2022.1
• PetaLinux Tools 2022.1
• FPGA Drive FMC Gen4
• M.2 NVMe PCIe Solid State Drive
• One of the supported carriers listed here

Target designs

This repo contains several designs that target various supported development boards and their FMC connectors. The table below lists the target design name, the M2 ports supported by the design and the FMC connector on which to connect the FPGA Drive FMC Gen4. Some of the target designs require a license to generate a bitstream with the AMD Xilinx tools.

Target board	Target design	M2 ports	FMC Slot	License required
KC705	kc705_hpc	SSD1	HPC	YES
KC705	kc705_lpc	SSD1	LPC	YES
KCU105	kcu105_hpc	SSD1	HPC	YES
KCU105	kcu105_hpc_dual	SSD1 & SSD2	HPC	YES
KCU105	kcu105_lpc	SSD1	LPC	YES
PicoZed 7015	pz_7015	SSD1	LPC	NO
PicoZed 7030	pz_7030	SSD1	LPC	NO
UltraZed-EV carrier	uzev_dual	SSD1 & SSD2	HPC	NO
VC707	vc707_hpc1	SSD1	HPC1	YES
VC707	vc707_hpc2	SSD1	HPC2	YES
VC709	vc709_hpc	SSD1	HPC	YES
VCK190	vck190_fmcp1	SSD1 & SSD2	FMCP1	YES
VCK190	vck190_fmcp2	SSD1 & SSD2	FMCP2	YES
VMK180	vmk180_fmcp1	SSD1 & SSD2	FMCP1	YES
VMK180	vmk180_fmcp2	SSD1 & SSD2	FMCP2	YES
VCU118	vcu118	SSD1	FMCP	YES
VCU118	vcu118_dual	SSD1 & SSD2	FMCP	YES
ZC706	zc706_hpc	SSD1	HPC	YES
ZC706	zc706_lpc	SSD1	HPC	YES
ZCU104	zcu104	SSD1	LPC	NO
ZCU106	zcu106_hpc0	SSD1	HPC0	NO
ZCU106	zcu106_hpc0_dual	SSD1 & SSD2	HPC0	NO
ZCU106	zcu106_hpc1	SSD1	HPC1	NO
ZCU111	zcu111	SSD1	FMCP	YES
ZCU111	zcu111_dual	SSD1 & SSD2	FMCP	YES
ZCU208	zcu208	SSD1	FMCP	YES
ZCU208	zcu208_dual	SSD1 & SSD2	FMCP	YES

Build instructions

Clone the repo:

git clone https://github.com/fpgadeveloper/fpga-drive-aximm-pcie.git

Source Vivado and PetaLinux tools:

source <path-to-petalinux>/2022.1/settings.sh
source <path-to-vivado>/2022.1/settings64.sh

Build all (Vivado project and PetaLinux):

cd fpga-drive-aximm-pcie/PetaLinux
make petalinux TARGET=uzev_dual

More comprehensive build instructions can be found in the user guide:

• For Windows users
• For Linux users

Contribute

We strongly encourage community contribution to these projects. Please make a pull request if you would like to share your work:

• if you've spotted and fixed any issues
• if you've added designs for other target platforms
• if you've added software support for other devices

Thank you to everyone who supports us!

About us

Opsero Inc. is a team of FPGA developers delivering FPGA products and design services to start-ups and tech companies. Follow our blog, FPGA Developer, for news, tutorials and updates on the awesome projects we work on.

fpga-drive-aximm-pcie's Issues

[ZCU106] Cannot detect the SSD

I followed the steps provided in the documentation and successfully generated the image files for the ZCU106 development board. The board boots up without any problem. But, somehow, I can't detect the SSDs.

I faced the same problem with both of the following designs created for ZCU106

• ZCU106 HPC0
• ZCU106 HPC0 Dual

The output of the lspci command for HPC0 Dual configuration is as follows:

0000:00:00.0 PCI bridge Xilinx Corporation Device 9134
0001:00:00.0 PCI bridge Xilinx Corporation Device 9134

The SSD part is one of the tested ones by Opsero: Samsung 970 EVO 500GB

ZCU106 constraint file

In the zcu106-hpc0.xdc:

# MGT locations
# SSD1 HPC0_DP0-3 (PCIe lanes 0-3) are connected to MGT bank 226 (X0Y12-X0Y15) in this order: 0->2, 1->1, 2->3, 3->0
# To rearrange the preplacement, we do this: 0->X0Y11 (temporary), 3->X0Y12, 2->X0Y15, 0->X0Y14, 1->X0Y13 (already placed)
set_property LOC GTHE4_CHANNEL_X0Y14 [get_cells {_i/xdma_0/inst/pcie4_ip_i/inst/gt_top_i/diablo_gt.diablo_gt_phy_wrapper/gt_wizard.gtwizard_top_i/_xdma_0_0_pcie4_ip_gt_i/inst/gen_gtwizard_gthe4_top._xdma_0_0_pcie4_ip_gt_gtwizard_gthe4_inst/gen_gtwizard_gthe4.gen_channel_container[3].gen_enabled_channel.gthe4_channel_wrapper_inst/channel_inst/gthe4_channel_gen.gen_gthe4_channel_inst[0].GTHE4_CHANNEL_PRIM_INST}]
set_property LOC GTHE4_CHANNEL_X0Y12 [get_cells {_i/xdma_0/inst/pcie4_ip_i/inst/gt_top_i/diablo_gt.diablo_gt_phy_wrapper/gt_wizard.gtwizard_top_i/_xdma_0_0_pcie4_ip_gt_i/inst/gen_gtwizard_gthe4_top._xdma_0_0_pcie4_ip_gt_gtwizard_gthe4_inst/gen_gtwizard_gthe4.gen_channel_container[3].gen_enabled_channel.gthe4_channel_wrapper_inst/channel_inst/gthe4_channel_gen.gen_gthe4_channel_inst[3].GTHE4_CHANNEL_PRIM_INST}]
set_property LOC GTHE4_CHANNEL_X0Y15 [get_cells {_i/xdma_0/inst/pcie4_ip_i/inst/gt_top_i/diablo_gt.diablo_gt_phy_wrapper/gt_wizard.gtwizard_top_i/_xdma_0_0_pcie4_ip_gt_i/inst/gen_gtwizard_gthe4_top._xdma_0_0_pcie4_ip_gt_gtwizard_gthe4_inst/gen_gtwizard_gthe4.gen_channel_container[3].gen_enabled_channel.gthe4_channel_wrapper_inst/channel_inst/gthe4_channel_gen.gen_gthe4_channel_inst[2].GTHE4_CHANNEL_PRIM_INST}]
set_property LOC GTHE4_CHANNEL_X0Y14 [get_cells {_i/xdma_0/inst/pcie4_ip_i/inst/gt_top_i/diablo_gt.diablo_gt_phy_wrapper/gt_wizard.gtwizard_top_i/_xdma_0_0_pcie4_ip_gt_i/inst/gen_gtwizard_gthe4_top._xdma_0_0_pcie4_ip_gt_gtwizard_gthe4_inst/gen_gtwizard_gthe4.gen_channel_container[3].gen_enabled_channel.gthe4_channel_wrapper_inst/channel_inst/gthe4_channel_gen.gen_gthe4_channel_inst[0].GTHE4_CHANNEL_PRIM_INST}]
set_property LOC GTHE4_CHANNEL_X0Y13 [get_cells {_i/xdma_0/inst/pcie4_ip_i/inst/gt_top_i/diablo_gt.diablo_gt_phy_wrapper/gt_wizard.gtwizard_top_i/_xdma_0_0_pcie4_ip_gt_i/inst/gen_gtwizard_gthe4_top.*_xdma_0_0_pcie4_ip_gt_gtwizard_gthe4_inst/gen_gtwizard_gthe4.gen_channel_container[3].gen_enabled_channel.gthe4_channel_wrapper_inst/channel_inst/gthe4_channel_gen.gen_gthe4_channel_inst[1].GTHE4_CHANNEL_PRIM_INST}]

Should the first GTHE4_CHANNEL_X0Y14 be changed to GTHE4_CHANNEL_X0Y11?

Thank you.

Running with KCU105

Hi Jeff,

I am trying to run unsuccessfully the kcu105-hpc-dual project. I have previously ran this project with the kc705 board before and it worked fine. I have also ran the loop-back test (with both boards) and it also works fine. The problem after I execute all of the steps, there is no feedback on the terminal.

Because the kcu105-hpc_-dual_ did not work, I tried the kcu105-hpc, but when I build it through the .bat file, the tcl console got stuck at create_bd_cell -type ip -vlnv xilinx.com:ip:ddr4 ddr4_0. I remember that it got stuck at some line (maybe the same?) when I've tried to build the "-dual" project. Then I force quit and tried again and there I went, just as I have for this non dual project. The single SSD project also didn't work.

I am running on a Windows 10 machine with Vivado 2018.2, but I have access to Linux, with the same version of Vivado installed.

Thank you,

some problems about this script of 18.2 version

Hello I found in reference for the design of zynq-pci-express-root-complex-design-in-vivado and another project by using script to generate. After practice, I found that both projects have certain problems.
When using the the script of 18.2 version to generate the project, the first thing I encountered was that UART IP was not called. After I added UART IP, I found that the ELF file could not be downloaded, and the SDK reported an error showing address 0x8000_0000 download error. Considering that address edit may be a problem, I changed the address of BAR0 to 0x4000_0000, and vivado reported an critical warning again.

[BD 41-1288] 
Address segment </axi_pcie_0/S_AXI/BAR0> in </processing_system7_0/Data> at <0x40000000 [ 256M ]> is illegal. 
This address exceeds the base address limitations <0x80000000 [ 1G ]> of the interface(s) </processing_system7_0/M_AXI_GP1> through which it is accessed by this address space.

In addition, I designed the program according to the Zynq-PCI-Express-Root-Complex-Design-in-Vivado tutorial, and the program downloaded normally, but when I executed the XAxiPcie_ReadReg function, it got stuck. How can I fix it ?

when I try to configure the SSD through AXI BAR, I cannot write to it.

Hello, we clone repo with project for Vivado 2019.2, but when I try to configure the SSD through AXI BAR, I cannot write to it. Bridge Enable has been turned on.Do you know what happened? Is the gen3 porting gen2 driver not supporting BAR?

ZCU106 ELF downloading failed

When running command "petalinux-boot --jtag --fpga --bitstream ./images/linux/system.bit"
It shows "Failed to download PetaLinux/zcu106_hpc0/images/linux/bl31.elf"

But the SD card method works.

Thank you.

S_AXI_LITE's assigned address is out of range

As shown the CTRL0 address was assigned to 0x04_0000_0000.
However, the S_AXI_LITE address channel is 32-bit, which is incompatible with assigned address.

Booting KCU105 HPC image without FPGA Drive

Is this supported? I want to see if the image will boot before getting a FPGA Drive.
I generated the KCU105 image successfully with Vivado 2020.2

When booting I get stuck at this point:

Starting kernel ...

Ramdisk addr 0xaf7a4000,
FDT at 0xaf79d000
earlycon: ns16550a0 at MMIO 0x44a01000 (options '115200n8')
printk: bootconsole [ns16550a0] enabled
cma: Reserved 16 MiB at 0xae400000
Linux version 5.4.0-xilinx-v2020.2 (oe-user@oe-host) (gcc version 9.2.0 (GCC)) #1 Wed Jul 20 21:41:51 UTC 2022
setup_memory: max_mapnr: 0x7ffff
setup_memory: min_low_pfn: 0x80000
setup_memory: max_low_pfn: 0xb0000
setup_memory: max_pfn: 0xfffff
Zone ranges:
  DMA      [mem 0x0000000080000000-0x00000000afffffff]
  Normal   empty
  HighMem  [mem 0x00000000b0000000-0x00000000ffffefff]
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x0000000080000000-0x00000000ffffefff]
Initmem setup node 0 [mem 0x0000000080000000-0x00000000ffffefff]
setup_cpuinfo: initialising cpu 0
setup_cpuinfo: Using full CPU PVR support
wt_msr_noirq
pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
pcpu-alloc: [0] 0
Built 1 zonelists, mobility grouping on.  Total pages: 522559
Kernel command line: console=tDentry cache hash table entries: 131072 (order: 7, 524288 bytes, linear)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 2044036K/2097148K available (5556K kernel code, 182K rwdata, 1576K rodata, 170K init, 565K bss, 36728K reserved, 16384K cma-reserved, 1310716K highmem)
Kernel virtual memory layout:
  * 0xfffea000..0xfffff000  : fixmap
  * 0xff800000..0xffc00000  : highmem PTEs
  * 0xff7ff000xilinx_timer_init: Timer base: 0xd0020000, Clocksource base: 0xd0020010
clocksource: xilinx_clocksource: mask: 0xffffffff max_c/amba_pl/timer@41c00000: irq=2, cpu_id 0
xilinx_timer_shutdown
xilinx_timer_set_periodic
Calibrating delay loop... 49.35 BogoMIPS (lpj=246784)
pid_max: default: 4096 minimum: 301
Mount-cache hash table entries: 2048 (order: 1, 8192 bytes, linear)
Mountpoint-cache hash table entries: 2048 (order: 1, 8random: get_random_u32 called from bucket_table_alloc.isra.0+0x70/0x218 with crng_init=0
clocksource: jiffies: mask: 0xffffffffNET: Registered protocol family omic allocations
audit: initializing netlink subsys (disabled)
PCI: Probing PCI hardware
audit: type=2000 audit(0.290:1): state=initialized audit_enabled=0 res=1
vgaarb: loaded
clocksource: Switched to clocksource xilinx_clocksource
NET: Registered protocol family 2
tcp_listen_portaddr_hash hash table entries: 512 (order: 1, 12288 bytes, linear)
TCP established hash table entries: 8192 (order: 3, 32768 bytes, linear)
TCP Trying to unpack rootfs image as initramfs...
random: fast init done
Freeing initrd memory: 8556K
workingset: timestamp_bits=14 max_order=19 bucket_order=5
Key type cifs.idmap registered
romfs: ROMFS MTD (C) 2007 Red Hat, Inc.
bounce: pool size: 64 pages
io scheduler mq-deadline registered
io scheduler kyber registered
GPIO IRQ not connected
XGpio: gpio@40000000: registered, base is 511

Thanks a lot

Memory access failed

hello，

I am working with this repo to port to my customized board, which has a combo of zynq xc7z100ffg900 and 4lane nvme connecter.

I build this project using the zc706_pcie, and then changed the part. I can build the bitstream and the vitis project.

Vivado/Vitis Version : 2020.2
repo branch : master
host computer: ubuntu18.04

However, when I program my board, the vitis will give error message.

error message：
17:38:55 ERROR : Memory write error at 0x80000000. APB Memory access port is disabled

Error running build-kcu105-lpc.bar

Hi again, Jeff.

Running the latest (cloned 6 July) build-kcu105-lpc.bat gives the following error from Vivado:

****** Vivado v2017.2 (64-bit)
  **** SW Build 1909853 on Thu Jun 15 18:39:09 MDT 2017
  **** IP Build 1909766 on Thu Jun 15 19:58:00 MDT 2017
    ** Copyright 1986-2017 Xilinx, Inc. All Rights Reserved.

source build-kcu105-lpc.tcl
# set version_required "2017.2"
no such variable
    (read trace on "::env(XILINX)")
    invoked from within
"split $::env(XILINX) /"
    invoked from within
"lindex [split $::env(XILINX) /] 3"
    invoked from within
"set ver [lindex [split $::env(XILINX) /] 3]"
    (file "build-kcu105-lpc.tcl" line 8)
INFO: [Common 17-206] Exiting Vivado at Thu Jul  6 15:38:02 2017...

The throughput of zc706 and single ssd

Hi,

I want to know the write or read speed when using zc706 and Samsung 970 EVO. Is there some result?

[ZCU106] Boot stalls without proper connection with the FMC extension card

I successfully generate the .bit and .hdf files and import them from the PetaLinux tool. Inside the Petalinux, I activate the required kernel modules-drivers for the PCIe and NVMe. Also for the rootfs, to monitor the devices I include some packages such as lspci, lsblk, mount, etc.

The board boots up successfully if I connect the extension card to the FMC connector. (I can't detect(lspci) the SSDs, but it's another problem.) But, somehow, the boot process stalls if I don't connect the extension card. A few seconds after powering up, the boot process stalls and prints the following message to the serial console:

INFO: rcu_sched detected stalls on CPUs/tasks:

Luckily, I managed to detect the driver causing the stall. It was the XDMA PL PCIe driver. If I don't include the XDMA driver in the kernel, the board successfully boots up regardless of the existence of the extension card.

What would be the reason for it? Is there a problem with the PL design, or is it not even a problem at all?

ZC706 and FPGA Drive SSD nvme0: Identify controller failed(-4)

Hello, I use my own FPGA Drive module and clone repo with project for Vivado 2020.2, but when I try to boot petalinux on ZC706 Evalution Borad, I got the errors. The logs like: nvme nvme0: I/O XX QID 0 timeout, disable controller; nvme nvme0: Identify Controller failed（-4）； nvme nvme0： Removing atfer probe failure status: -5.
And I tried downloading the prebuilt boot files and testing them on my hardware. I got the same errors.

Boot stall on VCK190

I followed the guide to build this project for target vck190_fmcp1. Running: make petalinux TARGET=vck190_fmcp1 works just fine, but when I program the board, the boot process stalls early on with no obvious errors:

[0.015]****************************************
[0.070]Xilinx Versal Platform Loader and Manager
[0.126]Release 2022.1   Apr 11 2022  -  09:29:50
[0.183]Platform Version: v2.0 PMC: v2.0, PS: v2.0
[0.247]BOOTMODE: 0xE, MULTIBOOT: 0xF0000000
[0.303]****************************************
[0.521]Non Secure Boot
[3.399]PLM Initialization Time
[3.447]***********Boot PDI Load: Started***********
[3.507]Loading PDI from SD1_LS
[3.555]Monolithic/Master Device
[285.899]282.366 ms: PDI initialization time
[285.959]+++Loading Image#: 0x1, Name: lpd, Id: 0x04210002
[286.027]---Loading Partition#: 0x1, Id: 0xC
[337.869] 51.758 ms for Partition#: 0x1, Size: 2912 Bytes
[342.775]---Loading Partition#: 0x2, Id: 0x0
[347.209] 0.518 ms for Partition#: 0x2, Size: 48 Bytes
[351.404]---Loading Partition#: 0x3, Id: 0x0
[369.105] 13.783 ms for Partition#: 0x3, Size: 58912 Bytes
[371.409]---Loading Partition#: 0x4, Id: 0x0
[379.466] 4.139 ms for Partition#: 0x4, Size: 5888 Bytes
PSM Firmware version: 2022.1 [Build: Apr 11 2022 09:29:50 ]
[386.867]+++Loading Image#: 0x2, Name: fpd, Id: 0x0420C003
[391.919]---Loading Partition#: 0x5, Id: 0x8
[399.547] 3.711 ms for Partition#: 0x5, Size: 1568 Bytes
[404.000]+++Loading Image#: 0x3, Name: apu_subsystem, Id: 0x1C000000
[407.602]---Loading Partition#: 0x6, Id: 0x0

I have attempted to boot via SD card, and JTAG (using the same SD card image for rootfs), with the same results for either method.

NOTE: When I boot via jtag I use the command: petalinux-boot --jteg --kernel. I omitted the --fpga flag as described here as I get the error message: "Invalid argument "--fpga" for system arch "versal""

I am running on Xilinx 2022-1 on a Ubuntu 20 machine.

The IP is set as GEN3 x4, but the connection is trained as GEN1 X2.

Hello, I am running this project on KCU105, the IP is set as GEN3 x4, but the connection is trained as GEN1 X2, may I ask what is wrong?

VC707 and FPGA Drive SSD can't see console logs during the petalinux booting.

Hello, we bought your FPGA Drive module and clone repo with project for Vivado 2018.2, but when I try to boot petalinux on vc707, I can't see any console logs. I've already see this issue #1 (comment) but for me it's not working. But in qemu it works well. When I create simple project with microblaze and all prerequisites for petalinux I can boot linux and see console logs. May you have any suggestion to solve my problem? Also I did all steps from this article https://www.fpgadeveloper.com/2016/04/connecting-an-ssd-to-an-fpga-running-petalinux.html/ but it also didn't work for me.

Need help with simulation of fpga-drive-aximm-pcie-2018.2_1 on vivado

Is this the correct place to ask?

Jaime

Multiple issues while building the SW project

Executing the build-sdk.bat as directed has several issues with dec2019 release for zcu106 dev kit.

1- Warning: No app will be generated for Zynq Ultrascale+ designs. ZCU106 dev kit has a Zynq Uscale+ device
2- ELF does not exist for zcu106_ssd_test. Chking build status: no files matched glob pattern "./_ssd_test"

I am not verse in .tcl and have not solve this yet. :(

The board_part definition was not found for em.avnet.com:picozed_7015_fmc2:part0:1.1.

excuse me, there is a question, how can i fix it?

Can't get Linux to boot KCU105

I'm trying to get Linux up and running on a KCU105 with no luck. I can get all the way through the PCI device enumeration example no problem using the steps outlined in this repo's README. I then switch over to my Linux virtual machine and the instructions for the KC705 from your blog. I make it all the way through configuring and building Linux and attempting to boot over JTAG. I've got a serial console open on my Windows host watching the relevant COM ports at the baud rate and settings specified by your instructions. The commands I use are shown below:

$ petalinux-boot --jtag --fpga --bitstream ../kcu105_hpc_pcie_wrapper.bit
$ petalinux-boot --jtag --kernel

The first command runs and returns fine. The second command runs and returns, however after waiting for a number of minutes, there is no output to my serial consoles.

After a number of attempts, I tried to boot with QEMU to see if it was the image:

$ petalinux-boot --qemu --kernel

Which boots fine, however I did notice this line:

...
PCI: CLS 0 bytes, default 32
Skipping unavailable RESET gpio -2 (reset)
futex hash table entries: 16 (order: -4, 448 bytes)
audit: initializing netlink subsys (disabled)
audit: type=2000 audit(2.020:1): initialized
workingset: timestamp_bits=30 max_order=18 bucket_order=0
romfs: ROMFS MTD (C) 2007 Red Hat, Inc.
bounce: pool size: 64 pages
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
console [ttyS0] disabled
44a00000.serial: ttyS0 at MMIO 0x44a01000 (irq = 3, base_baud = 6250000) is a 16550A
console [ttyS0] enabled
brd: module loaded
libphy: Fixed MDIO Bus: probed
NET: Registered protocol family 17
Key type encrypted registered
Warning: unable to open an initial console.
Freeing unused kernel memory: 7264K (c0510000 - c0c28000)
This architecture does not have kernel memory protection.
random: fast init done

PetaLinux 2017.1 plnx_microblaze /dev/ttyS0

plnx_microblaze login:

I'm wondering if this could be contributing to my issues. Any help you'd be able to provide would be deeply appreciated. I can provide the full output from the QEMU boot if necessary