sbates130272 / p2pmem-test

A user-space test platform for testing the p2pdma Linux kernel framework with NVMe CMBs and other PCIe BAR memory.

p2pmem-test: A tool for userspace testing of the p2pdma kernel framework

Welcome to p2pmem-test, a utility for testing PCI Peer-2-Peer (P2P) communication between p2pmem and NVMe devices. This utility becomes very interesting as NVMe devices become p2pmem devices via the Controller Memory Buffer (CMB) and Persistent Memory Region (PMR) features.

Installing

Should be as simple as:

git submodule update --init
make
sudo make install

Contributing

p2pmem-test is an active project. We will happily consider Pull Requests (PRs) submitted against the official repo.

Getting Started

1. In order to run any P2P traffic, you'll need to have a (Linux based) OS that supports p2pdma. This framework is available in all Linux kernels 4.20 or newer, however it is NOT compiled in by default. You will almost certainly have to compile a kernel from source and install that. The instructions on how to do this are beyond the scope of this document but there is a tool that can help in my kernel-tools repo. UPDATE: Mainline userspace support was added in v6.2. This allows userspace applications (like p2pmem-test) to directly map p2pmem via a sysfs file. While this implies no changes to p2pmem-test it does change the invocation (see later).

2. If you are not using a NVMe SSD as your p2pmem device or you are working on a pre-v6,2 kernel You probably want to install the separate p2pmem-pci modul installed to expose the device to userspace. Once installed, you should see the device exposed as a /dev/p2pmemX. Note howeever this step is no longer mandatory as this driver is included in the kernel package files generated by kernel-tools. UPDATE: This approach is no longer needed for NVMe CMB-based p2pmem regions as they are now exposed in mainline via a sysfs entry.

3. Make sure your system has at least one p2pdma capable device. Examples include an Eideticom IOMEM device, a Microsemi NVRAM card or a CMB enabled NVMe SSD that supports the WDS and RDS features (e.g. The Everspin NVNitro card or the Eideticom NoLoad^TM device). Basically this is any PCI EP capable of exposing a BAR with a driver that ties into the p2pdma framework (so things like FPGAs and GPUs can be made to work too).

4. In addition to the p2pmem capable device you need at least one other NVMe SSD. This does not have to be CMB enabled (but it is OK if it does support CMB), any standard NVMe SSD will do. Preferably you will have more than one NVMe SSD but its not a requirement. Ideally place the two (or more) devices noted in the two previous points behind a PCIe switch (for example the Microsemi Switchtec or a PLX switch). If you do not have a switch you can connect both devices to the Root Complex (RC) on the CPU but two things may happen:

   • Performance may drop. Many RCs are inefficient at routing P2P traffic.
   • It might not work at all. Many RCs block P2P traffic. The Linux kernel now maintains a white-list of working root ports.

5. You'll almost certainly want to disable the IOMMU in your system via either the BIOS or the kernel. There are many cases where you can enable the IOMMU but things certainly get a bit trickier so I recommend you start with IOMMU off and then go from there.

6. You may also need to disable the PCIe Access Control Services (ACS) by either the BIOS and/or the kernel because TLP redirection (activated as part of ACS) kills P2P traffic.

7. Finally, p2pmem-test requires the libargconfig submodule. You'll need to either clone recursively or via git submodule update --init.

NVMe CMBs vs PCI Bounce Buffering

When one of the NVMe devices and the p2pmem device are the same PCI EP then CMB should be automatically used. This means the NVMe device should detect the data is in its CMB and do an internal data movement rather than an external DMA. If the p2pmem device is neither of the NVMe devices then two external DMAs will occur. We refer to this second option as a bounce buffer for obvious reasons.

Examples

Pre-v6.2 Kernels

./p2pmem-test /dev/nvme0n1 /dev/nvme1n1 /dev/p2pmem0 -c 1 -s 4k --check

v6.2+ Kernels

./p2pmem-test /dev/nvme0n1 /dev/nvme1n1 /sys/bus/pci/devices/0000\:03\:00.0/p2pmem/allocate  -c 1 -s 4k --check

(Note the PCIe device number may be different on your system).

Copy one 4KB chunk from /dev/nvme0n1 to /dev/nvme0n1 via the memory exposed by /dev/p2pmem0. Perform a check on the data (i.e. write know data to /dev/nvme0n1 and validate that by reading /dev/nvme1n1 after the p2pmem based transfer).

Help

p2pmem-test -h

p2pmem-test's Issues

Is P2PDMA supported on Arm64 platform

Hi Stephen, I tried to use this test suite to perform p2pdma test on an ARM64 server platform. The kernel version which I'm using is 6.6.7. I have one Nvme device with 4MiB CMB, when I enabled P2PDMA and tried to use the p2pmem, it failed. I check the kernel log and the CMB register failed:

the BAR is not in the mhp range. Do you know is that a kernel restriction or ARM64 platform restriction.

Update code to support new userspace access to p2pdma

Version 6.2 of the Linux kernel added direct userspace support for NVMe CMBs and filesystems backed by NVMe SSDs (see this). Update this tool to leverage this. Note no changes to this code may be needed since we may only need to pass the sysfs file into the application.

Update the documentation

The README.md is quite out of date. Update and convert to proper Markdown.

use PCIE P2P ,can NIC direct acces nvme disk?

HI,

I am interested in this technology and have a question:
if use PCIE P2P,can NIC direct acces nvme disk? (maybe like GPUDirect Storage)

Thank you for your reply.

Fix perror issue

Right now there are several places we call perror but do not exit. Handle these cases correctly via exit, return or goto as necessary.

Requires IOMMU to be disabled. Is there any other mechanism to perform P2P DMA when IOMMU is enabled

In the read me it says IOMMU must be disabled including perhaps ACS. So how can one perform P2P DMA when IOMMU is enabled on a system.

Any thoughts?

Regards,
Ramesh

Direct data transfers between NVMe SSDs having CMB support

Hi Stephen,

I would like to know how p2pdma framework can be used for below scenario,

1. I have 2 NVMe SSDs(A and B) with CMB ,WDS and RDS enabled
2. Transfer data from source SSD A to destination SSD B with no host memory involvement and less host CPU involvement

Please let me know how to achieve direct transfers between 2 drives using CMBs

Thanks and regards!!

waht p2p Mem ?

my arch is cpu + switch + ssd + fpga(pcie ep)，my data path need from fpga pcie ep data into ssd,only data pass switch rather than cpu。 How to write fpga(pcie ep) linux driver ？ Have reference examples ？

Add multi-thread support

To increase the performance of p2pmem-test use pthreads to use the p2pmem via multiple threads.

Add host access option

For initial debug it would be useful to have a mode whereby the host directly peeks and pokes at the p2pmem. We might do this with a range of alignments to check how the p2pmem device behaves with non-aligned accesses.