spcl / smi

Streaming Message Interface: High-Performance Distributed Memory Programming on Reconfigurable Hardware

Streaming Message Interface

Streaming Message Interface is a a distributed memory HLS programming model for FPGAs that provides the convenience of message passing for HLS-programmed hardware devices. Instead of bulk transmission, typical of message passing model, with SMI messages are streamed across the network during computation, allowing communication to be seamlessly integrated into pipelined designs.

This repository contains an high-level synthesis implementation of SMI targeting OpenCL and Intel FPGAs, and all the applications used for the evaluation perfomed in the paper: "Streaming Message Interface: High-Performance Distributed Memory Programming on Reconfigurable Hardware", Tiziano De Matteis, Johannes de Fine Licht, Jakub Beránek, and Torsten Hofler. In Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, 2019 (SC 2019).

Please refer to the wiki and to the paper for a reference on how to use SMI for your own distributed FPGA programs.

Reproducing the paper experiments

All the tests and evaluations reported in the paper have been performed on a set of Bittware 520N cards (Stratix 10), each of them equipped with 4 network connections (QSFP modules) operating at 40Gbit/s.

Requirements

The library depends on:

• CMake for configuration
• Intel FPGA SDK for OpenCL pro, version 18.1 (http://fpgasoftware.intel.com/opencl/). Experimental: support for v19+
• GCC (version 5+)
• An MPI implementation (e.g. OpenMPI)
• Python (version 3+)
• CLang (version 8+)

Compilation

After cloning this repository, make sure you clone the submodule dependency, by executing the following command:

git submodule update --init

The project uses CMake for configuration. To configure the project and build the bitstreams and executables:

mkdir build
cd build
cmake ..

The experiments shown in the paper are organized in two subdirectories of the CMake folder, microbenchmarks and examples.

For each of them the following targets are offered:

• make <application>_emulator builds the emulation version of the FPGA program;
• make <application>_host builds the host program;
• make <application>_<program>_aoc_report generates the report;
• make <application>_<program>_aoc_build builds the hardware (can take several hours).

The applications presents in the repository are the following. For the details please refer to the paper:

Microbenchmarks

• bandwidth: bandwidth microbenchmark: an MPMD application composed by two programs, namely bandwidth_0 (sender) and bandwidth_1 (receiver);
• latency: latency microbenchmark: an MPMD application composed by two programs, namely latency_0 (source) and latency_1 (destination).
• injection: injection microbenchmark: an MPMD application composed by two programs, namely injection_0 (sender) and injection_1 (receiver).
• broadcast: broadcast microbenchmark: an SPMD application (broadcast)
• reduce: reduce microbenchmark: an SPMD application (reduce)
• scatter: scatter microbenchmark (not included in the paper): an SPMD application (scatter)
• gather: gather microbenchmark (not included in the paper): an SPMD application (gather)

Application examples

• stencil_smi: stencil application, smi implementation. It is composed by a single program (stencil_smi);
• stencil_onchip: on chip version of the stencil application;
• gesummv_smi: gesummv, smi implementation: composed by a two programs (gesummv_rank0 and gesummv_rank1);
• gesummv_onchip: on chip version of the gesummv application.

Unit tests

To enable unit tests, please execute cmake with the following flag -DENABLE_TESTS=ON

Please Note: all the host programs have been written by considering the target architecture used in the paper, which is characterized by a set of nodes each one having 2 FPGAs. If you are using a different setup, please adjust the host programs.

Example

Suppose that the user wants to execute the stencil_smi application in emulation. The following steps must be performed:

cd examples
# Compile the emulation version
make stencil_smi_emulator -j
# Compile the host program
make stencil_smi_host
cd stencil_smi
# Execute the program
env  CL_CONTEXT_EMULATOR_DEVICE_INTELFPGA=8 mpirun -np 8 ./stencil_smi_host emulator <num-timesteps>

To generate the report, from the examples directory in the CMake folder, the user must execute:

make  stencil_smi_stencil_smi_aoc_report

The report will be stored under examples/stencil_smi/stencil_smi.

Stencil parameters

For the stencil application, the stencil sizes and number of ranks in either dimension are configured using CMake parameters:

cmake .. -DSMI_STENCIL_SIZE_X=8192 -DSMI_STENCIL_SIZE_Y=8192 -DSMI_STENCIL_NUM_PROCS_X=2 -DSMI_STENCIL_NUM_PROCS_Y=2

Other parameters include SMI_VECTORIZATION_WIDTH, SMI_DATATYPE, SMI_FMAX, and SMI_ROUTING_FILE.

smi's Issues

[Intel Compiler v19+] Some collectives fail in compilation

For some collectives (broadcast, scatter, gather), v19+ internally fails in synthesis or in the generation of the report.

The message reported by the compiler is meaningless. For example, while generating broadcast report it fails with:

Stack dump:
0.      Program arguments: /mnt/beegfs/apps/ault/intelFPGA_pro/19.1/hld/linux64/bin/../../llvm/bin/aocl-opt -march=fpga -O3 -board /apps/ault/intelFPGA_pro/19.1/hld/board/bittware_pcie/hardware/p520_max_sg280l/b
oard_spec.xml -fp-relaxed=true -scheduler-fmax=480 -dbg-info-enabled --soft-elementary-math=false -pass-remarks-output=pass-remarks.yaml broadcast.fpga.bc -o broadcast.kwgid.bc
1.      Running pass 'Function Pass Manager' on module 'broadcast.fpga.bc'.
2.      Running pass 'Increase latency of critical loop variables' on function '@app'
#0 0x00007f2e9a0bd1da llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/mnt/beegfs/apps/ault/intelFPGA_pro/19.1/hld/llvm/bin/../lib/libLLVM-8svn.so+0xc1e1da)
#1 0x00007f2e9a0bb17c llvm::sys::RunSignalHandlers() (/mnt/beegfs/apps/ault/intelFPGA_pro/19.1/hld/llvm/bin/../lib/libLLVM-8svn.so+0xc1c17c)
#2 0x00007f2e9a0bb2cc SignalHandler(int) (/mnt/beegfs/apps/ault/intelFPGA_pro/19.1/hld/llvm/bin/../lib/libLLVM-8svn.so+0xc1c2cc)
#3 0x00007f2e992925f0 __restore_rt (/lib64/libpthread.so.0+0xf5f0)
#4 0x00007f2e9b5156f0 (anonymous namespace)::LoopRecombine::findPathsAndConditions(llvm::Value*, (anonymous namespace)::LoopRecombine::TracingInfo&, llvm::SmallVector<std::pair<llvm::Value*, bool>, 2u>&) (/mnt/b
eegfs/apps/ault/intelFPGA_pro/19.1/hld/llvm/bin/../lib/libLLVM-8svn.so+0x20766f0)
#5 0x00007f2e9b515476 (anonymous namespace)::LoopRecombine::findPathsAndConditions(llvm::Value*, (anonymous namespace)::LoopRecombine::TracingInfo&, llvm::SmallVector<std::pair<llvm::Value*, bool>, 2u>&) (/mnt/b
eegfs/apps/ault/intelFPGA_pro/19.1/hld/llvm/bin/../lib/libLLVM-8svn.so+0x2076476)
#6 0x00007f2e9b5180db (anonymous namespace)::LoopRecombine::isValidForRecombine(llvm::PHINode*, llvm::Value*, llvm::Value*) (/mnt/beegfs/apps/ault/intelFPGA_pro/19.1/hld/llvm/bin/../lib/libLLVM-8svn.so+0x20790db
)
#7 0x00007f2e9b51a99a (anonymous namespace)::LoopRecombine::runOnFunction(llvm::Function&) (/mnt/beegfs/apps/ault/intelFPGA_pro/19.1/hld/llvm/bin/../lib/libLLVM-8svn.so+0x207b99a)
#8 0x00007f2e9a1f426e llvm::FPPassManager::runOnFunction(llvm::Function&) (/mnt/beegfs/apps/ault/intelFPGA_pro/19.1/hld/llvm/bin/../lib/libLLVM-8svn.so+0xd5526e)
#9 0x00007f2e9a1f4424 llvm::FPPassManager::runOnModule(llvm::Module&) (/mnt/beegfs/apps/ault/intelFPGA_pro/19.1/hld/llvm/bin/../lib/libLLVM-8svn.so+0xd55424)
#10 0x00007f2e9a1f51ec llvm::legacy::PassManagerImpl::run(llvm::Module&) (/mnt/beegfs/apps/ault/intelFPGA_pro/19.1/hld/llvm/bin/../lib/libLLVM-8svn.so+0xd561ec)
#11 0x0000561092006b15 main (/mnt/beegfs/apps/ault/intelFPGA_pro/19.1/hld/linux64/bin/../../llvm/bin/aocl-opt+0x29b15)
#12 0x00007f2e97bd6505 __libc_start_main (/lib64/libc.so.6+0x22505)
#13 0x00005610920076bd _start (/mnt/beegfs/apps/ault/intelFPGA_pro/19.1/hld/linux64/bin/../../llvm/bin/aocl-opt+0x2a6bd)
Error: Optimizer FAILED.

Bug: collective tests do not passes with balanced routing

The test that works with Integermessage and user-defined synchrony degree, stalls.
It seems that a loop occurs in the generated routing table.

By removing channels from the topology and part of the test (in the .cl files) it completes.

Loop initiziation interval greater of 1 for compiler v >=19.1

With the newer versions of the compiler (19.1 or 19.2) the II of loops is greater than 1.
This happens for both SMI device code as well as for loops in the user application.

Bug: rendezvous causes stall in specific situation

Description

If in the same kernel there are 2+ SMI channels opened toward the same destination, if the message length is small (=1 network packet) the rendezvous mechanism could cause a stall

Example

 SMI_Channel chan_send1=SMI_Open_send_channel(2, SMI_INT, my_rank+1, 0, comm);
 SMI_Channel chan_send2=SMI_Open_send_channel(2, SMI_INT, my_rank+1, 1, comm);

for(int i=0;i<2;i++)
   <push the data in the two channels>

On the receiver side symmetric operations are applied. This is broken in the following case:

• when i=1, we first push the second data elements in the channel. The network packet is sent. We have zero tokens and the rendezvous mechanism wait for a message from my_rank+1. This prevents the execution of the second push
• on the receiver side, we received the network packet. We perform the first pop. However, tokens is not zero, therefore we will not send the rendevous message. The second pop is stalled because the data will never arrive

Possible solution
Change tokens condition on push?