ct-clmsn / nofi

Nim wrapper of ROFI (Rust Open Fabrics Interface)

nofi - Nim OpenFabrics Interface

Nofi wraps the existing rofi interface implemented by Pacific Northwest National Laboratory (PNNL). Nofi provides the Nim programming language support for RDMA distributed communication (put and get), fundamental operations for PGAS applications.

This library extends rofi by providing a sequence type that wraps memory registered with the underlying rofi RDMA communicaton library. The sequence type only supports values that are of SomeNumber types. The sequence type provides element-access, slice, iterator, and partitioning support.

Additional functionality is provided to handle asynchronous put and get operations using Nim's defer statements, combining templates with try statements, and futures.

Nim applications using nofi require static compilation and linking!

Install

Download and install rofi

./configure --prefix=<PATH_TO_INSTALL_DIR>
make && make install
export LD_LIBRARY_PATH=<PATH_TO_INSTALL_DIR>/lib:$LD_LIBRARY_PATH

Modify makefile to point LIBDIR and INCDIR to the path set in <PATH_TO_INSTALL_DIR>. Use the makefile to see if your setup compiles.

make

Use the nimble tool to install nofi

nimble install nofi

Generate documentation from source

nimble doc nofi

Important Developer Notes

Nofi demonstrates the versitility of the rofi library and it's value as a lightweight technology for extending RDMA support via libfabric to languages that have a FFI (foreign function interface) compatible with C.

Users need to review the rofi documentation prior to using this library in order to understand rofi's feature set and potential limitations.

This library requires static compilation of an end user's Nim program. Please review the makefile to learn how to enable static compilation and linking with the Nim compiler.

Users must initialize and terminate all programs using nofi_init and nofi_finit. Not taking this step will result in undefined behavior.

Users are strongly encouraged to utilize Nim blocks and scoping rules to managing memory that has been registered with nofi. Users can review 'tests/test_sharray.nim' for an example of memory management using Nim blocks and scopes.

Examples

The directory 'tests/' provides several examples regarding how to interact with this library.

Licenses

• nofi is Boost Version 1.0 (2022-)
• rofi is BSD License (2020)

Date

24 February 2022

Author

Christopher Taylor

Special Thanks to the ROFI authors

• Roberto Gioiosa - [email protected]
• Ryan Friese - [email protected]
• Mark Raugas - [email protected]
• Pacific Northwest National Labs/US Department of Energy

Many Thanks

• The Nim community and user/developer forum

Dependencies

• nim 1.6.4
• libfabric
• Rust OpenFabrics Interface Transport Layer - rofi

rofi Dependencies

• GNU Autotools
• libfabric
• libibverbs
• uthash
• libatomic
• librt: Extended realtime library, generally installed on Linux development clusters
• pthreads: POSIX thread library, generally installed on Linux development clusters

nofi's Issues

Mini-review on bindings.nim

1. Since you reuse the cname exaclty, you can use importc without specifying the C name. Combining with push pragmas you get

{.deadCodeElim: on.}
{.push cdecl, importc.}
proc rofi_init*(provider:cstring) : int
proc rofi_finit*() : int
proc rofi_get_size*() : uint32
proc rofi_get_id*() : uint32
proc rofi_put*[T : SomeNumber](dst : ptr UncheckedArray[T], src : ptr UncheckedArray[T], size : uint64, id : uint32, flags : uint64) : int
proc rofi_iput*[T : SomeNumber](dst : ptr UncheckedArray[T], src : ptr UncheckedArray[T], size : uint64, id : uint32, flags : uint64) : int
proc rofi_get*[T : SomeNumber](dst : ptr UncheckedArray[T], src : ptr UncheckedArray[T], size : uint64, id : uint32, flags : uint64) : int 
proc rofi_iget*[T : SomeNumber](dst : ptr UncheckedArray[T], src : ptr UncheckedArray[T], size : uint64, id : uint32, flags : uint64) : int
proc rofi_isend*[T : SomeNumber](id: uint32, address : ptr UncheckedArray[T], size : uint64, flags : uint64) : int
proc rofi_irecv*[T : SomeNumber](id : uint32, address : ptr UncheckedArray[T], size : uint64, flags : uint64) : int
proc rofi_alloc*[T : SomeNumber](size : uint64, flags : uint64, buf : ptr ptr UncheckedArray[T]) : int
proc rofi_release*[T : SomeNumber](address : ptr UncheckedArray[T]) : int
proc rofi_barrier*() 
proc rofi_wait*() : int
proc rofi_get_remote_address*(address : pointer, id : uint64) : pointer
proc rofi_get_local_address_from_remote_address*(address : pointer, id : uint64) : pointer 

2. Can you confirm the return values of your proc? In Nim int is the same size as a pointer so int32 on 32-bit and int64 on 64 bit. In contrast C int are usually int32 (and maybe 16-bit on some embedded platforms?), for C compat Nim provides cint. This has seldom impact due to C integer promotion rules but still.