jadarve / lluvia

A real-time computer vision engine implemented on top of Vulkan API.

Low Level Unified VIsion Algorithms (LLUVIA)

lluvia.ai

Lluvia is a computer vision engined designed for real-time applications. It uses the Vulkan API to access the GPU and dispatch computations. Users can describe computations as a compute pipeline where nodes are compute shaders running on the GPU.

This demo is for my real-time optical flow algorithm running using Lluvia on a GTX-1080 GPU. For images of 1016x544 resolution, computation take around 1 millisecond to complete.

Supported Platforms

• Linux.
• Windows.
• Android (work in progress).

Try it out

Try the examples in Google Colab. Make sure the runtime is configured to GPU.

Workflow

Lluvia uses a compute graph to organize and schedule computations on the GPU. The development workflow circles around coding and debugging nodes in such a graph until the whole algorithm is built:

1. The node's inputs, outputs and parameters are described in a Lua script . This description will later be used to instantiate nodes in the graph.

2. The node's computation in the GPU is coded as a compute shader in Open GL Shading Language (GLSL). Shaders are compiled into SPIR-V intemediate representation for later load into the GPU.

3. The node's description and compute shader are loaded into Lluvia's runtime. After this, nodes can be instantiated to build the compute graph and be dispatched to the GPU.

From a user perspective, one needs to only care about describing nodes (inputs, outputs, compute shader, etc.) and connecting nodes to form a graph. Lluvia takes care of the low-level details of dispatching the graph for execution onto the GPU. This workflow allows porting the compute graph from one platform to another with ease.

License

Apache 2.0. See LICENSE for more details

lluvia's Issues

Utility Methods for Python

The Python library needs more convenience functions for repetitive tasks such as:

• Creation of Buffers and Images. This includes creation from Numpy arrays.
• Copy operations (Buffer-Buffer, Image-Buffer). The following is the current way to do it:

  cmdBuffer = session.createCommandBuffer()
  cmdBuffer.begin()
  cmdBuffer.changeImageLayout(img, 'TransferDstOptimal')
  cmdBuffer.copyBufferToImage(stageBuffer, img)
  cmdBuffer.changeImageLayout(img, 'General')
  cmdBuffer.end()
  session.run(cmdBuffer)

• Give the API a more "dynamic" flavour. For instance, running a compute node directly from the object:

  node = session.readComputeNode(...)
  node.run()

• Create ComputeNode objects from GLSL code.

Build `ComputeNode` from JSON

• Create ComputeNodeDescriptor from JSON feed.
• Build ComputeNode objects
• Bind buffer objects to node.

why?

lluvia/samples/imagePyramid/src/main.cpp

Line 30 in acd5c42

img.channels = 4; // overwrite to 4

In the previous line you have sending the pointer and this line you overwrite it?

Fix Python tests using Bazel

Bazel as build system?

• C++ library and executables
• C++ dependencies
• Cython
• Python library
• Tests

Read the docs page

Create the read the docs page :)

Port to macOS

Modify the build scripts to support compilation using the MacOS Vulkan SDK.

See more: https://www.lunarg.com/lunarg-releases-vulkan-sdk-1-0-69-0-for-mac/

createImageViewFromHost() in Python

A new convenience function to reduce the number of lines of code needed to create an ImageView.

Currently, I need to do this:

# GPU memory where the images will be allocated
memory = session.createMemory()
 
# reads the image and transfer it to a ll.Image object (in the GPU)
mouse = ll_util.readSampleImage('mouse')
mouse_gpu = memory.createImageFromHost(mouse)
 
# creates an ImageView used to read the pixels in the GPU shader
in_rgba = mouse_gpu.createImageView(ll.ImageFilterMode.Nearest, ll.ImageAddressMode.ClampToEdge, False, False)

A more convenient way would be:

# GPU memory where the images will be allocated
memory = session.createMemory()
 
# reads the image
mouse = ll_util.readSampleImage('mouse')
 
# creates an ImageView used to read the pixels in the GPU shader
in_rgba = memory.createImageViewFromHost(mouse, ll.ImageFilterMode.Nearest, ll.ImageAddressMode.ClampToEdge, False, False)

Color conversion nodes

• RGB 2 HSV

• HSV 2 RGB

• RGB 2 Lab*

• Lab* 2 RGB

• RGB 2 Gray

• RGB 2 XYZ

• XYZ 2 RGB

Revize shared_ptr passed by value or reference

See in which cases it is necessary to pass std::shared_ptr to methods by value or by reference.

Enable debug in Python

Python Unit Tests

Create unit test cases for the Python wrappers.

• Use Pytest as testing framework
• Check if it is possible to run the tests through the setup.py script.

Doxygen documentation

• Add Doxyfile.
• Define the documentation template for classes and methods.

Graph API

Define the compute graph API for building complex pipelines.

Inspirations:

• GStreamer
• TensorFlow

First examples

Code the first examples using the existing version of the engine to discover how to design the interface for developing real-life compute graphs.

Examples:

• Image pyramid.
• Image gradient with separable convolution masks.

Android Wrappers

• Verify that vulkan.hpp can be used on Android.
• Create android specific CMakeList files (platforms folder?)
• Test application.

Github Actions

Publish the python package to Pypi

Use Github actions

Continuous Integration for Windows

• Bazel cache for Linux and Windows.
• Publish python packages as release artifacts for both OSs.
• Try with Github Actions

Check `memoryTypeBits` when creating buffer

Check that the memory type in memoryTypeBits returned by device.getBufferMemoryRequirements(vkBuffer); is supported by the memory trying to allocate the buffer in Memory::createBuffer

See: https://www.khronos.org/registry/vulkan/specs/1.0-extensions/html/vkspec.html#VkMemoryRequirements

Excessive use of auto keyword

lluvia/samples/imagePyramid/src/main.cpp

Line 30 in acd5c42

img.channels = 4; // overwrite to 4

auto keyword it is not created for this behavior, you should use it only for acquire iterator and lambda function expression

Define the API for mapping buffers

• Consider blocking the memory page.
• Do the mapping from the buffer object, not the memory one.
• Return some type of smart pointer of the mapped region.
• Validate if the buffer is mappable.

Fix Buffer creation from Lua

The script below does not work:

local builder = ll.class(ll.ComputeNodeBuilder)

function builder.newDescriptor() 
    
    local desc = ll.ComputeNodeDescriptor.new()
    
    desc.builderName  = 'Square'
    desc.localShape   = ll.vec3ui.new(32, 1, 1)
    desc.gridShape    = ll.vec3ui.new(1, 1, 1)
    desc.program      = ll.getProgram('Square')
    desc.functionName = 'main'

    desc:addPort(ll.PortDescriptor.new(0, 'in_buffer', ll.PortDirection.In, ll.PortType.Buffer))
    desc:addPort(ll.PortDescriptor.new(1, 'out_buffer', ll.PortDirection.Out, ll.PortType.Buffer))

    return desc
end

function builder.onNodeInit(node)
    
    local in_buffer = node:getPort('in_buffer')

    -- Allocate out_buffer in the same Memory as in_buffer
    local memory = in_buffer.memory
    
    -- TODO: call createBuffer using default usage flags
    -- TODO: overload of createBuffer with in_buffer.usageFlags does not work. Need unsafe version
    local out_buffer = memory:createBuffer(in_buffer.size)
    
    -- bind the output
    node:bind('out_buffer', out_buffer)
    
    -- configure the dispath shapes according to the size of in_buffer
    node:configureGridShape(ll.vec3ui.new(in_buffer.size / 4, 1, 1))

end

ll.registerNodeBuilder('Square', builder)

is there something wrong in CMakeLists?

#################################################

SOURCE FILES SETTINGS

#################################################
add_subdirectory(core)--------------------------->core is not exsit, you means : cpp/core ?
add_subdirectory(tools)
add_subdirectory(samples)

creates the shaders custom target

include (cmake/shaders.cmake)

Now, there are some compilation errors with cmake...............

Python Packages

• Build Python packages for Linux and Windows.
• Create releases.

Wiki Pages

Create a first version of the project's wiki.

• About the project
• Getting started:
  • Building C++ library.
  • Building Python wrappers.
  • Basic usage C++ and Python

Wrap Vulkan enums in Lluvia enum class

This will allow me to control which flags are exposed to users. Some of the Vulkan flags I don't really use.

Build on OSX

• OS-specific compile flags in the repo.
• Include jobs in Github Actions to validate.
• Build the C++ library and Python wheel.

Python package for google colab

Build a Python wheel compatible with Google's colab.

Python wrappers

• Use Cython.
• Check possibility to automate parts of the generation. Like creating the Cython header files.

Code example

import lluvia.core as ll
import numpy as np

session = ll.createSession()

memory = session.createMemory(ll.MemoryFlags.HostVisible | ll.MemoryFlags.HostCoherent, 4096, False)

buf1 = memory.createBuffer(2048)
buf2 = memory.createBuffer(2048)

program = session.createProgram('add.spv')

# setting all this is boring! maybe could be read from a JSON
nodeDesc = ll.ComputeNodeDescriptor(program, 'main', [32, 1, 1], [32, 1, 1])
nodeDesc.parameters.append(ll.ParameterType.Buffer)
nodeDesc.parameters.append(ll.ParameterType.Buffer)

node = session.createComputeNode(nodeDesc)
node.bind(0, buf1)
node.bind(1, buf2)


# alternative way, read a JSON with all the information
"""
{
    "function": "main",
    "grid_x": 1,
    "grid_y": 1,
    "grid_z": 1,
    "local_x": 1024,
    "local_y": 1,
    "local_z": 1,
    "parameters": [
        "BUFFER",
        "BUFFER"
    ],
    "spirv": "gADABsAAAAZAAAA/QABADgAAQA="
}
"""
node2 = session.readComputeNode('pathToNodeDesc.json', localGroup=[16, 1, 1])
node2.grid = [32, 1, 1] # local workgroup cannot be changed from here
                        # as the node has been compiled.
node2.bind(0, buf1)
node2.bind(1, buf2)

# node2 can be saved to JSON. It contains all the relevant information
ll.writeComputeNode(node2, 'pathToFile.json')


cmdBuffer = session.createCommandBuffer()
cmdBuffer.run(node)

session.run(cmdBuffer)

# convert to numpy array
bufCopy = buf1.toHost([1024], np.dtype.float32)

buf2.fromHost(bufCopy)

Change to generic Objects in ComputeGraph

• Instead of adding specific containers for buffers, images, etc, use a single container of Object to store them all.
• Change the JSON format to include a single objects list. Distinguish each object with a type attribute.

Android/Java wrappers

Compile the library to Android. Check if the wrappers should be done manually or using something like JavaCpp https://github.com/bytedeco/javacpp

Also, check how to connect the camera2 API with Vulkan. A starting point could be this: https://stackoverflow.com/questions/43507536/how-to-connect-android-mediacodec-surface-to-vulkan

• https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#VK_ANDROID_external_memory_android_hardware_buffer
• https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/vkspec.html#AHardwareBuffer

Continuous Integration

• Check if Travis CI supports running the core tests.
• Build C++ library.
• Build Python wrappers.
• Build Doxygen.
• Build Python doc (Sphinx?).
• Run test (C++, Python).

Python Documentation

Check how to use Sphinx to generate the documentation pages.

Optical Flow Filter implementation

Implement the optical flow filter algorithm in https://github.com/jadarve/optical-flow-filter.

• Build compute pipeline depending on filter parameters (pyramid levels, maxflow).
• Shared/Static library?
• WebCam demo with OpenCV.

Define clang-format style

See https://clang.llvm.org/docs/ClangFormatStyleOptions.html

Optical Flow Demo

Code the optical flow algorithm in https://github.com/jadarve/optical-flow-filter using Lluvia.

Migrate from expand_template in Python to C++

• This will eliminate depending on Python for building pure C++ projects.
• Consider using https://github.com/pantor/inja

Python compilation using Bazel

Use bazel to compile the Python wrappers :)

Some useful stuff:

• https://github.com/horia141/bazel-pypi-package

Image objects

• Check which image formats are usable in compute shaders.
• Supported image formats. Number of channels and type.
• Image class.

Nodes library

Add Bazel rules to create node libraries. A node library is a collection of Lua and SPIR-V files packed together and ready to be used within a session.

Instruction for build

Hi @jadarve,

I would like to try lluvia. I tried compiling using CMake without success as some files cannot be found.

Is there any plan on adding build and installation instruction?

Thank you!

Organize glsl folder

Currently, the glsl folder in the repository contains shader code for testing purposes. In the future, this folder will contain general (library) code for the GPU.

Task

• Move the test shaders to a core/test/glsl folder.
• Adapt the CMake scripts to support shader compilation from several places.

replace this for std::tuple pattern

lluvia/samples/imagePyramid/src/main.cpp

Line 17 in acd5c42

using image_t = struct {

These structute should use a tuple pattern to manage this heterogenous struct

Change the way enum types are converted to and from strings

See the implementation of objectTypeToString and stringToObjectType coded in #14.

ll-info JSON output

Print the output of ll-info in JSON format instead of plain text.

Mediapipe Calculators

• Inclue Lluvia as a dependency on mediapipe's bazel scripts.
• Create calculators to run graphs within mediapipe.

Compilation for Linux for Tegra

• Compile the engine for the Linux for Tegra distribution.
• Run the optical-flow example in one board: TX2 or TX nano.

Update to rules_vulkan 0.0.3

Port Contracts

• Add more properties to the port descriptor in order to validate proper binding. Examples include:
  • Image channels
  • Image channel type
  • Image view sampled
  • Image view normalized coordinates

For instance, take RGBA2Gray node:

function builder.newDescriptor()

    local desc = ll.ComputeNodeDescriptor.new()

    desc:init(builder.name, ll.ComputeDimension.D2)

    -- TOTHINK: increased port contracts by checking internal attributes of the PortType
    -- For ImageView, check all image attributes + image view attributes.
    desc:addPort(ll.PortDescriptor.new(0, 'in_rgba', ll.PortDirection.In, ll.PortType.ImageView))
    desc:addPort(ll.PortDescriptor.new(1, 'out_gray', ll.PortDirection.Out, ll.PortType.ImageView))

    return desc
end

function builder.onNodeInit(node)

    local in_rgba = node:getPort('in_rgba')

    -- validate in_rgba is actually a rgba8ui image
    -- TODO: remove once port-contracts are implemented
    local err = ll.isValidImage(in_rgba, ll.ChannelCount.C4, ll.ChannelType.Uint8)
    if err ~= nil then
        error(builder.name .. ': error validating in_rgba: ' .. err)
    end

    -- ...
end

The code in the builder, as well as the likelihood of errors, can be reduced if the node internally validates if in_rgba has 4 channels and it's of the correct channel type.

⚠️ Consider also the parameterization of nodes according to channel type and count. For instance, ImagePyramid_r8ui is tailored to 8-bit single-channel images. Any other combination would require a new node definition.

Fix references in class hierarchy

Currently all objects that are created by the Session class hold a reference to it. Instead, hold a reference to the Vulkan resources needed by that object for maintaining its life-cycle. In particual, the vk::Device reference.