Arm® Ethos™-N Driver Stack

About the Arm Ethos-N neural processing unit (NPU)

The Arm Ethos-N NPUs improve the inference performance of neural networks. The NPUs target 8-bit integer quantized Convolutional Neural Networks (CNN). However, the NPUs also improve the performance of 16-bit integer CNNs and Recurrent Neural Networks (RNN). Please note that 16-bit integer and RNN support are not part of this driver stack release.

For more information, please refer to: https://www.arm.com/solutions/artificial-intelligence

About the Ethos-N driver stack

The Ethos-N driver stack targets the Ethos-N77 and Ethos-N78 set of NPUs. Note that support for Ethos-N77 has been deprecated and will be removed in 21.08.

The Ethos-N driver stack consists of several components.

The list of open source components are:

Arm NN: A software library that enables machine learning workloads on power efficient devices. On Linux®, applications can directly link to Arm NN. On Android™, you can use Arm NN as a backend for the Android NNAPI or applications can directly link to Arm NN.
Arm NN Android neural networks driver: Supports the Android NNAPI on the NPU. The Arm NN Android neural networks driver is optional.
Ethos-N NPU driver: Contains the user space component of the driver.
Ethos-N NPU kernel module: Contains the kernel space component of the driver.
Arm NN Ethos-N NPU backend: Contains the Ethos-N NPU backend for Arm NN.

The following software component is available under an Arm proprietary license:

Ethos-N NPU firmware binaries file: Contains the firmware that runs on the NPU.

Arm NN and the Arm NN Android neural networks driver are external downloads and links are provided below. All other components are part of this driver stack release.

Platform requirements

Your (target) platform must meet specific requirements to run the Ethos-N NPU driver. Your platform must have:

An Armv8-A application processor.
An Arm Ethos-N NPU.
At least 4GB of RAM.
At least 5GB of free storage space.

Secure mode

The Arm Ethos-N NPU will boot up in either secure or non-secure mode depending on how the hardware has been configured. For more information, see Arm Ethos-N78 NPU Technical Reference Manual.

To use the NPU in secure mode, the target platform must have a Trusted Firmware-A (TF-A) that includes the Arm Ethos-N NPU SiP service.

The Arm Ethos-N NPU SiP service can be found in the TF-A source tree, along with a reference implementation for how to use it on the Arm Juno platform. For instructions on how to get the TF-A source and how to build it, see TF-A documentation. Note that TF-A version v2.5 or later must be used. The build flag needed to enable the Arm Ethos-N NPU SiP service for the Arm Juno platform can be found here TF-A Arm Platform Build Options.

For information about how to port TF-A to another platform, see TF-A Porting Guide.

Build tools

To build the Ethos-N NPU software, you require some tools. You must install the following tools on your development platform:

A Linux distribution. An open-source operating system.
Git [Recommended: 2.17.1]. A version control system that software developers use for source code management.
SCons [Recommended: v3.0.1]. An open-source software construction tool.
Make [Recommended: 4.1]. A build automation tool.
Sparse [Recommended: 0.5.1]. A semantic parser for C.
GNU C and C++ and compilers [Recommended: 7.5.0]. Open-source tools for Arm processors.

Install the build tools

You must use specific tools to build the Ethos-N NPU driver. You can use a package manager to install the build tools. For example, enter the following commands to install the build tools on Ubuntu 18.04 in order to cross compile:

sudo apt install git \
    scons \
    make \
    sparse \
    gcc-aarch64-linux-gnu \
    g++-aarch64-linux-gnu \
    gcc \
    bison \
    flex \
    libssl-dev \
    bc

Additionally if you want to build unit tests for the Ethos-N NPU user space libraries, you need Catch2 [Recommended: v2.13.0]:

git clone --depth 1 https://github.com/catchorg/Catch2.git --branch v2.13.0 <path_to_catch>/Catch2

Install the Linux source tree

The Ethos-N driver stack source code depends on the Linux source tree to build the kernel module. You must configure the kernel to build the kernel module. Arm has tested version 4.9 of the Linux source tree in non-SMMU configurations and version 4.14, 4.19, 5.4 in SMMU configurations.

Download version 4.9, 4.14, 4.19 or 5.4 of the Linux source tree from www.kernel.org.
How you compile the driver affects how you configure the Linux kernel source tree:
- If you compile the driver natively, enter the following commands to configure the Linux kernel source tree:
```
make -C <path_to_kernel> defconfig
make -C <path_to_kernel> modules_prepare
```
- If you cross compile the driver, enter the following commands to configure the Linux kernel source tree:
```
make -C <path_to_kernel> ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig
make -C <path_to_kernel> ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- modules_prepare
```
Note that <path_to_kernel> is the directory where the Linux kernel tree is stored.

Building the Ethos-N driver stack

The Ethos-N driver stack is written using portable C++14 and the build system uses scons so it is possible to build for a wide variety of target platforms, from a wide variety of host environments.

Download the Ethos-N driver stack

You must download the different components of the driver stack to build the driver. The different components of the driver stack are available for download in different ways.

Enter the following commands to download Arm NN, the Ethos-N NPU driver, kernel module, and other components you require:

mkdir driver_stack
cd driver_stack
git clone https://github.com/Arm-software/armnn --branch v21.05
git clone https://github.com/Arm-software/ethos-n-driver-stack --branch 21.05

Configure SMMU support

Arm recommends that you configure the Linux kernel with Input/Output Memory Management Unit (IOMMU) support for use as one of the dependencies of the kernel driver. Arm has tested versions 4.14, 4.19 and 5.4 of the Linux kernel with IOMMU support.

Add the following flag to your Linux configuration to include all the dependencies the kernel module needs:

CONFIG_ARM_SMMU_V3=y

Ensure to comment out the following one since the SMMU driver cannot handle the SMMU v1 or v2 and the SMMU v3 both enabled at the same time:

# CONFIG_ARM_SMMU is not set

If you run the NPU without an IOMMU, you must create a reserved memory area. The reserved memory area must begin on a 512MB aligned address and must not be larger than 512MB.

Build the Ethos-N NPU driver

You must follow specific steps to build the Ethos-N NPU driver. You must build the Ethos-N NPU driver, Ethos-N NPU kernel module, and Arm NN. Depending on your system, you must run some of the following steps with appropriate privileges.

Copy the <path_to>/driver_stack/ethos-n-driver-stack/firmware/ethosn.bin file into the /lib/firmware/ folder of the target system that runs the Ethos-N NPU driver.

Note that <path_to> is the directory where the driver_stack directory is stored.
How you compile the driver affects how you build the Ethos-N NPU kernel module:

Note: By default, the kernel module is built for an NPU running in secure mode. If non-secure mode is needed, the following flag EXTRA_CCFLAGS=" -DETHOSN_NS" must be added to the make commands below. It is not possible to use a kernel module built for non-secure mode with an NPU running in secure mode.
- If you compile the driver natively, enter the following commands to build the Ethos-N NPU kernel module:
```
cd <path_to>/driver_stack/ethos-n-driver-stack/kernel-module
make -C <path_to_kernel> M=$PWD modules
```
- If you cross compile the driver, enter the following commands to build the Ethos-N NPU kernel module:
```
cd <path_to>/driver_stack/ethos-n-driver-stack/kernel-module
make -C <path_to_kernel> M=$PWD ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- modules
```
Note that <path_to_kernel> is the directory where the Linux kernel tree is stored.

It is recommended to strip unnecessary symbols from the driver.
- If you compile the driver natively:
```
strip --strip-unneeded <path_to>/driver_stack/ethos-n-driver-stack/kernel-module/ethosn.ko
```
- If you cross compile the driver:
```
aarch64-linux-gnu-strip --strip-unneeded <path_to>/driver_stack/ethos-n-driver-stack/kernel-module/ethosn.ko
```
Copy the kernel module <path_to>/driver_stack/ethos-n-driver-stack/kernel-module/ethosn.ko to the system that runs the Ethos-N NPU driver.
Enter the following command to load the kernel module on the target system:
```
insmod ethosn.ko
```
Enter the following commands to build the user space libraries of the Ethos-N NPU driver:
- If you compile the driver natively:
```
cd <path_to>/driver_stack/ethos-n-driver-stack/driver
scons
```
- If you cross compile the driver:
```
cd <path_to>/driver_stack/ethos-n-driver-stack/driver
scons platform=aarch64
```
Use the configuration options to include dependencies from non-standard locations and to install files into non-standard locations. Enter the following command to see all configuration options:
```
scons --help
```

Enter the following command to install the user space libraries of the Ethos-N NPU driver:

If you compile the driver natively:

cd <path_to>/driver_stack/ethos-n-driver-stack/driver
scons install_prefix=<install_directory> install

If you cross compile the driver:

cd <path_to>/driver_stack/ethos-n-driver-stack/driver
scons platform=aarch64 install_prefix=<install_directory> install

Enter the following commands to link the Ethos-N NPU backend to the Arm NN source tree:

cd <path_to>/driver_stack/armnn/src/backends
ln -s <path_to>/driver_stack/ethos-n-driver-stack/armnn-ethos-n-backend ethos-n

Build Arm NN for TensorFlow Lite. For instructions on building Arm NN, see https://developer.arm.com/solutions/machine-learning-on-arm/developer-material/how-to-guides/configure-the-arm-nn-sdk-build-environment

The following build options are required to the CMake call in the Build Arm NN section of the guide:
```
-DBUILD_TESTS=1
-DARMNNREF=1
-DETHOSN_SUPPORT=1
-DETHOSN_ROOT=<install_directory>
```
For cross compilation, please refer to https://github.com/ARM-software/armnn/blob/master/BuildGuideCrossCompilation.md

As part of the Arm NN build, the process automatically builds the Ethos-N NPU driver plug-in for Arm NN.

Arm uses TensorFlow Lite as an example. You can also build Arm NN for ONNX.
If you require Android NNAPI support, see the instructions for how to build the Arm NN Android NNAPI driver.

Running the Ethos-N NPU driver

There are multiple ways to exercise the Ethos-N NPU driver.

Running the Arm NN Ethos-N NPU backend unit tests. You need to have built Arm NN and the Ethos-N NPU driver.

If you have cross compiled you will need to copy the following files onto the target platform:
- All *.so* files built from Arm NN
- UnitTests built from Arm NN
- libEthosNSupport.so built from the Ethos-N NPU driver inside <install_directory>/lib/
- libEthosNDriver.so built from the Ethos-N NPU driver inside <install_directory>/lib/
Note: You may need to copy additional .so files depending on your toolchain and its runtime dependencies.

Some tests require data files as input. These can be found in the folders <path_to>/driver_stack/ethos-n-driver-stack/armnn-ethos-n-backend/test/replacement-tests and <path_to>/driver_stack/ethos-n-driver-stack/armnn-ethos-n-backend/test/mapping-tests. These two folders (and their contents) must be available to the UnitTests executable, under the paths armnn-ethos-n-backend/test/replacement-tests and armnn-ethos-n-backend/test/mapping-tests, respectively, relative to the current directory that you run UnitTests from. If you have cross compiled you will therefore need to copy these folders onto the target platform.

Set LD_LIBRARY_PATH so the supplied libraries can be found and run the UnitTests for the Ethos-N NPU.
```
LD_LIBRARY_PATH=<path_to_ethosn_libraries>:<path_to_armnn_libs> ./UnitTests --run_test=*EthosN*
```
Running the Ethos-N NPU driver user space unit tests.

You need to have built the driver with testing enabled:
- Add tests=1 to your scons commands to build the user space component unit tests.
- Make sure your CPATH scons variable (specified on the command-line) points to <path_to_catch>/Catch2/single_include/catch2/
If you have cross compiled you will need to copy the following files onto the target platform:
- UnitTests built for the Ethos-N NPU support library inside <path_to>/driver_stack/ethos-n-driver-stack/driver/support_library/build/release_<platform>/tests
- UnitTests built for the Ethos-N NPU command stream inside <path_to>/driver_stack/ethos-n-driver-stack/driver/support_library/command_stream/build/release_<platform>/tests
- UnitTests built for the Ethos-N NPU driver library inside <path_to>/driver_stack/ethos-n-driver-stack/driver/driver_library/build/release_<platform>_kmod/tests
- libEthosNSupport.so built from the Ethos-N NPU driver inside <install_directory>/lib/
- libEthosNDriver.so built from the Ethos-N NPU driver inside <install_directory>/lib/
- ethosn.ko built from the Ethos-N NPU driver inside <path_to>/driver_stack/ethos-n-driver-stack/kernel-module/
Note: You may need to copy additional .so files depending on your toolchain and its runtime dependencies.

Set LD_LIBRARY_PATH so the supplied libraries can be found and run the UnitTests for the Ethos-N NPU.

For each Ethos-N NPU driver user space component unit tests:
```
LD_LIBRARY_PATH=<path_to_ethosn_libraries> ./UnitTests
```

Running the ExecuteNetwork program provided by Arm NN. This supports running of TfLite models.

If you have cross compiled you will need to copy the following files onto the target platform:

All *.so* files built from Arm NN
ExecuteNetwork built from Arm NN in the tests/ folder
libEthosNSupport.so built from the Ethos-N NPU driver inside <install_directory>/lib/
libEthosNDriver.so built from the Ethos-N NPU driver inside <install_directory>/lib/

Note: You may need to copy additional .so files depending on your toolchain and its runtime dependencies.

The ExecuteNetwork program requires parameters passed in. Detail about these can be found by running:

LD_LIBRARY_PATH=<path_to_ethosn_libraries>:<path_to_armnn_libs> ./tests/ExecuteNetwork --help

The minimum set of parameters which need to be provided are:

-f The format of the model provided in e.g. 'tflite-binary'
-i A comma separated list of input tensor names in the model provided
-y A comma separated list of data types for the input tensors
-o A comma separated list of output tensor names in the model provided
-z A comma separated list of data types for the output tensors
-d Path to a whitespace separated list of values for the input
-m Path to the model to be run
-c The device you wish to run this inference on

An example of running Mobilenet_v1_1.0_224_quant.tflite on the Ethos-N NPU:

LD_LIBRARY_PATH=<path_to_ethosn_libraries>:<path_to_armnn_libs> ./tests/ExecuteNetwork -f tflite-binary -i input -y qasymm8 -o MobilenetV1/Predictions/Reshape_1 -z qasymm8 -d input_data.txt -m mobilenet_v1_1.0_224_quant.tflite -c EthosNAcc -c CpuRef

Firmware Binary

The ethosn.bin has been compiled with the following security related flags:

-Werror -Wall -Wextra -Wformat=2 -Wno-format-nonliteral -Wctor-dtor-privacy -Woverloaded-virtual -Wsign-promo -Wstrict-overflow=2 -Wswitch-default -Wconversion -Wold-style-cast -Wsign-conversion -Wno-missing-braces

Limitations

Functional limitations

The following features and feature combinations have known limitations in this Ethos-N driver stack release.

This release has only been tested with specific networks
- VGG16
- MobileNet v1-1-224
- SSD MobileNet v1
- InceptionV3
- InceptionV4
- FSRCNN
- Yolo V3
- ResNet v2-50
Running other networks may result in parts of the network being run by the Arm NN CPU reference backend.

Memory limitations

The driver expects that the minimum amount of memory available for an Arm Ethos-N NPU is 512 MB for systems that do not implement Arm® System Memory Management Unit (SMMU). The NPU might have problems booting up with less memory available.

Systems that implement Arm SMMU require a memory footprint of 3 MB to create all the page translations for the NPU memory accesses.

For more information on memory requirements and limitations, please see the documentation for your SoC.

License

The Ethos-N driver stack is composed of multiple components, each with their own license. The components are:

The Arm Ethos-N NPU driver, which is the collection of user space libraries.
The Arm NN backend, which interfaces the Ethos-N NPU driver to Arm NN.
The Arm Ethos-N NPU kernel module, to be used with the Linux kernel.
The Arm Ethos-N NPU firmware, which will be loaded by the kernel module onto the NPU.

License for the Ethos-N NPU driver and Arm NN backend

The Arm Ethos-N NPU driver and the Arm NN backend are provided under the Apache-2.0 license. See LICENSE and LICENSE for more information. Contributions to this project are accepted under the same license.

Copyright 2018-2021 Arm Limited

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Individual files contain the following tag instead of the full license text.

SPDX-License-Identifier: Apache-2.0

This enables machine processing of license information based on the SPDX License Identifiers that are available here: http://spdx.org/licenses/

License for the Ethos-N NPU kernel module

The Arm Ethos-N NPU kernel module is provided under the GPL v2 only license. See LICENSE for more information. Contributions to this project are accepted under the same license.

Individual files contain the following tag instead of the full license text.

SPDX-License-Identifier: GPL-2.0-only

EULA and TPIP for the Ethos-N NPU firmware

The Ethos-N NPU firmware binary is released under an EULA.

The Ethos-N NPU firmware binary was compiled against the CMSIS library, which is released under the Apache-2.0 license. See apache-2.0.txt for more information.

Trademarks and copyrights

Arm and Ethos are registered trademarks or trademarks of Arm Limited (or its subsidiaries) in the US and/or elsewhere.

Android is a trademark of Google LLC.

Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.

qpc-database / ethos-n-driver-stack Goto Github PK

ethos-n-driver-stack's Introduction