Skip to content

Linux NPU

Motherboard SoC Platform NPU
K1 RK3568 Rockchip 1 TOPS
K1B RK3568 Rockchip 1 TOPS
K1MINI RK3568 Rockchip 1 TOPS
K2B H618 Allwinner —
K2C H618 Allwinner —
K3 RK3562 Rockchip 1 TOPS
K3B RK3562 Rockchip 1 TOPS
K4B T113 Allwinner —
K5C A133 Allwinner —
K7 RK3576 Rockchip 6 TOPS
K7C RK3576 Rockchip 6 TOPS
K7S RK3576 Rockchip 6 TOPS
K8 RK3588 Rockchip 6 TOPS
K8D RK3588S2 Rockchip 6 TOPS
K9 T527 Allwinner 2 TOPS
K10B A733 Allwinner 3 TOPS
K11C RK3566 Rockchip 1 TOPS

Allwinner NPU

aw_npu_model_zoo supported platforms:

  • A733 / T736
  • T527 / MR527
  • MR536 / T536
  • V85x / R853

AW NPU toolchain supported platforms:

  • A733 / T736
  • T527 / MR527
  • MR536 / T536

Allwinner T527 / A733 and other platforms are equipped with the Vivante VIP9000 series NPU, supporting INT8/UINT8/INT16 quantization precision. They can import mainstream model formats such as ONNX, TensorFlow, Caffe, and PyTorch.

NPU development is carried out using the ACUITY Toolkit for model conversion (import → quantize → export NBG). The exported NBG file is directly deployed to the target board and inferred via the VIP Lite API or vpm_run.

AW NPU components:

  • ACUITY Toolkit — End‑to‑end model conversion / quantisation / compilation tool. Supports importing models from various AI frameworks and outputs NBG or OpenVX C source code.
  • VIPLite Driver — Embedded lightweight NPU driver (tens of KB of memory) that loads and runs NBG models. Pre‑installed in the board firmware.
  • TIM‑VX — Runtime inferencing interface for online inference. Supports real‑time inference on backends such as TensorFlow Lite and Android NN.
  • vpm_run — Board‑side NBG model testing tool, pre‑installed in the firmware.
  • aw_npu_model_zoo — Official model repository containing examples such as YOLOv5/v8/v11, MobileNet, OCR, CLIP, Zipformer, etc.

PC‑Side NPU Deployment

Development Environment Setup

For Allwinner NPU development, it is recommended to use a Docker image to set up the PC‑side development environment. The Docker image includes the vpm_compile model compilation tool, the Acuity Toolkit quantisation/conversion tools, and the VIP Lite Runtime libraries.

# Place linux_aw_npu/ in the Ubuntu home directory (e.g., copied via network drive or Samba)
# Enter the workspace directory
mkdir -p ~/npu_workspace && cd ~/npu_workspace

# Extract the Docker image
# linux_aw_npu/docker/ubuntu-npu_v2.0.10.2.tar.zip needs to be unzipped first to obtain the tar file
unzip ~/linux_aw_npu/docker/ubuntu-npu_v2.0.10.2.tar.zip
# Produces ubuntu-npu_v2.0.10.2.tar

# Load the Docker image
sudo docker load -i ubuntu-npu_v2.0.10.2.tar

# Start the container, mounting the model zoo to /workspace
sudo docker run --ipc=host -itd \
  -v ~/linux_aw_npu/model_zoo/v0.8.0:/workspace \
  --name npu_env ubuntu-npu:v2.0.10.2 /bin/bash

Model Conversion

The Acuity Toolkit is the core tool for model conversion, supporting workflows such as import, quantize, and export. The following procedure uses K10B (A733) as an example, based on the Allwinner NPU Docker image ubuntu-npu:v2.0.10.2 (acuity v6.30.22) and awnpu_model_zoo-v0.8.0.

YOLOv5s Conversion Example

The following uses the yolov5s_rt.onnx model from the model zoo (which has had its post‑processing nodes removed):

# Enter the Docker container
sudo docker exec -it npu_env /bin/bash

# Set environment variables
export ACUITY_PATH=$HOME/acuity-toolkit-whl-6.30.22/bin
export VIV_SDK=$HOME/Vivante_IDE/VivanteIDE5.11.0/cmdtools

# Enter the conversion directory
cd /workspace/examples/yolov5/convert_model

# 1. Model import
pegasus import onnx --model yolov5s_rt.onnx \
  --output-model yolov5s_rt.json --output-data yolov5s_rt.data

# 2. Model quantisation (uint8, 12 calibration images)
pegasus quantize --model yolov5s_rt.json --model-data yolov5s_rt.data \
  --device CPU --with-input-meta yolov5s_rt_inputmeta.yml \
  --iterations 12 --rebuild \
  --model-quantize yolov5s_rt_uint8.quantize \
  --quantizer asymmetric_affine --qtype uint8

# 3. Export NBG model (for A733 platform)
pegasus export ovxlib --model yolov5s_rt.json --model-data yolov5s_rt.data \
  --dtype quantized --model-quantize yolov5s_rt_uint8.quantize \
  --batch-size 1 --save-fused-graph \
  --target-ide-project 'linux64' \
  --optimize VIP9000NANODI_PLUS_PID0X1000003B \
  --viv-sdk ${VIV_SDK} --pack-nbg-unify \
  --with-input-meta yolov5s_rt_inputmeta.yml \
  --postprocess-file yolov5s_rt_postprocess_file.yml \
  --output-path ./wksp/yolov5s_rt_uint8/yolov5s_rt_uint8

Platform optimisation parameter mapping:

Platform --optimize Parameter
V85x / R853 VIP9000PICO_PID0XEE
T527 / MR527 VIP9000NANOSI_PLUS_PID0X10000016
MR536 / T536 VIP9000NANODI_PLUS_PID0X1000003B
A733 / T736 VIP9000NANODI_PLUS_PID0X1000003B

Using the helper scripts provided in the model zoo (one‑click import → quantise → export):

cd /workspace/examples/yolov5/convert_model
./convert_model_env.sh                     # Create script symlinks
./pegasus_import.sh yolov5s_rt            # Import
./pegasus_quantize.sh yolov5s_rt uint8 12 # Quantise
./pegasus_export_ovx_nbg.sh yolov5s_rt uint8 a733  # Export A733 NBG

The exported model is located at ../model/yolov5s_rt_uint8_a733.nb. Use nbinfo to inspect model information:

nbinfo -a yolov5s_rt_uint8_a733.nb
# Input:  3x640x640x1, UINT8, non-quant
# Output: 80x80x85x3 + 40x40x85x3 + 20x20x85x3, FP32
# Total Video Memory: 19 MB

Validated version: acuity v6.30.22 + awnpu_model_zoo-v0.8.0, resulting NBG is approximately 5.3 MB.

Board‑Side NPU Deployment

Driver Verification

# Check VIP Core driver
lsmod | grep vipcore

# Check device node
ls -l /dev/vipcore

# Check NPU frequency
cat /sys/class/devfreq/3600000.npu/cur_freq
cat /sys/class/devfreq/3600000.npu/available_frequencies
cat /sys/class/devfreq/3600000.npu/governor

Pre‑installed Test Resources

The SDK provides pre‑installed test resources at /aw-test/npu/vpm_run/:

Resource Path
NBG model v3 /aw-test/npu/vpm_run/v3/network_binary.nb
NBG model v2 /aw-test/npu/vpm_run/v2/network_binary.nb
Input data /aw-test/npu/vpm_run/input_0.dat
Configuration file /aw-test/npu/vpm_run/a733_sample.txt

Inference Test

Use vpm_run to test NBG models:

# Create test configuration
echo "[network]" > /tmp/npu_test/test.txt
echo "/tmp/npu_test/yolov5s_rt_uint8_a733.nb" >> /tmp/npu_test/test.txt
echo "[input]" >> /tmp/npu_test/test.txt
echo "/aw-test/npu/vpm_run/input_0.dat" >> /tmp/npu_test/test.txt

# Push model to the board (NBG obtained from linux_aw_npu/nbg_models/)
adb push ~/linux_aw_npu/nbg_models/yolov5s_rt_uint8_a733.nb /tmp/npu_test/

# Run inference (-l loop count, -d device ID, -b 1 skips output saving)
adb shell "vpm_run -s /tmp/npu_test/test.txt -l 5 -d 0 -b 1"

# 224x224 classification model (using pre‑installed resources)
adb shell "vpm_run -s /aw-test/npu/vpm_run/a733_sample.txt -l 5 -d 0 --show_top5 1 --save_txt 0"

Expected output example (K10B YOLOv5s 640x640):

init vip lite, driver version=0x00020003...
VIPLite driver software library version 2.0.3.4-AW-2025-05-16
vip lite init OK.
cid=0x1000003b, device_count=1
  device[0] core_count=1
create network 0: 7844 us.
input 0 dim 3 640 640 1, data_format=2
ouput 0 dim 80 80 85 3
ouput 1 dim 40 40 85 3
ouput 2 dim 20 20 85 3
memory pool size=3892224byte
run time for this network 0: 20375 us.
profile inference time=20096us
vpm run ret=0

VIP Lite Runtime API

End‑side application development is based on the VIP Lite API, supporting C programming for both Linux and Android systems. Example code is available in the examples/ directory of the model zoo.

Native on‑board compilation (recommended to avoid glibc compatibility issues):

The K10B Debian 11 system comes with gcc 10.2 and libopencv‑dev, allowing you to compile and run examples directly on the board.

# 1. Push source code and runtime libraries to the board
adb shell "mkdir -p /tmp/yolov5_build/model"
adb push ~/linux_aw_npu/model_zoo/v0.8.0/examples/yolov5/*.cpp /tmp/yolov5_build/
adb push ~/linux_aw_npu/model_zoo/v0.8.0/examples/yolov5/*.h /tmp/yolov5_build/
adb push ~/linux_aw_npu/model_zoo/v0.8.0/common/npuruntime/npulib.* /tmp/yolov5_build/
adb push ~/linux_aw_npu/model_zoo/v0.8.0/common/npuruntime/npu_util.* /tmp/yolov5_build/
adb push ~/linux_aw_npu/model_zoo/v0.8.0/common/npuruntime/include/vip_lite*.h /tmp/yolov5_build/
adb push ~/linux_aw_npu/model_zoo/v0.8.0/common/npuruntime/lib_linux_aarch64/A733/*.so /tmp/yolov5_build/
adb push ~/linux_aw_npu/nbg_models/yolov5s_rt_uint8_a733.nb /tmp/yolov5_build/model/
adb push ~/linux_aw_npu/model_zoo/v0.8.0/examples/yolov5/model/dog.jpg /tmp/yolov5_build/model/

# 2. Compile on the board
adb shell "cd /tmp/yolov5_build && g++ -std=c++11 -O2 *.cpp -o yolov5_demo -I. -L. -lNBGlinker -lVIPhal \$(pkg-config --cflags --libs opencv4) -lm -ldl"

# 3. Run inference
adb shell "cd /tmp/yolov5_build && LD_LIBRARY_PATH=. ./yolov5_demo -nb model/yolov5s_rt_uint8_a733.nb -i model/dog.jpg"

Expected output example (YOLOv5s, 640x640):

Tip

If cross‑compiling with Ubuntu 22.04's gcc‑aarch64‑linux‑gnu, note that it links against glibc 2.35, which is not compatible with Debian 11 (glibc 2.31). Native on‑board compilation is recommended.

input  0 dim 3 640 640 1
output 0 dim 80 80 85 3
output 1 dim 40 40 85 3
output 2 dim 20 20 85 3
create network 0: 13257 us
run time: 21611 us
detection num: 3
 16: 92%, [134, 226, 307, 545], dog
  7: 69%, [471,  77, 689, 173], truck
  1: 52%, [162, 125, 559, 423], bicycle

Rockchip NPU

rknn-toolkit2 supported platforms:

  • RK3588 Series
  • RK3576 Series
  • RK3566/RK3568 Series
  • RK3562 Series
  • RV1103/RV1106
  • RV1103B/RV1106B
  • RV1126B
  • RK2118

rknn_model_zoo supported platforms:

  • Supports RK3562, RK3566, RK3568, RK3576, RK3588, RV1126B
  • Limited support for RV1103, RV1106
  • Supports RV1109, RV1126, RK1808

rknn-llm supported platforms:

  • RK3588 Series
  • RK3576 Series
  • RK3562 Series
  • RV1126B Series

Rockchip RK356X, RK3576, and RK3588 series chips all feature an integrated NPU (Neural Processing Unit), delivering INT8 computing power of up to 1 TOPS and 6 TOPS respectively. They fully support mainstream deep learning frameworks such as TensorFlow, TF‑Lite, PyTorch, Caffe, and ONNX. For detailed specifications, refer to the respective chip datasheets.

As specialised hardware designed for neural network operations, the NPU's core mission is to efficiently accelerate AI‑related neural network algorithms, particularly in machine vision, natural language processing, and other key scenarios. As AI applications continue to expand, RKNPU‑based solutions now cover a wide range of fields, including face tracking, gesture and body pose tracking, image classification, video surveillance, automatic speech recognition (ASR), and advanced driver‑assistance systems (ADAS), meeting diverse AI deployment needs.

To help developers build AI projects on RKNPU, Rockchip provides a full suite of RKNN components that cover the entire development, deployment, and optimisation workflow, including RKNPU2, RKNN Toolkit2, RKLLM‑Toolkit, and the RKNPU driver. Each component plays a distinct and complementary role:

  • RKNPU2 — Core part of the development kit. Provides runtime libraries such as librknnrt.so and supports C/C++ programming interfaces. Can be used directly for RKNN model deployment and inference on Linux and Android systems, delivering the underlying runtime foundation for AI applications.
  • RKNN Toolkit2 — Python‑based development kit supporting both x86 and arm64 platforms. Integrates a complete set of features including model conversion, quantisation, inference testing, performance and memory evaluation, quantisation accuracy analysis, and model encryption. The included RKNN Toolkit Lite2 sub‑module allows RKNN models to be deployed directly on Rockchip boards, simplifying on‑board development.
  • RKNN Model — Rockchip's proprietary model format tailored to its NPU hardware architecture. Compared to standard model formats, RKNN models achieve higher inference performance and runtime efficiency on Rockchip NPUs, serving as the core carrier of RKNPU acceleration capabilities.
  • RKLLM‑Toolkit — Dedicated toolchain for large language model (LLM) development. Supports quantisation and conversion of Hugging Face‑format LLMs on a PC, producing RKLLM models optimised for RKNPU deployment, facilitating rapid LLM deployment on Rockchip platforms.
  • RKNPU Driver — Provides the interface between NPU hardware and the system. The Rockchip board firmware already includes the driver, so developers can start AI project development directly on the existing firmware without additional driver development, lowering the deployment barrier.

Additionally, Rockchip provides the RKNN Model Zoo repository, which is developed on top of the RKNN toolchain and integrates complete deployment examples for mainstream AI algorithms. It supports both Python API and C API development interfaces. Developers can quickly get started by referring to these examples, significantly shortening the development cycle for AI projects.

The system supports deployment of rknn-toolkit2, rknn_model_zoo, and rknn-llm. Click the links to access the official Rockchip online documentation.

PC‑Side NPU Deployment

rknn-toolkit2 Environment Setup

Download RKNN Repositories

Repositories can be downloaded via the command line or from the OneDrive.

# Create a Projects folder
mkdir Projects

# Enter the directory
cd Projects

# Download RKNN-Toolkit2 repository
git clone https://github.com/airockchip/rknn-toolkit2.git --depth 1

# Download RKNN Model Zoo repository
git clone https://github.com/airockchip/rknn_model_zoo.git --depth 1

# Notes:
# 1. The --depth 1 parameter clones only the latest commit.
# 2. If git clone fails, you can also download the zip archive directly from GitHub and extract it to this directory.
Install Miniforge Conda
conda -V
# Expected output: conda 23.3.1, indicating Miniforge conda version 23.3.1
# If conda: command not found, Miniforge is not installed

wget -c https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh

chmod 777 Miniforge3-Linux-x86_64.sh
bash Miniforge3-Linux-x86_64.sh

source ~/miniforge3/bin/activate # Directory where Miniforge is installed
# After success, the prompt becomes:
# (base) xxx@xxx:~$

conda create -n toolkit2 python=3.8

conda activate toolkit2
# After success, the prompt becomes:
# (toolkit2) xxx@xxx:~$
Install RKNN‑Toolkit2

Install via pip:

pip install rknn-toolkit2 -i https://pypi.org/simple

# If RKNN-Toolkit2 is already installed, upgrade it with:
pip install rknn-toolkit2 -i https://pypi.org/simple --upgrade

Install using a local wheel package:

# Enter the rknn-toolkit2 directory
cd Projects/rknn-toolkit2/rknn-toolkit2

# Select the appropriate requirements file based on your Python version and architecture:
# cpxx is the Python version number
pip install -r packages/x86_64/requirements_cpxx.txt
# pip install -r packages/arm64/arm64_requirements_cpxx.txt

# Install RKNN-Toolkit2
# Select the appropriate wheel package file based on your Python version and architecture:
# x.x.x is the RKNN-Toolkit2 version, cpxx is the Python version number
pip install packages/x86_64/rknn_toolkit2-x.x.x-cpxx-cpxx-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
# pip install packages/arm64/rknn_toolkit2-x.x.x-cpxx-cpxx-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Verify the installation. If installation fails, refer to Section 10.2 ("Tool Installation Issues") in the Rockchip_RKNPU_User_Guide_RKNN_SDK_CN.pdf document, which provides detailed troubleshooting steps for RKNN‑Toolkit2 environment setup issues.

# Enter Python interactive mode
python
# Import the RKNN class
from rknn.api import RKNN

RKNN Model Conversion

# Enter the rknn_model_zoo/examples/yolov5/model directory
cd Projects/rknn_model_zoo/examples/yolov5/model

# Run the download_model.sh script to download the YOLOv5 ONNX model
# The downloaded ONNX model will be saved to model/yolov5s_relu.onnx
./download_model.sh

# Enter the rknn_model_zoo/examples/yolov5/python directory
cd Projects/rknn_model_zoo/examples/yolov5/python

# Run convert.py to convert the ONNX model to RKNN format
# Usage: python convert.py model_path [rk3566|rk3588|rk3562] [i8/fp] [output_path]
python convert.py ../model/yolov5s_relu.onnx rk3588 i8 ../model/yolov5s_relu.rknn

Cross‑Compiling rknn_model_zoo

Install Compilation Dependencies
sudo apt install libopencv-dev git cmake make gcc g++ libsndfile1-dev -y
Download GCC Cross‑Compilation Toolchain

For 64‑bit target systems:

https://releases.linaro.org/components/toolchain/binaries/6.3-2017.05/aarch64-linux-gnu/gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu.tar.xz

For 32‑bit target systems:

https://releases.linaro.org/components/toolchain/binaries/6.3-2017.05/arm-linuxgnueabihf/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf.tar.xz

# Directory structure
Projects
├── rknn-toolkit2
├── rknn_model_zoo
└── gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu # This path is used when compiling RKNN C demos
Cross‑Compile
# Add this to the beginning of the build-linux.sh script
GCC_COMPILER=Projects/gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu

# Enter the rknn_model_zoo directory
cd Projects/rknn_model_zoo

# Run the build-linux.sh script
# Usage: ./build-linux.sh -t <target> -a <arch> -d <build_demo_name> [-b <build_type>] [-m]
# -t : target (rk356x/rk3588) # Platform type; rk3568 and rk3566 are both grouped under rk356x
# -a : arch (aarch64/armhf)   # Target system architecture
# -d : demo name              # Corresponds to a subfolder under examples/, e.g., yolov5, mobilenet
# -b : build_type (Debug/Release)
# -m : enable address sanitizer, build_type must be set to Debug
./build-linux.sh -t rk356x -a aarch64 -d yolov5

Board‑Side NPU Deployment

If the board does not have the RKNN Server and Runtime libraries installed, or if their versions are inconsistent, the RKNPU2 environment needs to be reinstalled.

Typically, the development board comes with a consistent version of the RKNPU2 environment pre‑installed.

(Note: 1. If using an RKNN model with dynamic input dimensions, the RKNN Server and Runtime library versions must be >= 1.5.0. 2. The RKNN Server, Runtime library, and RKNN‑Toolkit2 versions must all be consistent; it is recommended to install the latest versions.)

Verify the RKNPU2 Environment

# Query RKNPU2 driver version
dmesg | grep -i rknpu

# Start rknn_server
restart_rknn.sh
# "start rknn server, version: x.x.x" indicates successful startup, meaning the RKNPU2 environment is installed

# Query rknn_server version
strings /usr/bin/rknn_server | grep -i "rknn_server version"

# Query librknnrt.so library version
strings /usr/lib/librknnrt.so | grep -i "librknnrt version"

Install / Update the RKNPU2 Environment

# rknpu2 directory
Projects/rknn-toolkit2/rknpu2

# Copy rknn_server to the board
# Note: For 64‑bit Linux, BOARD_ARCH corresponds to aarch64; for 32‑bit systems, it corresponds to armhf.
Projects/rknn-toolkit2/rknpu2/runtime/Linux/rknn_server/${BOARD_ARCH}/usr/bin/* /usr/bin

# Copy librknnrt.so to the board
Projects/rknn-toolkit2/rknpu2/runtime/Linux/librknn_api/${BOARD_ARCH}/librknnrt.so /usr/lib

# Set executable permissions
chmod +x /usr/bin/rknn_server
chmod +x /usr/bin/start_rknn.sh
chmod +x /usr/bin/restart_rknn.sh

# Restart rknn_server
restart_rknn.sh

Native Compilation of rknn_model_zoo on the Board

Install compilation dependencies:

sudo apt install libopencv-dev git cmake make gcc g++ libsndfile1-dev -y

Compile:

# Enter the rknn_model_zoo directory
cd rknn_model_zoo

# Run the build-linux.sh script
# Usage: ./build-linux.sh -t <target> -a <arch> -d <build_demo_name> [-b <build_type>] [-m]
# -t : target (rk356x/rk3588) # Platform type; rk3568 and rk3566 are both grouped under rk356x
# -a : arch (aarch64/armhf)   # Target system architecture
# -d : demo name              # Corresponds to a subfolder under examples/, e.g., yolov5, mobilenet
# -b : build_type (Debug/Release)
# -m : enable address sanitizer, build_type must be set to Debug
./build-linux.sh -t rk356x -a aarch64 -d yolov5