Linux DeepSeek Model Deployment¶

Motherboard	SoC	Platform
K7 / K7C / K7S	RK3576	Rockchip
K8 / K8D	RK3588	Rockchip

Rockchip Platform¶

This document describes how to deploy the distilled DeepSeek-R1-Distill-Qwen-1.5B model to Rockchip platforms using RKLLM, with NPU-accelerated inference.

Chip platforms: RK3576 / RK3588

System versions: Debian 12 / Debian 11

Development Environment Setup¶

RKLLM SDK Documentation¶

doc
└── Rockchip_RKLLM_SDK_CN.pdf

RKNPU Driver¶

rknpu-driver
└── rknpu_driver_0.9.6_20240322.tar.bz2

RKLLM-Toolkit¶

rkllm-toolkit
├── examples
│   └── huggingface
│       └── test.py
├── packages
│   └── md5sum.txt
│   └── rkllm_toolkit-1.0.0-cp38-cp38-linux_x86_64.whl

Runtime Model Download¶

rkllm-runtime
├── example
│   └── src
│       └── main.cpp
│       └── build-android.sh
│       └── build-linux.sh
│       └── CMakeLists.txt
│       └── Readme.md
├── runtime
│   └── Android
│       └── librkllm_api
│           └── arm64-v8a
│               └── librkllmrt.so      # RKLLM Runtime library
│           └── include
│               └── rkllm.h            # Runtime header file
│   └── Linux
│       └── librkllm_api
│           └── aarch64
│               └── librkllmrt.so
│           └── include
│               └── rkllm.h

Deployment and Execution¶

Target Board Environment Setup¶

1. Obtain the rk35xx-rkllm-deepseek.tar.gz package.

$ ls /rkllm-DeepSeek/
    DeepSeek-R1-Distill-Qwen-1.5B.rkllm
    lib/librkllmrt.so
    llm_demo

Note

Test package contents:
DeepSeek-R1-Distill-Qwen-1.5B.rkllm is the converted model file.
llm_demo is the compiled LLM test program.

2. Copy the test package to the RK3588 board and set the environment variable:

$ export LD_LIBRARY_PATH=./lib

Running the Test¶

$ ./llm_demo DeepSeek-R1-Distill-Qwen-1.5B.rkllm 10000 10000

Performance Analysis¶

For the math problem: Solve the equations x+y=12, 2x+4y=34, find the values of x and y, the RK3588 achieves 14.93 tokens per second. Set export RKLLM_LOG_LEVEL=1 for detailed logging.

Phase	Total Time (ms)	Tokens	Time per Token (ms)	Tokens per Second
Prefill	429.63	81	5.30	188.53
Generation	56103.71	851	66.99	14.93