Instructions to use AXERA-TECH/Qwen2.5-3B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AXERA-TECH/Qwen2.5-3B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AXERA-TECH/Qwen2.5-3B-Instruct")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("AXERA-TECH/Qwen2.5-3B-Instruct", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AXERA-TECH/Qwen2.5-3B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AXERA-TECH/Qwen2.5-3B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen2.5-3B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/AXERA-TECH/Qwen2.5-3B-Instruct

SGLang

How to use AXERA-TECH/Qwen2.5-3B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AXERA-TECH/Qwen2.5-3B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen2.5-3B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AXERA-TECH/Qwen2.5-3B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AXERA-TECH/Qwen2.5-3B-Instruct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use AXERA-TECH/Qwen2.5-3B-Instruct with Docker Model Runner:
```
docker model run hf.co/AXERA-TECH/Qwen2.5-3B-Instruct
```

Qwen2.5-3B-Instruct

This version of Qwen2.5-3B-Instruct has been converted to run on the Axera NPU using w8a16 and w4a16 quantization.

This model has been optimized with the following LoRA:

Compatible with Pulsar2 version: 4.1

Feature

Support for longer contexts, in this sample it's 2k
Support context dialogue
System prompt kvcache is supported

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo

Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

AXera NPU AXEngine LLM Runtime

AXera NPU AXCL LLM Runtime

Convert script

The follow show how to convert Qwen2.5-3B-Instruct-GPTQ-Int8

pulsar2 llm_build --input_path Qwen/Qwen2.5-3B-Instruct-GPTQ-Int8  \
                  --output_path Qwen/Qwen2.5-3B-Instruct-GPTQ-Int8-ctx-ax650 \
                  --hidden_state_type bf16 --kv_cache_len 2047 --prefill_len 128 \
                  --last_kv_cache_len 128 \
                  --last_kv_cache_len 256 \
                  --last_kv_cache_len 384 \
                  --last_kv_cache_len 512 \
                  --last_kv_cache_len 640 \
                  --last_kv_cache_len 768 \
                  --last_kv_cache_len 896 \
                  --last_kv_cache_len 1024 \
                  --chip AX650 -c 1 --parallel 8

Support Platform

AX650
- AX650N DEMO Board
- M4N-Dock(爱芯派Pro)
- M.2 Accelerator card

Chips	w8a16	w4a16	DDR(w8)	Flash(w8)	DDR(w4)	Flash(w4)
AX650					2.8GB	2.9GB

How to use

推荐使用 ax-llm 项目提供的安装脚本在板端直接完成安装:

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

Inference

root@ax650:/mnt/sdcard/sunyijie/projects/llm/Qwen2.5-3B-Instruct/qwen2.5-3b-ctx-int4-ax650# axllm run ./
15:10:16.789 INF Init:2700 | LLM init start
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
 97% | ###############################  |  38 /  39 [35.22s<36.14s, 1.08 count/s] init post axmodel ok,remain_cmm(7450 MB)
15:10:52.011 INF Init:3023 | max_token_len : 2047
15:10:52.011 INF Init:3026 | kv_cache_size : 256, kv_cache_num: 2047
15:10:52.011 INF init_groups_from_model:1827 | prefill_token_num : 128
15:10:52.011 INF init_groups_from_model:2075 | decode grp: 0, gid: 0, max_token_len : 2047
15:10:52.011 INF init_groups_from_model:2079 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 128, symbolic_cap: 1
15:10:52.011 INF init_groups_from_model:2079 | prefill grp: 1, gid: 2, history_cap: 128, total_cap: 256, symbolic_cap: 128
15:10:52.012 INF init_groups_from_model:2079 | prefill grp: 2, gid: 3, history_cap: 256, total_cap: 384, symbolic_cap: 256
15:10:52.012 INF init_groups_from_model:2079 | prefill grp: 3, gid: 4, history_cap: 384, total_cap: 512, symbolic_cap: 384
15:10:52.012 INF init_groups_from_model:2079 | prefill grp: 4, gid: 5, history_cap: 512, total_cap: 640, symbolic_cap: 512
15:10:52.012 INF init_groups_from_model:2079 | prefill grp: 5, gid: 6, history_cap: 640, total_cap: 768, symbolic_cap: 640
15:10:52.012 INF init_groups_from_model:2079 | prefill grp: 6, gid: 7, history_cap: 768, total_cap: 896, symbolic_cap: 768
15:10:52.012 INF init_groups_from_model:2079 | prefill grp: 7, gid: 8, history_cap: 896, total_cap: 1024, symbolic_cap: 896
15:10:52.012 INF init_groups_from_model:2079 | prefill grp: 8, gid: 9, history_cap: 1024, total_cap: 1152, symbolic_cap: 1024
15:10:52.012 INF init_groups_from_model:2086 | prefill_max_token_num : 1152
15:10:52.012 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  39 /  39 [35.22s<35.22s, 1.11 count/s] embed_selector init ok
15:10:52.013 INF load_config:554 | load config:
15:10:52.013 INF load_config:554 | {
15:10:52.013 INF load_config:554 |     "enable_repetition_penalty": false,
15:10:52.013 INF load_config:554 |     "enable_temperature": true,
15:10:52.013 INF load_config:554 |     "enable_top_k_sampling": true,
15:10:52.013 INF load_config:554 |     "enable_top_p_sampling": false,
15:10:52.013 INF load_config:554 |     "penalty_window": 20,
15:10:52.013 INF load_config:554 |     "repetition_penalty": 1.2,
15:10:52.013 INF load_config:554 |     "temperature": 0.9,
15:10:52.013 INF load_config:554 |     "top_k": 10,
15:10:52.013 INF load_config:554 |     "top_p": 0.8
15:10:52.013 INF load_config:554 | }
15:10:52.013 INF Init:3138 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
----------------------------------------
prompt >> hello
15:11:27.211 INF SetKVCache:3721 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:20 prefer_symbolic_group:0
15:11:27.211 INF SetKVCache:3743 | current prefill_max_token_num:1152
15:11:27.242 INF SetKVCache:3759 | first run
15:11:27.268 INF Run:4287 | input token num : 20, prefill_split_num : 1
15:11:27.268 INF Run:4367 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=20
15:11:27.603 INF Run:4680 | ttft: 335.08 ms
Hello there!!! How can I assist you today?

15:11:28.642 NTC Run:5095 | hit eos,decode avg 8.67 token/s
15:11:28.642 INF GetKVCache:3698 | precompute_len:30, remaining:1122 (tracked)

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for AXERA-TECH/Qwen2.5-3B-Instruct

Base model

Qwen/Qwen2.5-3B

Finetuned

Qwen/Qwen2.5-3B-Instruct

Finetuned

(1667)

this model

Collection including AXERA-TECH/Qwen2.5-3B-Instruct

Qwen2.5

Collection

11 items • Updated Jun 8