Hugging Face's logo

AlexWortega
/
openjev

Text Classification
Transformers
Safetensors
English
nli
cross-encoder
qwen3.5
reranker
image-text-to-text
Model card Files
xet
 Community
5
AlexWortega commited on
Commit
058a6c2
·
verified ·
1 Parent(s): 552759d

add qwen3.5-4b-nli-v5-nvfp4 (ModelOpt NVFP4 PTQ + QAD) with SGLang bench

Browse files
Files changed (18) hide show
  1. .gitattributes +1 -0
  2. qwen3.5-4b-nli-v5-nvfp4/README.md +54 -0
  3. qwen3.5-4b-nli-v5-nvfp4/bench/ptq_4bv5_nvfp4.json +53 -0
  4. qwen3.5-4b-nli-v5-nvfp4/bench/ptq_4bv5_nvfp4_awq.json +53 -0
  5. qwen3.5-4b-nli-v5-nvfp4/bench/qad_4bv5_nvfp4.json +125 -0
  6. qwen3.5-4b-nli-v5-nvfp4/bench/sgl_acc_bf16.json +60 -0
  7. qwen3.5-4b-nli-v5-nvfp4/bench/sgl_acc_fp8.json +60 -0
  8. qwen3.5-4b-nli-v5-nvfp4/bench/sgl_acc_nvfp4.json +60 -0
  9. qwen3.5-4b-nli-v5-nvfp4/bench/sgl_speed_bf16.json +34 -0
  10. qwen3.5-4b-nli-v5-nvfp4/bench/sgl_speed_fp8.json +34 -0
  11. qwen3.5-4b-nli-v5-nvfp4/bench/sgl_speed_nvfp4.json +34 -0
  12. qwen3.5-4b-nli-v5-nvfp4/chat_template.jinja +154 -0
  13. qwen3.5-4b-nli-v5-nvfp4/config.json +230 -0
  14. qwen3.5-4b-nli-v5-nvfp4/hf_quant_config.json +88 -0
  15. qwen3.5-4b-nli-v5-nvfp4/model.safetensors +3 -0
  16. qwen3.5-4b-nli-v5-nvfp4/preprocessor_config.json +21 -0
  17. qwen3.5-4b-nli-v5-nvfp4/tokenizer.json +3 -0
  18. qwen3.5-4b-nli-v5-nvfp4/tokenizer_config.json +36 -0
.gitattributes CHANGED
@@ -56,3 +56,4 @@ qwen3.5-0.8b-nli-v2s-long/tokenizer.json filter=lfs diff=lfs merge=lfs -text
56
  qwen3.5-4b-nli-v5/tokenizer.json filter=lfs diff=lfs merge=lfs -text
57
  qwen3.5-2b-nli-v5/tokenizer.json filter=lfs diff=lfs merge=lfs -text
58
  qwen3.5-0.8b-nli-v5/tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
56
  qwen3.5-4b-nli-v5/tokenizer.json filter=lfs diff=lfs merge=lfs -text
57
  qwen3.5-2b-nli-v5/tokenizer.json filter=lfs diff=lfs merge=lfs -text
58
  qwen3.5-0.8b-nli-v5/tokenizer.json filter=lfs diff=lfs merge=lfs -text
59
+ qwen3.5-4b-nli-v5-nvfp4/tokenizer.json filter=lfs diff=lfs merge=lfs -text
qwen3.5-4b-nli-v5-nvfp4/README.md ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # openjev qwen3.5-4b-nli-v5 — NVFP4
2
+
3
+ NVFP4 (W4A4, FP8 block scales, group 16) version of [`qwen3.5-4b-nli-v5`](../qwen3.5-4b-nli-v5), made with
4
+ [NVIDIA Model-Optimizer](https://github.com/NVIDIA/Model-Optimizer) 0.47 and exported with `export_hf_checkpoint`
5
+ (`quant_method: modelopt`). The 3-way `score` head, embeddings, vision tower and the GDN `in_proj_a/b` / `conv1d`
6
+ stay in bf16. Size 3.95 GB vs 9.1 GB for bf16. Same template and label order as the bf16 model:
7
+ `"Premise: {premise}\nHypothesis: {hypothesis}"` -> `[contradiction, entailment, neutral]`.
8
+
9
+ ## How it was made
10
+ 1. PTQ: `mtq.quantize` with `NVFP4_DEFAULT_CFG` (+ `score` excluded), max calibration on 512 rows of the v5 training mix.
11
+ 2. QAD (quantization-aware distillation, the Model-Optimizer `llm_qat` recipe adapted to a seq-cls head): the NVFP4
12
+ student is trained on 30k v5-mix rows with KL(teacher || student) over the 3 logits, teacher = the bf16 v5 model;
13
+ lr 1e-5 cosine, bs 16, 1 epoch, scales frozen. ~40 min on one RTX PRO 6000.
14
+ 3. Export to packed NVFP4.
15
+
16
+ ## Accuracy
17
+ Served with SGLang on real FP4 kernels (RTX PRO 6000 Blackwell), same server flags for all rows:
18
+
19
+ | task | bf16 | fp8 (on the fly) | **NVFP4 (this)** |
20
+ |---|---|---|---|
21
+ | MNLI m+mm | 0.898 | 0.898 | 0.894 |
22
+ | ANLI r1 / r2 / r3 | 0.782 / 0.665 / 0.626 | 0.776 / 0.664 / 0.629 | 0.771 / 0.675 / 0.618 |
23
+ | WANLI | 0.766 | 0.768 | 0.761 |
24
+ | SciTail | 0.952 | 0.952 | 0.948 |
25
+ | ConTRoL | 0.735 | 0.735 | 0.722 |
26
+
27
+ Plain PTQ without QAD loses up to 4 points on the hard sets (ANLI r3 0.585, ConTRoL 0.698); QAD recovers most of it.
28
+
29
+ ## Speed
30
+ SGLang 0.5.18, one RTX PRO 6000 Blackwell, `--chunked-prefill-size 16384 --max-prefill-tokens 65536 --disable-radix-cache`,
31
+ saturated throughput on random token ids (tok/s):
32
+
33
+ | input length | bf16 | fp8 | **NVFP4** |
34
+ |---|---|---|---|
35
+ | 64 | 36.1k | 48.9k | **64.3k** |
36
+ | 256 | 37.7k | 51.3k | **68.8k** |
37
+ | 1024 | 37.5k | 51.4k | **69.0k** |
38
+ | 4096 | 36.0k | 48.5k | **63.9k** |
39
+ | 16384 | 30.7k | 39.2k | **48.7k** |
40
+
41
+ p50 latency of a single request at 4k / 16k tokens: bf16 119 / 547 ms, fp8 86 / 434 ms, NVFP4 67 / 351 ms.
42
+ Raw numbers: `bench/`.
43
+
44
+ ## Serving
45
+ Needs a Blackwell GPU (sm100 / sm120). With SGLang and the openjev external model package (`code/sglang_openjev`):
46
+
47
+ ```bash
48
+ SGLANG_EXTERNAL_MODEL_PACKAGE=sglang_openjev python -m sglang.launch_server --model-path qwen3.5-4b-nli-v5-nvfp4 \
49
+ --is-embedding --json-model-override-args '{"architectures": ["Qwen3_5ForConditionalGeneration"]}' \
50
+ --attention-backend triton
51
+ ```
52
+
53
+ SGLang picks up `modelopt_fp4` from `hf_quant_config.json`. flashinfer JIT-compiles the FP4 activation-quantize kernel on
54
+ first start, so `CUDA_HOME` must point to a CUDA toolkit whose nvcc matches its headers (13.0 worked).
qwen3.5-4b-nli-v5-nvfp4/bench/ptq_4bv5_nvfp4.json ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cfg": "nvfp4",
3
+ "n_calib": 512,
4
+ "model": "../ckpt/hf/qwen3.5-4b-nli-v5",
5
+ "val_nli_v5": {
6
+ "argmax_agree": 0.942,
7
+ "mean_abs_dp": 0.04296731576323509,
8
+ "max_abs_dp": 0.8394484519958496,
9
+ "acc": 0.916,
10
+ "acc_bf16": 0.942
11
+ },
12
+ "mnli": {
13
+ "validation_matched": {
14
+ "acc": 0.8908813041263373,
15
+ "n": 9815
16
+ },
17
+ "validation_mismatched": {
18
+ "acc": 0.8945280716029292,
19
+ "n": 9832
20
+ }
21
+ },
22
+ "anli_r1": {
23
+ "acc": 0.757,
24
+ "n": 1000,
25
+ "two_class": false
26
+ },
27
+ "anli_r2": {
28
+ "acc": 0.651,
29
+ "n": 1000,
30
+ "two_class": false
31
+ },
32
+ "anli_r3": {
33
+ "acc": 0.585,
34
+ "n": 1200,
35
+ "two_class": false
36
+ },
37
+ "wanli": {
38
+ "acc": 0.765,
39
+ "n": 5000,
40
+ "two_class": false
41
+ },
42
+ "scitail": {
43
+ "acc": 0.9539040451552211,
44
+ "n": 2126,
45
+ "two_class": true
46
+ },
47
+ "control": {
48
+ "acc": 0.698136645962733,
49
+ "n": 805,
50
+ "two_class": false
51
+ },
52
+ "export_error": "RuntimeError: Invalid device string: 'bfloat16'"
53
+ }
qwen3.5-4b-nli-v5-nvfp4/bench/ptq_4bv5_nvfp4_awq.json ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cfg": "nvfp4_awq",
3
+ "n_calib": 512,
4
+ "model": "../ckpt/hf/qwen3.5-4b-nli-v5",
5
+ "val_nli_v5": {
6
+ "argmax_agree": 0.959,
7
+ "mean_abs_dp": 0.03219675272703171,
8
+ "max_abs_dp": 0.9887496829032898,
9
+ "acc": 0.925,
10
+ "acc_bf16": 0.942
11
+ },
12
+ "mnli": {
13
+ "validation_matched": {
14
+ "acc": 0.8986245542536934,
15
+ "n": 9815
16
+ },
17
+ "validation_mismatched": {
18
+ "acc": 0.9006305939788446,
19
+ "n": 9832
20
+ }
21
+ },
22
+ "anli_r1": {
23
+ "acc": 0.759,
24
+ "n": 1000,
25
+ "two_class": false
26
+ },
27
+ "anli_r2": {
28
+ "acc": 0.647,
29
+ "n": 1000,
30
+ "two_class": false
31
+ },
32
+ "anli_r3": {
33
+ "acc": 0.6075,
34
+ "n": 1200,
35
+ "two_class": false
36
+ },
37
+ "wanli": {
38
+ "acc": 0.7608,
39
+ "n": 5000,
40
+ "two_class": false
41
+ },
42
+ "scitail": {
43
+ "acc": 0.955785512699906,
44
+ "n": 2126,
45
+ "two_class": true
46
+ },
47
+ "control": {
48
+ "acc": 0.7043478260869566,
49
+ "n": 805,
50
+ "two_class": false
51
+ },
52
+ "export_error": "RuntimeError: Invalid device string: 'bfloat16'"
53
+ }
qwen3.5-4b-nli-v5-nvfp4/bench/qad_4bv5_nvfp4.json ADDED
@@ -0,0 +1,125 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cfg": "nvfp4",
3
+ "args": {
4
+ "model": "../ckpt/hf/qwen3.5-4b-nli-v5",
5
+ "out": "../results/qad_4bv5_nvfp4.json",
6
+ "save": "../ckpt/4b-v5-nvfp4-qad",
7
+ "cfg": "nvfp4",
8
+ "data": "data/nli_v5_sample60k.jsonl.gz",
9
+ "val": "data/nli_v5_val.jsonl.gz",
10
+ "n_calib": 512,
11
+ "n_train": 30000,
12
+ "bs": 16,
13
+ "lr": 1e-05,
14
+ "warmup": 0.05,
15
+ "temp": 1.0,
16
+ "ce_weight": 0.0,
17
+ "max_len": 1024,
18
+ "eval_every": 500,
19
+ "tasks": [
20
+ "mnli",
21
+ "anli_r1",
22
+ "anli_r2",
23
+ "anli_r3",
24
+ "wanli",
25
+ "scitail",
26
+ "control"
27
+ ],
28
+ "eval_bs": 8
29
+ },
30
+ "curve": [
31
+ {
32
+ "step": 500,
33
+ "loss": 0.035344036804227014,
34
+ "argmax_agree": 0.9645,
35
+ "mean_abs_dp": 0.022110234946012497,
36
+ "max_abs_dp": 0.9942297339439392,
37
+ "acc": 0.9255,
38
+ "acc_bf16": 0.942
39
+ },
40
+ {
41
+ "step": 1000,
42
+ "loss": 0.03905459138098649,
43
+ "argmax_agree": 0.9755,
44
+ "mean_abs_dp": 0.017070824280381203,
45
+ "max_abs_dp": 0.9948465824127197,
46
+ "acc": 0.9325,
47
+ "acc_bf16": 0.942
48
+ },
49
+ {
50
+ "step": 1500,
51
+ "loss": 0.02403435970255259,
52
+ "argmax_agree": 0.977,
53
+ "mean_abs_dp": 0.016855977475643158,
54
+ "max_abs_dp": 0.9949026703834534,
55
+ "acc": 0.933,
56
+ "acc_bf16": 0.942
57
+ },
58
+ {
59
+ "step": 1875,
60
+ "loss": 0.03303498500937053,
61
+ "argmax_agree": 0.973,
62
+ "mean_abs_dp": 0.01758590154349804,
63
+ "max_abs_dp": 0.9951850175857544,
64
+ "acc": 0.931,
65
+ "acc_bf16": 0.942
66
+ }
67
+ ],
68
+ "ptq_val": {
69
+ "argmax_agree": 0.942,
70
+ "mean_abs_dp": 0.04296731576323509,
71
+ "max_abs_dp": 0.8394484519958496,
72
+ "acc": 0.916,
73
+ "acc_bf16": 0.942
74
+ },
75
+ "qad_val": {
76
+ "step": 1875,
77
+ "loss": 0.03303498500937053,
78
+ "argmax_agree": 0.973,
79
+ "mean_abs_dp": 0.01758590154349804,
80
+ "max_abs_dp": 0.9951850175857544,
81
+ "acc": 0.931,
82
+ "acc_bf16": 0.942
83
+ },
84
+ "mnli": {
85
+ "validation_matched": {
86
+ "acc": 0.8946510443199185,
87
+ "n": 9815
88
+ },
89
+ "validation_mismatched": {
90
+ "acc": 0.8951383238405207,
91
+ "n": 9832
92
+ }
93
+ },
94
+ "anli_r1": {
95
+ "acc": 0.783,
96
+ "n": 1000,
97
+ "two_class": false
98
+ },
99
+ "anli_r2": {
100
+ "acc": 0.676,
101
+ "n": 1000,
102
+ "two_class": false
103
+ },
104
+ "anli_r3": {
105
+ "acc": 0.6166666666666667,
106
+ "n": 1200,
107
+ "two_class": false
108
+ },
109
+ "wanli": {
110
+ "acc": 0.7656,
111
+ "n": 5000,
112
+ "two_class": false
113
+ },
114
+ "scitail": {
115
+ "acc": 0.951552210724365,
116
+ "n": 2126,
117
+ "two_class": true
118
+ },
119
+ "control": {
120
+ "acc": 0.7229813664596273,
121
+ "n": 805,
122
+ "two_class": false
123
+ },
124
+ "export_error": "RuntimeError: Invalid device string: 'bfloat16'"
125
+ }
qwen3.5-4b-nli-v5-nvfp4/bench/sgl_acc_bf16.json ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "sglang": {
3
+ "mnli": {
4
+ "acc": 0.8982020667540715,
5
+ "pairs": 19647,
6
+ "tokens": 882108,
7
+ "sec": 26.19,
8
+ "pairs_per_s": 750.0,
9
+ "tok_per_s": 33675
10
+ },
11
+ "anli_r1": {
12
+ "acc": 0.782,
13
+ "pairs": 1000,
14
+ "tokens": 105114,
15
+ "sec": 2.92,
16
+ "pairs_per_s": 342.9,
17
+ "tok_per_s": 36043
18
+ },
19
+ "anli_r2": {
20
+ "acc": 0.665,
21
+ "pairs": 1000,
22
+ "tokens": 103598,
23
+ "sec": 2.87,
24
+ "pairs_per_s": 348.3,
25
+ "tok_per_s": 36085
26
+ },
27
+ "anli_r3": {
28
+ "acc": 0.6258333333333334,
29
+ "pairs": 1200,
30
+ "tokens": 110413,
31
+ "sec": 3.1,
32
+ "pairs_per_s": 387.1,
33
+ "tok_per_s": 35617
34
+ },
35
+ "wanli": {
36
+ "acc": 0.7656,
37
+ "pairs": 5000,
38
+ "tokens": 201533,
39
+ "sec": 5.96,
40
+ "pairs_per_s": 839.6,
41
+ "tok_per_s": 33842
42
+ },
43
+ "scitail": {
44
+ "acc": 0.9520225776105362,
45
+ "pairs": 2126,
46
+ "tokens": 93280,
47
+ "sec": 2.74,
48
+ "pairs_per_s": 775.3,
49
+ "tok_per_s": 34018
50
+ },
51
+ "control": {
52
+ "acc": 0.7354037267080745,
53
+ "pairs": 805,
54
+ "tokens": 493350,
55
+ "sec": 13.18,
56
+ "pairs_per_s": 61.1,
57
+ "tok_per_s": 37436
58
+ }
59
+ }
60
+ }
qwen3.5-4b-nli-v5-nvfp4/bench/sgl_acc_fp8.json ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "sglang": {
3
+ "mnli": {
4
+ "acc": 0.8984046914293602,
5
+ "pairs": 19647,
6
+ "tokens": 882108,
7
+ "sec": 19.53,
8
+ "pairs_per_s": 1005.8,
9
+ "tok_per_s": 45158
10
+ },
11
+ "anli_r1": {
12
+ "acc": 0.776,
13
+ "pairs": 1000,
14
+ "tokens": 105114,
15
+ "sec": 2.19,
16
+ "pairs_per_s": 457.1,
17
+ "tok_per_s": 48047
18
+ },
19
+ "anli_r2": {
20
+ "acc": 0.664,
21
+ "pairs": 1000,
22
+ "tokens": 103598,
23
+ "sec": 2.13,
24
+ "pairs_per_s": 469.5,
25
+ "tok_per_s": 48644
26
+ },
27
+ "anli_r3": {
28
+ "acc": 0.6291666666666667,
29
+ "pairs": 1200,
30
+ "tokens": 110413,
31
+ "sec": 2.29,
32
+ "pairs_per_s": 524.0,
33
+ "tok_per_s": 48214
34
+ },
35
+ "wanli": {
36
+ "acc": 0.7684,
37
+ "pairs": 5000,
38
+ "tokens": 201533,
39
+ "sec": 4.4,
40
+ "pairs_per_s": 1135.2,
41
+ "tok_per_s": 45755
42
+ },
43
+ "scitail": {
44
+ "acc": 0.9524929444967074,
45
+ "pairs": 2126,
46
+ "tokens": 93280,
47
+ "sec": 2.04,
48
+ "pairs_per_s": 1041.0,
49
+ "tok_per_s": 45674
50
+ },
51
+ "control": {
52
+ "acc": 0.7354037267080745,
53
+ "pairs": 805,
54
+ "tokens": 493350,
55
+ "sec": 9.69,
56
+ "pairs_per_s": 83.1,
57
+ "tok_per_s": 50905
58
+ }
59
+ }
60
+ }
qwen3.5-4b-nli-v5-nvfp4/bench/sgl_acc_nvfp4.json ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "sglang": {
3
+ "mnli": {
4
+ "acc": 0.8938251571899507,
5
+ "pairs": 19647,
6
+ "tokens": 882108,
7
+ "sec": 16.17,
8
+ "pairs_per_s": 1215.1,
9
+ "tok_per_s": 54556
10
+ },
11
+ "anli_r1": {
12
+ "acc": 0.771,
13
+ "pairs": 1000,
14
+ "tokens": 105114,
15
+ "sec": 1.64,
16
+ "pairs_per_s": 608.2,
17
+ "tok_per_s": 63934
18
+ },
19
+ "anli_r2": {
20
+ "acc": 0.675,
21
+ "pairs": 1000,
22
+ "tokens": 103598,
23
+ "sec": 1.62,
24
+ "pairs_per_s": 618.9,
25
+ "tok_per_s": 64119
26
+ },
27
+ "anli_r3": {
28
+ "acc": 0.6175,
29
+ "pairs": 1200,
30
+ "tokens": 110413,
31
+ "sec": 1.75,
32
+ "pairs_per_s": 687.6,
33
+ "tok_per_s": 63264
34
+ },
35
+ "wanli": {
36
+ "acc": 0.761,
37
+ "pairs": 5000,
38
+ "tokens": 201533,
39
+ "sec": 3.42,
40
+ "pairs_per_s": 1460.2,
41
+ "tok_per_s": 58855
42
+ },
43
+ "scitail": {
44
+ "acc": 0.9482596425211665,
45
+ "pairs": 2126,
46
+ "tokens": 93280,
47
+ "sec": 1.59,
48
+ "pairs_per_s": 1335.5,
49
+ "tok_per_s": 58598
50
+ },
51
+ "control": {
52
+ "acc": 0.7217391304347827,
53
+ "pairs": 805,
54
+ "tokens": 493350,
55
+ "sec": 7.22,
56
+ "pairs_per_s": 111.5,
57
+ "tok_per_s": 68333
58
+ }
59
+ }
60
+ }
qwen3.5-4b-nli-v5-nvfp4/bench/sgl_speed_bf16.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "sglang": {
3
+ "64": {
4
+ "p50_ms": 23.8,
5
+ "p95_ms": 26.2,
6
+ "pairs_per_s": 563.3,
7
+ "tok_per_s": 36050
8
+ },
9
+ "256": {
10
+ "p50_ms": 24.8,
11
+ "p95_ms": 27.1,
12
+ "pairs_per_s": 147.1,
13
+ "tok_per_s": 37653
14
+ },
15
+ "1024": {
16
+ "p50_ms": 36.5,
17
+ "p95_ms": 37.7,
18
+ "pairs_per_s": 36.7,
19
+ "tok_per_s": 37545
20
+ },
21
+ "4096": {
22
+ "p50_ms": 118.9,
23
+ "p95_ms": 123.0,
24
+ "pairs_per_s": 8.8,
25
+ "tok_per_s": 35991
26
+ },
27
+ "16384": {
28
+ "p50_ms": 547.0,
29
+ "p95_ms": 554.9,
30
+ "pairs_per_s": 1.9,
31
+ "tok_per_s": 30686
32
+ }
33
+ }
34
+ }
qwen3.5-4b-nli-v5-nvfp4/bench/sgl_speed_fp8.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "sglang": {
3
+ "64": {
4
+ "p50_ms": 23.8,
5
+ "p95_ms": 24.6,
6
+ "pairs_per_s": 764.5,
7
+ "tok_per_s": 48927
8
+ },
9
+ "256": {
10
+ "p50_ms": 24.1,
11
+ "p95_ms": 24.5,
12
+ "pairs_per_s": 200.2,
13
+ "tok_per_s": 51263
14
+ },
15
+ "1024": {
16
+ "p50_ms": 27.6,
17
+ "p95_ms": 27.8,
18
+ "pairs_per_s": 50.2,
19
+ "tok_per_s": 51381
20
+ },
21
+ "4096": {
22
+ "p50_ms": 86.1,
23
+ "p95_ms": 86.4,
24
+ "pairs_per_s": 11.8,
25
+ "tok_per_s": 48510
26
+ },
27
+ "16384": {
28
+ "p50_ms": 433.5,
29
+ "p95_ms": 436.4,
30
+ "pairs_per_s": 2.4,
31
+ "tok_per_s": 39186
32
+ }
33
+ }
34
+ }
qwen3.5-4b-nli-v5-nvfp4/bench/sgl_speed_nvfp4.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "sglang": {
3
+ "64": {
4
+ "p50_ms": 24.6,
5
+ "p95_ms": 25.7,
6
+ "pairs_per_s": 1004.5,
7
+ "tok_per_s": 64291
8
+ },
9
+ "256": {
10
+ "p50_ms": 24.0,
11
+ "p95_ms": 24.9,
12
+ "pairs_per_s": 268.7,
13
+ "tok_per_s": 68775
14
+ },
15
+ "1024": {
16
+ "p50_ms": 25.5,
17
+ "p95_ms": 26.7,
18
+ "pairs_per_s": 67.4,
19
+ "tok_per_s": 68989
20
+ },
21
+ "4096": {
22
+ "p50_ms": 67.0,
23
+ "p95_ms": 67.4,
24
+ "pairs_per_s": 15.6,
25
+ "tok_per_s": 63940
26
+ },
27
+ "16384": {
28
+ "p50_ms": 350.6,
29
+ "p95_ms": 350.9,
30
+ "pairs_per_s": 3.0,
31
+ "tok_per_s": 48721
32
+ }
33
+ }
34
+ }
qwen3.5-4b-nli-v5-nvfp4/chat_template.jinja ADDED
@@ -0,0 +1,154 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- set image_count = namespace(value=0) %}
2
+ {%- set video_count = namespace(value=0) %}
3
+ {%- macro render_content(content, do_vision_count, is_system_content=false) %}
4
+ {%- if content is string %}
5
+ {{- content }}
6
+ {%- elif content is iterable and content is not mapping %}
7
+ {%- for item in content %}
8
+ {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
9
+ {%- if is_system_content %}
10
+ {{- raise_exception('System message cannot contain images.') }}
11
+ {%- endif %}
12
+ {%- if do_vision_count %}
13
+ {%- set image_count.value = image_count.value + 1 %}
14
+ {%- endif %}
15
+ {%- if add_vision_id %}
16
+ {{- 'Picture ' ~ image_count.value ~ ': ' }}
17
+ {%- endif %}
18
+ {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
19
+ {%- elif 'video' in item or item.type == 'video' %}
20
+ {%- if is_system_content %}
21
+ {{- raise_exception('System message cannot contain videos.') }}
22
+ {%- endif %}
23
+ {%- if do_vision_count %}
24
+ {%- set video_count.value = video_count.value + 1 %}
25
+ {%- endif %}
26
+ {%- if add_vision_id %}
27
+ {{- 'Video ' ~ video_count.value ~ ': ' }}
28
+ {%- endif %}
29
+ {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
30
+ {%- elif 'text' in item %}
31
+ {{- item.text }}
32
+ {%- else %}
33
+ {{- raise_exception('Unexpected item type in content.') }}
34
+ {%- endif %}
35
+ {%- endfor %}
36
+ {%- elif content is none or content is undefined %}
37
+ {{- '' }}
38
+ {%- else %}
39
+ {{- raise_exception('Unexpected content type.') }}
40
+ {%- endif %}
41
+ {%- endmacro %}
42
+ {%- if not messages %}
43
+ {{- raise_exception('No messages provided.') }}
44
+ {%- endif %}
45
+ {%- if tools and tools is iterable and tools is not mapping %}
46
+ {{- '<|im_start|>system\n' }}
47
+ {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
48
+ {%- for tool in tools %}
49
+ {{- "\n" }}
50
+ {{- tool | tojson }}
51
+ {%- endfor %}
52
+ {{- "\n</tools>" }}
53
+ {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
54
+ {%- if messages[0].role == 'system' %}
55
+ {%- set content = render_content(messages[0].content, false, true)|trim %}
56
+ {%- if content %}
57
+ {{- '\n\n' + content }}
58
+ {%- endif %}
59
+ {%- endif %}
60
+ {{- '<|im_end|>\n' }}
61
+ {%- else %}
62
+ {%- if messages[0].role == 'system' %}
63
+ {%- set content = render_content(messages[0].content, false, true)|trim %}
64
+ {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
65
+ {%- endif %}
66
+ {%- endif %}
67
+ {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
68
+ {%- for message in messages[::-1] %}
69
+ {%- set index = (messages|length - 1) - loop.index0 %}
70
+ {%- if ns.multi_step_tool and message.role == "user" %}
71
+ {%- set content = render_content(message.content, false)|trim %}
72
+ {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
73
+ {%- set ns.multi_step_tool = false %}
74
+ {%- set ns.last_query_index = index %}
75
+ {%- endif %}
76
+ {%- endif %}
77
+ {%- endfor %}
78
+ {%- if ns.multi_step_tool %}
79
+ {{- raise_exception('No user query found in messages.') }}
80
+ {%- endif %}
81
+ {%- for message in messages %}
82
+ {%- set content = render_content(message.content, true)|trim %}
83
+ {%- if message.role == "system" %}
84
+ {%- if not loop.first %}
85
+ {{- raise_exception('System message must be at the beginning.') }}
86
+ {%- endif %}
87
+ {%- elif message.role == "user" %}
88
+ {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
89
+ {%- elif message.role == "assistant" %}
90
+ {%- set reasoning_content = '' %}
91
+ {%- if message.reasoning_content is string %}
92
+ {%- set reasoning_content = message.reasoning_content %}
93
+ {%- else %}
94
+ {%- if '</think>' in content %}
95
+ {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
96
+ {%- set content = content.split('</think>')[-1].lstrip('\n') %}
97
+ {%- endif %}
98
+ {%- endif %}
99
+ {%- set reasoning_content = reasoning_content|trim %}
100
+ {%- if loop.index0 > ns.last_query_index %}
101
+ {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
102
+ {%- else %}
103
+ {{- '<|im_start|>' + message.role + '\n' + content }}
104
+ {%- endif %}
105
+ {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
106
+ {%- for tool_call in message.tool_calls %}
107
+ {%- if tool_call.function is defined %}
108
+ {%- set tool_call = tool_call.function %}
109
+ {%- endif %}
110
+ {%- if loop.first %}
111
+ {%- if content|trim %}
112
+ {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
113
+ {%- else %}
114
+ {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
115
+ {%- endif %}
116
+ {%- else %}
117
+ {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
118
+ {%- endif %}
119
+ {%- if tool_call.arguments is defined %}
120
+ {%- for args_name, args_value in tool_call.arguments|items %}
121
+ {{- '<parameter=' + args_name + '>\n' }}
122
+ {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
123
+ {{- args_value }}
124
+ {{- '\n</parameter>\n' }}
125
+ {%- endfor %}
126
+ {%- endif %}
127
+ {{- '</function>\n</tool_call>' }}
128
+ {%- endfor %}
129
+ {%- endif %}
130
+ {{- '<|im_end|>\n' }}
131
+ {%- elif message.role == "tool" %}
132
+ {%- if loop.previtem and loop.previtem.role != "tool" %}
133
+ {{- '<|im_start|>user' }}
134
+ {%- endif %}
135
+ {{- '\n<tool_response>\n' }}
136
+ {{- content }}
137
+ {{- '\n</tool_response>' }}
138
+ {%- if not loop.last and loop.nextitem.role != "tool" %}
139
+ {{- '<|im_end|>\n' }}
140
+ {%- elif loop.last %}
141
+ {{- '<|im_end|>\n' }}
142
+ {%- endif %}
143
+ {%- else %}
144
+ {{- raise_exception('Unexpected message role.') }}
145
+ {%- endif %}
146
+ {%- endfor %}
147
+ {%- if add_generation_prompt %}
148
+ {{- '<|im_start|>assistant\n' }}
149
+ {%- if enable_thinking is defined and enable_thinking is false %}
150
+ {{- '<think>\n\n</think>\n\n' }}
151
+ {%- else %}
152
+ {{- '<think>\n' }}
153
+ {%- endif %}
154
+ {%- endif %}
qwen3.5-4b-nli-v5-nvfp4/config.json ADDED
@@ -0,0 +1,230 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Qwen3_5ForSequenceClassification"
4
+ ],
5
+ "dtype": "bfloat16",
6
+ "id2label": {
7
+ "0": "contradiction",
8
+ "1": "entailment",
9
+ "2": "neutral"
10
+ },
11
+ "image_token_id": 248056,
12
+ "label2id": {
13
+ "contradiction": 0,
14
+ "entailment": 1,
15
+ "neutral": 2
16
+ },
17
+ "model_type": "qwen3_5",
18
+ "nli_template": "Premise: {premise}\nHypothesis: {hypothesis}",
19
+ "pad_token_id": 248044,
20
+ "problem_type": "single_label_classification",
21
+ "text_config": {
22
+ "attention_bias": false,
23
+ "attention_dropout": 0.0,
24
+ "attn_output_gate": true,
25
+ "bos_token_id": null,
26
+ "dtype": "bfloat16",
27
+ "eos_token_id": 248044,
28
+ "full_attention_interval": 4,
29
+ "head_dim": 256,
30
+ "hidden_act": "silu",
31
+ "hidden_size": 2560,
32
+ "initializer_range": 0.02,
33
+ "intermediate_size": 9216,
34
+ "layer_types": [
35
+ "linear_attention",
36
+ "linear_attention",
37
+ "linear_attention",
38
+ "full_attention",
39
+ "linear_attention",
40
+ "linear_attention",
41
+ "linear_attention",
42
+ "full_attention",
43
+ "linear_attention",
44
+ "linear_attention",
45
+ "linear_attention",
46
+ "full_attention",
47
+ "linear_attention",
48
+ "linear_attention",
49
+ "linear_attention",
50
+ "full_attention",
51
+ "linear_attention",
52
+ "linear_attention",
53
+ "linear_attention",
54
+ "full_attention",
55
+ "linear_attention",
56
+ "linear_attention",
57
+ "linear_attention",
58
+ "full_attention",
59
+ "linear_attention",
60
+ "linear_attention",
61
+ "linear_attention",
62
+ "full_attention",
63
+ "linear_attention",
64
+ "linear_attention",
65
+ "linear_attention",
66
+ "full_attention"
67
+ ],
68
+ "linear_conv_kernel_dim": 4,
69
+ "linear_key_head_dim": 128,
70
+ "linear_num_key_heads": 16,
71
+ "linear_num_value_heads": 32,
72
+ "linear_value_head_dim": 128,
73
+ "mamba_ssm_dtype": "float32",
74
+ "max_position_embeddings": 262144,
75
+ "mlp_only_layers": [],
76
+ "model_type": "qwen3_5_text",
77
+ "mtp_num_hidden_layers": 1,
78
+ "mtp_use_dedicated_embeddings": false,
79
+ "num_attention_heads": 16,
80
+ "num_hidden_layers": 32,
81
+ "num_key_value_heads": 4,
82
+ "pad_token_id": 248044,
83
+ "partial_rotary_factor": 0.25,
84
+ "rms_norm_eps": 1e-06,
85
+ "rope_parameters": {
86
+ "mrope_interleaved": true,
87
+ "mrope_section": [
88
+ 11,
89
+ 11,
90
+ 10
91
+ ],
92
+ "partial_rotary_factor": 0.25,
93
+ "rope_theta": 10000000,
94
+ "rope_type": "default"
95
+ },
96
+ "tie_word_embeddings": true,
97
+ "use_cache": true,
98
+ "vocab_size": 248320
99
+ },
100
+ "tie_word_embeddings": true,
101
+ "transformers_version": "5.17.0",
102
+ "video_token_id": 248057,
103
+ "vision_config": {
104
+ "deepstack_visual_indexes": [],
105
+ "depth": 24,
106
+ "dtype": "bfloat16",
107
+ "hidden_act": "gelu_pytorch_tanh",
108
+ "hidden_size": 1024,
109
+ "in_channels": 3,
110
+ "initializer_range": 0.02,
111
+ "intermediate_size": 4096,
112
+ "model_type": "qwen3_5_vision",
113
+ "num_heads": 16,
114
+ "num_position_embeddings": 2304,
115
+ "out_hidden_size": 2560,
116
+ "patch_size": 16,
117
+ "rope_parameters": {
118
+ "rope_theta": 10000.0,
119
+ "rope_type": "axial"
120
+ },
121
+ "spatial_merge_size": 2,
122
+ "temporal_patch_size": 2
123
+ },
124
+ "vision_end_token_id": 248054,
125
+ "vision_start_token_id": 248053,
126
+ "quantization_config": {
127
+ "config_groups": {
128
+ "group_0": {
129
+ "input_activations": {
130
+ "dynamic": false,
131
+ "num_bits": 4,
132
+ "type": "float",
133
+ "group_size": 16
134
+ },
135
+ "weights": {
136
+ "dynamic": false,
137
+ "num_bits": 4,
138
+ "type": "float",
139
+ "group_size": 16
140
+ },
141
+ "targets": [
142
+ "Linear"
143
+ ]
144
+ }
145
+ },
146
+ "ignore": [
147
+ "model.language_model.embed_tokens",
148
+ "model.language_model.layers.0.linear_attn.conv1d",
149
+ "model.language_model.layers.0.linear_attn.in_proj_a",
150
+ "model.language_model.layers.0.linear_attn.in_proj_b",
151
+ "model.language_model.layers.1.linear_attn.conv1d",
152
+ "model.language_model.layers.1.linear_attn.in_proj_a",
153
+ "model.language_model.layers.1.linear_attn.in_proj_b",
154
+ "model.language_model.layers.10.linear_attn.conv1d",
155
+ "model.language_model.layers.10.linear_attn.in_proj_a",
156
+ "model.language_model.layers.10.linear_attn.in_proj_b",
157
+ "model.language_model.layers.12.linear_attn.conv1d",
158
+ "model.language_model.layers.12.linear_attn.in_proj_a",
159
+ "model.language_model.layers.12.linear_attn.in_proj_b",
160
+ "model.language_model.layers.13.linear_attn.conv1d",
161
+ "model.language_model.layers.13.linear_attn.in_proj_a",
162
+ "model.language_model.layers.13.linear_attn.in_proj_b",
163
+ "model.language_model.layers.14.linear_attn.conv1d",
164
+ "model.language_model.layers.14.linear_attn.in_proj_a",
165
+ "model.language_model.layers.14.linear_attn.in_proj_b",
166
+ "model.language_model.layers.16.linear_attn.conv1d",
167
+ "model.language_model.layers.16.linear_attn.in_proj_a",
168
+ "model.language_model.layers.16.linear_attn.in_proj_b",
169
+ "model.language_model.layers.17.linear_attn.conv1d",
170
+ "model.language_model.layers.17.linear_attn.in_proj_a",
171
+ "model.language_model.layers.17.linear_attn.in_proj_b",
172
+ "model.language_model.layers.18.linear_attn.conv1d",
173
+ "model.language_model.layers.18.linear_attn.in_proj_a",
174
+ "model.language_model.layers.18.linear_attn.in_proj_b",
175
+ "model.language_model.layers.2.linear_attn.conv1d",
176
+ "model.language_model.layers.2.linear_attn.in_proj_a",
177
+ "model.language_model.layers.2.linear_attn.in_proj_b",
178
+ "model.language_model.layers.20.linear_attn.conv1d",
179
+ "model.language_model.layers.20.linear_attn.in_proj_a",
180
+ "model.language_model.layers.20.linear_attn.in_proj_b",
181
+ "model.language_model.layers.21.linear_attn.conv1d",
182
+ "model.language_model.layers.21.linear_attn.in_proj_a",
183
+ "model.language_model.layers.21.linear_attn.in_proj_b",
184
+ "model.language_model.layers.22.linear_attn.conv1d",
185
+ "model.language_model.layers.22.linear_attn.in_proj_a",
186
+ "model.language_model.layers.22.linear_attn.in_proj_b",
187
+ "model.language_model.layers.24.linear_attn.conv1d",
188
+ "model.language_model.layers.24.linear_attn.in_proj_a",
189
+ "model.language_model.layers.24.linear_attn.in_proj_b",
190
+ "model.language_model.layers.25.linear_attn.conv1d",
191
+ "model.language_model.layers.25.linear_attn.in_proj_a",
192
+ "model.language_model.layers.25.linear_attn.in_proj_b",
193
+ "model.language_model.layers.26.linear_attn.conv1d",
194
+ "model.language_model.layers.26.linear_attn.in_proj_a",
195
+ "model.language_model.layers.26.linear_attn.in_proj_b",
196
+ "model.language_model.layers.28.linear_attn.conv1d",
197
+ "model.language_model.layers.28.linear_attn.in_proj_a",
198
+ "model.language_model.layers.28.linear_attn.in_proj_b",
199
+ "model.language_model.layers.29.linear_attn.conv1d",
200
+ "model.language_model.layers.29.linear_attn.in_proj_a",
201
+ "model.language_model.layers.29.linear_attn.in_proj_b",
202
+ "model.language_model.layers.30.linear_attn.conv1d",
203
+ "model.language_model.layers.30.linear_attn.in_proj_a",
204
+ "model.language_model.layers.30.linear_attn.in_proj_b",
205
+ "model.language_model.layers.4.linear_attn.conv1d",
206
+ "model.language_model.layers.4.linear_attn.in_proj_a",
207
+ "model.language_model.layers.4.linear_attn.in_proj_b",
208
+ "model.language_model.layers.5.linear_attn.conv1d",
209
+ "model.language_model.layers.5.linear_attn.in_proj_a",
210
+ "model.language_model.layers.5.linear_attn.in_proj_b",
211
+ "model.language_model.layers.6.linear_attn.conv1d",
212
+ "model.language_model.layers.6.linear_attn.in_proj_a",
213
+ "model.language_model.layers.6.linear_attn.in_proj_b",
214
+ "model.language_model.layers.8.linear_attn.conv1d",
215
+ "model.language_model.layers.8.linear_attn.in_proj_a",
216
+ "model.language_model.layers.8.linear_attn.in_proj_b",
217
+ "model.language_model.layers.9.linear_attn.conv1d",
218
+ "model.language_model.layers.9.linear_attn.in_proj_a",
219
+ "model.language_model.layers.9.linear_attn.in_proj_b",
220
+ "model.visual*",
221
+ "score"
222
+ ],
223
+ "quant_algo": "NVFP4",
224
+ "producer": {
225
+ "name": "modelopt",
226
+ "version": "0.47.0"
227
+ },
228
+ "quant_method": "modelopt"
229
+ }
230
+ }
qwen3.5-4b-nli-v5-nvfp4/hf_quant_config.json ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "producer": {
3
+ "name": "modelopt",
4
+ "version": "0.47.0"
5
+ },
6
+ "quantization": {
7
+ "quant_algo": "NVFP4",
8
+ "kv_cache_quant_algo": null,
9
+ "group_size": 16,
10
+ "exclude_modules": [
11
+ "model.language_model.embed_tokens",
12
+ "model.language_model.layers.0.linear_attn.conv1d",
13
+ "model.language_model.layers.0.linear_attn.in_proj_a",
14
+ "model.language_model.layers.0.linear_attn.in_proj_b",
15
+ "model.language_model.layers.1.linear_attn.conv1d",
16
+ "model.language_model.layers.1.linear_attn.in_proj_a",
17
+ "model.language_model.layers.1.linear_attn.in_proj_b",
18
+ "model.language_model.layers.10.linear_attn.conv1d",
19
+ "model.language_model.layers.10.linear_attn.in_proj_a",
20
+ "model.language_model.layers.10.linear_attn.in_proj_b",
21
+ "model.language_model.layers.12.linear_attn.conv1d",
22
+ "model.language_model.layers.12.linear_attn.in_proj_a",
23
+ "model.language_model.layers.12.linear_attn.in_proj_b",
24
+ "model.language_model.layers.13.linear_attn.conv1d",
25
+ "model.language_model.layers.13.linear_attn.in_proj_a",
26
+ "model.language_model.layers.13.linear_attn.in_proj_b",
27
+ "model.language_model.layers.14.linear_attn.conv1d",
28
+ "model.language_model.layers.14.linear_attn.in_proj_a",
29
+ "model.language_model.layers.14.linear_attn.in_proj_b",
30
+ "model.language_model.layers.16.linear_attn.conv1d",
31
+ "model.language_model.layers.16.linear_attn.in_proj_a",
32
+ "model.language_model.layers.16.linear_attn.in_proj_b",
33
+ "model.language_model.layers.17.linear_attn.conv1d",
34
+ "model.language_model.layers.17.linear_attn.in_proj_a",
35
+ "model.language_model.layers.17.linear_attn.in_proj_b",
36
+ "model.language_model.layers.18.linear_attn.conv1d",
37
+ "model.language_model.layers.18.linear_attn.in_proj_a",
38
+ "model.language_model.layers.18.linear_attn.in_proj_b",
39
+ "model.language_model.layers.2.linear_attn.conv1d",
40
+ "model.language_model.layers.2.linear_attn.in_proj_a",
41
+ "model.language_model.layers.2.linear_attn.in_proj_b",
42
+ "model.language_model.layers.20.linear_attn.conv1d",
43
+ "model.language_model.layers.20.linear_attn.in_proj_a",
44
+ "model.language_model.layers.20.linear_attn.in_proj_b",
45
+ "model.language_model.layers.21.linear_attn.conv1d",
46
+ "model.language_model.layers.21.linear_attn.in_proj_a",
47
+ "model.language_model.layers.21.linear_attn.in_proj_b",
48
+ "model.language_model.layers.22.linear_attn.conv1d",
49
+ "model.language_model.layers.22.linear_attn.in_proj_a",
50
+ "model.language_model.layers.22.linear_attn.in_proj_b",
51
+ "model.language_model.layers.24.linear_attn.conv1d",
52
+ "model.language_model.layers.24.linear_attn.in_proj_a",
53
+ "model.language_model.layers.24.linear_attn.in_proj_b",
54
+ "model.language_model.layers.25.linear_attn.conv1d",
55
+ "model.language_model.layers.25.linear_attn.in_proj_a",
56
+ "model.language_model.layers.25.linear_attn.in_proj_b",
57
+ "model.language_model.layers.26.linear_attn.conv1d",
58
+ "model.language_model.layers.26.linear_attn.in_proj_a",
59
+ "model.language_model.layers.26.linear_attn.in_proj_b",
60
+ "model.language_model.layers.28.linear_attn.conv1d",
61
+ "model.language_model.layers.28.linear_attn.in_proj_a",
62
+ "model.language_model.layers.28.linear_attn.in_proj_b",
63
+ "model.language_model.layers.29.linear_attn.conv1d",
64
+ "model.language_model.layers.29.linear_attn.in_proj_a",
65
+ "model.language_model.layers.29.linear_attn.in_proj_b",
66
+ "model.language_model.layers.30.linear_attn.conv1d",
67
+ "model.language_model.layers.30.linear_attn.in_proj_a",
68
+ "model.language_model.layers.30.linear_attn.in_proj_b",
69
+ "model.language_model.layers.4.linear_attn.conv1d",
70
+ "model.language_model.layers.4.linear_attn.in_proj_a",
71
+ "model.language_model.layers.4.linear_attn.in_proj_b",
72
+ "model.language_model.layers.5.linear_attn.conv1d",
73
+ "model.language_model.layers.5.linear_attn.in_proj_a",
74
+ "model.language_model.layers.5.linear_attn.in_proj_b",
75
+ "model.language_model.layers.6.linear_attn.conv1d",
76
+ "model.language_model.layers.6.linear_attn.in_proj_a",
77
+ "model.language_model.layers.6.linear_attn.in_proj_b",
78
+ "model.language_model.layers.8.linear_attn.conv1d",
79
+ "model.language_model.layers.8.linear_attn.in_proj_a",
80
+ "model.language_model.layers.8.linear_attn.in_proj_b",
81
+ "model.language_model.layers.9.linear_attn.conv1d",
82
+ "model.language_model.layers.9.linear_attn.in_proj_a",
83
+ "model.language_model.layers.9.linear_attn.in_proj_b",
84
+ "model.visual*",
85
+ "score"
86
+ ]
87
+ }
88
+ }
qwen3.5-4b-nli-v5-nvfp4/model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:24e9c2a49d013caece8e87c1d0e857e83bf689dc7fd4436168bc3d2a61de58b7
3
+ size 3953796544
qwen3.5-4b-nli-v5-nvfp4/preprocessor_config.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "size": {
3
+ "longest_edge": 16777216,
4
+ "shortest_edge": 65536
5
+ },
6
+ "patch_size": 16,
7
+ "temporal_patch_size": 2,
8
+ "merge_size": 2,
9
+ "image_mean": [
10
+ 0.5,
11
+ 0.5,
12
+ 0.5
13
+ ],
14
+ "image_std": [
15
+ 0.5,
16
+ 0.5,
17
+ 0.5
18
+ ],
19
+ "processor_class": "Qwen3VLProcessor",
20
+ "image_processor_type": "Qwen2VLImageProcessorFast"
21
+ }
qwen3.5-4b-nli-v5-nvfp4/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a2cdd2e108566b09079afa8d266e9e65e7c280f27b771218237a06de5ba9cd86
3
+ size 19989592
qwen3.5-4b-nli-v5-nvfp4/tokenizer_config.json ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "audio_bos_token": "<|audio_start|>",
4
+ "audio_eos_token": "<|audio_end|>",
5
+ "audio_token": "<|audio_pad|>",
6
+ "backend": "tokenizers",
7
+ "bos_token": null,
8
+ "clean_up_tokenization_spaces": false,
9
+ "eos_token": "<|im_end|>",
10
+ "errors": "replace",
11
+ "image_token": "<|image_pad|>",
12
+ "is_local": true,
13
+ "local_files_only": false,
14
+ "max_length": 2048,
15
+ "model_max_length": 262144,
16
+ "model_specific_special_tokens": {
17
+ "audio_bos_token": "<|audio_start|>",
18
+ "audio_eos_token": "<|audio_end|>",
19
+ "audio_token": "<|audio_pad|>",
20
+ "image_token": "<|image_pad|>",
21
+ "video_token": "<|video_pad|>",
22
+ "vision_bos_token": "<|vision_start|>",
23
+ "vision_eos_token": "<|vision_end|>"
24
+ },
25
+ "pad_token": "<|endoftext|>",
26
+ "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
27
+ "split_special_tokens": false,
28
+ "stride": 0,
29
+ "tokenizer_class": "Qwen2Tokenizer",
30
+ "truncation_side": "right",
31
+ "truncation_strategy": "longest_first",
32
+ "unk_token": null,
33
+ "video_token": "<|video_pad|>",
34
+ "vision_bos_token": "<|vision_start|>",
35
+ "vision_eos_token": "<|vision_end|>"
36
+ }