emilstabil commited on
Commit
4834705
·
verified ·
1 Parent(s): df068c7

Add under-800M Danish ASR benchmark comparison

Browse files
Files changed (3) hide show
  1. README.md +21 -0
  2. comparison.png +2 -2
  3. size-vs-accuracy.png +2 -2
README.md CHANGED
@@ -124,6 +124,27 @@ The NeMo checkpoint processed the benchmark at **795.8x realtime** on one A100.
124
  The official run saved raw per-sample outputs, and an independent offline
125
  rescore reproduced 12.37% exactly.
126
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
127
  ### Native GGUF quantization
128
 
129
  The F32, F16, Q8_0, Q6_K, Q5_K, Q5_0, Q4_K, and Q4_0 files are
 
124
  The official run saved raw per-sample outputs, and an independent offline
125
  rescore reproduced 12.37% exactly.
126
 
127
+ ### Comparison with models below 800M parameters
128
+
129
+ ![Mean WER of Ekko v1 Tiny and all twelve open-weight leaderboard models below 800M parameters](comparison.png)
130
+
131
+ The chart combines Tiny's Q5_0 release score with all 12 open-weight models
132
+ with reported parameter counts below 800M and results for all five test sets in the
133
+ [leaderboard snapshot of 4 October 2026](https://huggingface.co/datasets/RyeAI/danish-asr-leaderboard/tree/d5445121bb307df26ccf58d2a1fc04ae387ec38a/results).
134
+ Tiny is not yet a submitted row in that snapshot. The scores are equal-weight
135
+ means across five benchmark test sets totaling 26,780 rows. Tiny's 12.37% is 0.27
136
+ percentage points below the next entry, Svale 600M at 12.64%. These are point
137
+ estimates from this snapshot; the leaderboard may change as models are added.
138
+ Parameter labels use the sizes advertised on the leaderboard where available;
139
+ Tiny uses its published 110M size.
140
+
141
+ <details>
142
+ <summary>Accuracy against parameter count</summary>
143
+
144
+ ![Mean WER versus model parameter count for the same thirteen models](size-vs-accuracy.png)
145
+
146
+ </details>
147
+
148
  ### Native GGUF quantization
149
 
150
  The F32, F16, Q8_0, Q6_K, Q5_K, Q5_0, Q4_K, and Q4_0 files are
comparison.png CHANGED

Git LFS Details

  • SHA256: 1d8f02a3066ee0efe7861ab149045b69eae59df3c753a489590f59e2e3811c34
  • Pointer size: 131 Bytes
  • Size of remote file: 229 kB

Git LFS Details

  • SHA256: a971303be13f230b99b4bcfd0c9e5912f39f2be08de2135054ee2a941b7df810
  • Pointer size: 131 Bytes
  • Size of remote file: 120 kB
size-vs-accuracy.png CHANGED

Git LFS Details

  • SHA256: 5193a9dddac49f033cd75c6a0950f1cbaea160c66d570044cad41cce512282d4
  • Pointer size: 131 Bytes
  • Size of remote file: 211 kB

Git LFS Details

  • SHA256: d28be59652d0f1fc948e1059f4fc526455925135c133ca8bb5eb0e28d3f2ef8e
  • Pointer size: 130 Bytes
  • Size of remote file: 53.8 kB