reson8-thomas commited on
Commit
ec9a91b
·
verified ·
1 Parent(s): e2e60d8

Update Rumble-2-Flash weights (2026-11-10) + auxiliary CTC head

Browse files

New weights converted with the same native-Transformers key mapping as the previous upload; config, tokenizer, chat template and processor files unchanged. Adds ctc_head.py and ctc/ (auxiliary CTC head + vocab) and a README section.

README.md CHANGED
@@ -81,3 +81,50 @@ Pass these options through vLLM's `chat_template_kwargs`, or directly to
81
  | `digits` | `true` | Numbers as digits; `false` writes them as words. |
82
  | `filler` | `false` | `true` keeps fillers such as "um" and "uh". |
83
  | `verbatim` | `true` | `false` collapses repeated words. |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
  | `digits` | `true` | Numbers as digits; `false` writes them as words. |
82
  | `filler` | `false` | `true` keeps fillers such as "um" and "uh". |
83
  | `verbatim` | `true` | `false` collapses repeated words. |
84
+
85
+ ## Auxiliary CTC head
86
+
87
+ The repo also contains the model's auxiliary CTC head: one linear layer (896 -> 99,631) on the output of the audio encoder's final
88
+ LayerNorm (`model.audio_tower.ln_post`). It was trained jointly with the model and gives a transcript from the audio encoder alone,
89
+ with no LLM decoding: greedy CTC, 13 frames per second of audio, 8 s encoder attention windows. It is fast, but much less accurate
90
+ than the model's normal output.
91
+
92
+ | File | Contents |
93
+ |---|---|
94
+ | `ctc/ctc_head.safetensors` | `weight` [99631, 896] and `bias` [99631], fp32 |
95
+ | `ctc/ctc_vocab.json` | `token_ids`: head class -> token id of this repo's tokenizer (class *k* = token *k*); `blank`: 99630 |
96
+ | `ctc_head.py` | `ResonantCTC`: loads the head, runs the audio encoder with 8 s windows, greedy CTC -> text |
97
+
98
+ Transformers and vLLM ignore these files (the head sits in `ctc/` so that vLLM does not load it as model weights).
99
+
100
+ ```python
101
+ import sys
102
+ import librosa
103
+ import torch
104
+ from huggingface_hub import snapshot_download
105
+ from transformers import AutoProcessor, Qwen3ASRForConditionalGeneration
106
+
107
+ path = snapshot_download("Reson8/Rumble-2-Flash")
108
+ sys.path.insert(0, path)
109
+ from ctc_head import ResonantCTC
110
+
111
+ processor = AutoProcessor.from_pretrained(path)
112
+ model = Qwen3ASRForConditionalGeneration.from_pretrained(path, dtype=torch.bfloat16).to("cuda")
113
+ ctc = ResonantCTC.from_pretrained(path, model, processor)
114
+
115
+ audio, _ = librosa.load("audio.wav", sr=16000)
116
+ print(ctc.transcribe(audio))
117
+ ```
118
+
119
+ Command line, from the downloaded repo: `python ctc_head.py <repo dir> audio.wav` (needs `soundfile`).
120
+
121
+ **FLEURS-30, greedy CTC** (test split, all 22,586 utterances = 77 h; WER, CER for zh / ja / ko / th; no language prompt):
122
+ mean **47.1** (the previous CTC head: 50.1). Best: en 16.3, ko 20.6, vi 23.2, zh 24.1. Weakest: el 95, ar 70, fi 66.
123
+
124
+ | ar | cs | da | de | el | en | es | fa | fi | fil | fr | hi | hu | id | it | ja | ko | mk | ms | nl | no | pl | pt | ro | ru | sv | th | tr | vi | zh |
125
+ |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
126
+ | 70.1 | 60.3 | 55.0 | 39.4 | 94.7 | 16.3 | 25.4 | 63.1 | 66.3 | 49.5 | 38.2 | 58.1 | 63.0 | 44.3 | 33.7 | 27.7 | 20.6 | 66.0 | 51.9 | 43.0 | 47.2 | 57.3 | 31.8 | 49.8 | 55.6 | 56.0 | 24.4 | 57.2 | 23.2 | 24.1 |
127
+
128
+ Known limitations: the head emits sub-word pieces frame by frame, so word starts are sometimes doubled or mis-cased; scripts whose
129
+ characters span several byte-level tokens can come out as invalid UTF-8 (U+FFFD); numbers mix digits and words, as the head has no
130
+ `digits` option.
conversion_info.json CHANGED
@@ -8,5 +8,13 @@
8
  "Materialize omitted tied lm_head from unchanged input embedding weights for strict loading.",
9
  "Preserve original EOS, padding, audio and user token IDs.",
10
  "Preserve Reson8 chat controls and add native HF assistant-prefill rendering."
11
- ]
 
 
 
 
 
 
 
 
12
  }
 
8
  "Materialize omitted tied lm_head from unchanged input embedding weights for strict loading.",
9
  "Preserve original EOS, padding, audio and user token IDs.",
10
  "Preserve Reson8 chat controls and add native HF assistant-prefill rendering."
11
+ ],
12
+ "weights_update_2026_11_10": {
13
+ "model.safetensors": "new Rumble-2-Flash weights, converted with the same key mapping (re-converting the previous source reproduces the previous model.safetensors 457/457 tensors bitwise); config, tokenizer, chat template and processor files unchanged",
14
+ "added": [
15
+ "ctc/ctc_head.safetensors",
16
+ "ctc/ctc_vocab.json",
17
+ "ctc_head.py"
18
+ ]
19
+ }
20
  }
ctc/ctc_head.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c13befc1c60a8410185a730cdccefc825df12ac8af75d4411fdff8106291ede7
3
+ size 357476284
ctc/ctc_vocab.json ADDED
The diff for this file is too large to render. See raw diff
 
ctc_head.py ADDED
@@ -0,0 +1,179 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Auxiliary CTC head for Reson8/Rumble-2-Flash: greedy CTC transcripts from the audio encoder alone.
2
+
3
+ What it is
4
+ One linear layer (896 -> 99,631) on the output of the audio encoder's final LayerNorm (audio_tower.ln_post), i.e. before
5
+ the multi-modal projector that feeds the LLM. Classes 0..99,629 are this repo's own tokens (ctc/ctc_vocab.json maps head
6
+ class -> token id of the repo tokenizer); the last class is the CTC blank. One encoder frame = 1/13 s (13 frames per second).
7
+ The head was trained jointly with the model (decoder + CTC loss) on the same data, using the 8 s encoder attention windows
8
+ of Qwen3-ASR (n_window_infer = 800 mel frames); this file runs the encoder with those windows.
9
+
10
+ Greedy decoding: argmax per frame -> collapse repeats -> drop blanks -> token ids -> the model's own tokenizer (byte-level BPE).
11
+ No LLM decoding, so it is fast, but it is much less accurate than the model's normal output (see the README).
12
+
13
+ import torch
14
+ from transformers import AutoProcessor, Qwen3ASRForConditionalGeneration
15
+ from ctc_head import ResonantCTC
16
+ repo = "Reson8/Rumble-2-Flash" # or a local copy of the repo
17
+ processor = AutoProcessor.from_pretrained(repo)
18
+ model = Qwen3ASRForConditionalGeneration.from_pretrained(repo, dtype=torch.bfloat16).to("cuda")
19
+ ctc = ResonantCTC.from_pretrained(repo, model, processor)
20
+ print(ctc.transcribe(wav)) # wav: 16 kHz mono float32 numpy array (or a list of them)
21
+
22
+ Command line: python ctc_head.py <repo id or dir> audio.wav [--device cpu|cuda:0]
23
+ """
24
+ import json, os
25
+ import numpy as np
26
+ import torch
27
+ import torch.nn.functional as F
28
+
29
+ # in a sub-folder: vLLM loads every top-level *.safetensors of a model dir as model weights
30
+ HEAD_FILE, VOCAB_FILE = "ctc/ctc_head.safetensors", "ctc/ctc_vocab.json"
31
+
32
+
33
+ def _resolve(path_or_repo, name, **hub_kwargs):
34
+ if os.path.isdir(path_or_repo):
35
+ return os.path.join(path_or_repo, name)
36
+ from huggingface_hub import hf_hub_download
37
+ return hf_hub_download(path_or_repo, name, **hub_kwargs)
38
+
39
+
40
+ def _cnn_len(n):
41
+ """Mel frames -> encoder frames (Qwen3-ASR _get_feat_extract_output_lengths): 13 per 100 mel frames."""
42
+ r = n % 100
43
+ f = (r - 1) // 2 + 1
44
+ return ((f - 1) // 2 + 1 - 1) // 2 + 1 + (n // 100) * 13
45
+
46
+
47
+ def _audio_tower(model):
48
+ """The audio encoder of a transformers Qwen3ASRForConditionalGeneration (model.model.audio_tower; thinker.audio_tower in qwen-asr)."""
49
+ for path in ("model.audio_tower", "audio_tower", "thinker.audio_tower"):
50
+ m = model
51
+ try:
52
+ for a in path.split("."): m = getattr(m, a)
53
+ return m
54
+ except AttributeError: continue
55
+ raise AttributeError("no audio_tower found on the model")
56
+
57
+
58
+ class ResonantCTC:
59
+ def __init__(self, audio_tower, tokenizer, feature_extractor, weight, bias, token_ids, blank):
60
+ self.tower, self.tok, self.fe = audio_tower, tokenizer, feature_extractor
61
+ p = next(audio_tower.parameters())
62
+ self.device, self.dtype = p.device, p.dtype # the head runs in the encoder's dtype (bf16, as trained)
63
+ self.proj = torch.nn.Linear(weight.shape[1], weight.shape[0]).to(self.device, self.dtype)
64
+ with torch.no_grad():
65
+ self.proj.weight.copy_(weight); self.proj.bias.copy_(bias)
66
+ self.token_ids, self.blank = list(token_ids), int(blank)
67
+ assert weight.shape[0] == len(self.token_ids) + 1 and self.blank == len(self.token_ids), "ctc_vocab.json does not match the head"
68
+ self.window = _cnn_len(100) * (audio_tower.n_window_infer // (audio_tower.n_window * 2)) # 13 * 8 = 104 frames (8 s)
69
+ self._captured, self._capture = [], False
70
+ # the head's input: a forward hook on the encoder's final LayerNorm
71
+ audio_tower.ln_post.register_forward_hook(lambda mod, inp, out: self._captured.append(out) if self._capture else None)
72
+
73
+ @classmethod
74
+ def from_pretrained(cls, path_or_repo, model, processor, **hub_kwargs):
75
+ """model: the transformers Qwen3ASRForConditionalGeneration; processor: its AutoProcessor."""
76
+ from safetensors.torch import load_file
77
+ sd = load_file(_resolve(path_or_repo, HEAD_FILE, **hub_kwargs))
78
+ voc = json.load(open(_resolve(path_or_repo, VOCAB_FILE, **hub_kwargs)))
79
+ tower = _audio_tower(model)
80
+ return cls(tower, processor.tokenizer, processor.feature_extractor, sd["weight"], sd["bias"], voc["token_ids"], voc["blank"])
81
+
82
+ # ------------------------------------------------------------------------------------------- features + encoder
83
+ @torch.no_grad()
84
+ def log_mel(self, wav):
85
+ """16 kHz mono float32 -> [128, T] log-mel, the model's Whisper-style features (as its processor computes them)."""
86
+ fe = self.fe
87
+ x = torch.as_tensor(np.asarray(wav, dtype=np.float32), device=self.device)[None]
88
+ st = torch.stft(x, fe.n_fft, fe.hop_length, window=torch.hann_window(fe.n_fft, device=self.device), return_complex=True)
89
+ filt = torch.from_numpy(np.asarray(fe.mel_filters)).to(self.device, torch.float32)
90
+ ls = torch.clamp(filt.T @ (st[..., :-1].abs() ** 2), min=1e-10).log10()
91
+ ls = torch.maximum(ls, ls.amax(dim=(1, 2), keepdim=True) - 8.0)
92
+ return ((ls + 4.0) / 4.0)[0]
93
+
94
+ @torch.no_grad()
95
+ def ln_post_states(self, feats):
96
+ """[128, T] log-mel -> [frames, 896] audio_tower.ln_post output, 8 s attention windows (the forward hook on ln_post captures it).
97
+ Same computation as the stock Qwen3-ASR encoder forward with flash_attention_2: 100-frame conv chunks, sinusoidal positions
98
+ per chunk, then every layer attends within windows of 104 frames counted from the start of the clip."""
99
+ t = self.tower; W = t.n_window * 2
100
+ x = feats.to(self.device, self.dtype)
101
+ T = x.shape[1]; n = -(-T // W)
102
+ lens = [W] * n
103
+ if T % W: lens[-1] = T % W
104
+ pad_to = W if (n > 1 or lens[0] == W) else lens[0]
105
+ chunks = torch.nn.utils.rnn.pad_sequence(list(x.transpose(0, 1).split(lens, 0)), batch_first=True) # [n, <=W, 128]
106
+ if chunks.shape[1] < pad_to: chunks = F.pad(chunks, (0, 0, 0, pad_to - chunks.shape[1]))
107
+ chunks = chunks.transpose(1, 2).unsqueeze(1) # [n, 1, 128, W]
108
+ embs = []
109
+ for part in chunks.split(getattr(t, "conv_chunksize", 500), dim=0):
110
+ e = F.gelu(t.conv2d1(part)); e = F.gelu(t.conv2d2(e)); e = F.gelu(t.conv2d3(e))
111
+ b, c, f, tt = e.shape
112
+ embs.append(t.conv_out(e.permute(0, 3, 1, 2).reshape(b, tt, c * f)))
113
+ e = torch.cat(embs, 0)
114
+ e = e + t.positional_embedding.positional_embedding[: e.shape[1], :].unsqueeze(0).to(e.dtype)
115
+ h = torch.cat([e[i, : _cnn_len(L)] for i, L in enumerate(lens)], 0) # [frames, 896]
116
+ S = h.shape[0]; win = [self.window] * (S // self.window) + ([S % self.window] if S % self.window else [])
117
+ for layer in t.layers:
118
+ r = h; a = layer.self_attn
119
+ z = layer.self_attn_layer_norm(h)
120
+ q, k, v = (p(z).view(S, a.num_heads, -1) for p in (a.q_proj, a.k_proj, a.v_proj))
121
+ o, s0 = [], 0
122
+ for w in win: # attention inside each 8 s window only
123
+ qq, kk, vv = (y[s0:s0 + w].transpose(0, 1).unsqueeze(0) for y in (q, k, v))
124
+ o.append(F.scaled_dot_product_attention(qq, kk, vv, scale=a.scaling)[0].transpose(0, 1)); s0 += w
125
+ h = r + a.out_proj(torch.cat(o, 0).reshape(S, -1))
126
+ h = h + layer.fc2(layer.activation_fn(layer.fc1(layer.final_layer_norm(h))))
127
+ self._captured.clear(); self._capture = True
128
+ try: t.ln_post(h)
129
+ finally: self._capture = False
130
+ return self._captured.pop()
131
+
132
+ # ------------------------------------------------------------------------------------------- CTC
133
+ @torch.no_grad()
134
+ def logits(self, ln_post_states):
135
+ """[frames, 896] -> [frames, 99631] head logits (last class = blank)."""
136
+ h = ln_post_states.to(self.device, self.dtype)
137
+ return torch.cat([self.proj(h[i:i + 4096]) for i in range(0, h.shape[0], 4096)])
138
+
139
+ def greedy_ids(self, logits):
140
+ """Greedy CTC -> token ids of the repo tokenizer."""
141
+ s = torch.unique_consecutive(logits.argmax(-1)).tolist()
142
+ return [self.token_ids[c] for c in s if c != self.blank]
143
+
144
+ def decode(self, ln_post_states):
145
+ """ln_post output of ONE clip (e.g. captured by your own hook on audio_tower.ln_post) -> text."""
146
+ return self.tok.decode(self.greedy_ids(self.logits(ln_post_states)), skip_special_tokens=True,
147
+ clean_up_tokenization_spaces=False).strip()
148
+
149
+ @torch.no_grad()
150
+ def transcribe(self, audio):
151
+ """16 kHz mono float32 array (or a list of them) -> greedy CTC transcript(s)."""
152
+ if isinstance(audio, (list, tuple)):
153
+ return [self.transcribe(a) for a in audio]
154
+ return self.decode(self.ln_post_states(self.log_mel(audio)))
155
+
156
+
157
+ def _read_audio(path):
158
+ import soundfile as sf
159
+ w, sr = sf.read(path, dtype="float32", always_2d=True)
160
+ w = w.mean(1)
161
+ if sr != 16000:
162
+ import torchaudio
163
+ w = torchaudio.functional.resample(torch.from_numpy(w), sr, 16000).numpy()
164
+ return w
165
+
166
+
167
+ if __name__ == "__main__":
168
+ import argparse
169
+ ap = argparse.ArgumentParser(description="Greedy CTC transcript with the auxiliary CTC head")
170
+ ap.add_argument("model", help="repo id (Reson8/Rumble-2-Flash) or local directory")
171
+ ap.add_argument("audio", nargs="+")
172
+ ap.add_argument("--device", default="cuda:0" if torch.cuda.is_available() else "cpu")
173
+ args = ap.parse_args()
174
+ from transformers import AutoProcessor, Qwen3ASRForConditionalGeneration
175
+ processor = AutoProcessor.from_pretrained(args.model)
176
+ model = Qwen3ASRForConditionalGeneration.from_pretrained(args.model, dtype=torch.bfloat16).to(args.device)
177
+ ctc = ResonantCTC.from_pretrained(args.model, model, processor)
178
+ for p in args.audio:
179
+ print(f"{p}\t{ctc.transcribe(_read_audio(p))}")
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:55dc79851ce26ccb58f0249d7d7eb7b1b9d3b1db13daadd6f1650b41c5cb59b5
3
  size 1017318776
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6dc35af40ad150e50767382a9d401eb9af9c3a1ad4e915c61fe05aecfa79a3e1
3
  size 1017318776