Instructions to use laion/moss-voice-identity-loras with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use laion/moss-voice-identity-loras with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
MOSS voice-acting v2 -- per-voice identity-recovery LoRAs
⚠️ Do not merge this adapter into the base weights
merge_and_unload(), merge_adapter(), and any offline "bake the LoRA into the checkpoint"
script will destroy the model irrecoverably. This is not a performance caveat. Read this
before you write a deployment script.
Why
This adapter targets audio_lm_heads.0 … audio_lm_heads.11 and text_lm_head — 12 of its 23
target modules. In this architecture those output heads are weight-tied to the input
embeddings: tie_weights() sets
audio_lm_heads[i].weight IS audio_embeddings[i].weight # the same tensor, not a copy
text_lm_head.weight IS transformer.embed_tokens.weight
They are one allocation with two names. So when a merge adds B @ A * (alpha/r) into the head
weight, it writes that delta straight into the embedding table at the same time. The model
then reads its own inputs through a matrix that has been shifted by an output-side correction.
Generation does not fail loudly — it degrades into noise or into a fixed babble, and the damage
is inside the checkpoint you just saved. There is nothing to unmerge afterwards, because the
original values are gone.
Verify it yourself in three lines
Do not take our word for it:
m = base.model if hasattr(base, "model") else base
print(m.audio_lm_heads[0].weight.data_ptr() == m.audio_embeddings[0].weight.data_ptr())
# True -> same storage, merging corrupts the embeddings
What to do instead
Load with PEFT and leave the adapter unmerged. Set its strength through the scaling factor:
from peft import PeftModel
model = PeftModel.from_pretrained(base, "<this repo>", adapter_name="a").to(dev).eval()
# do NOT call model.merge_and_unload()
def set_weight(model, name, w):
"""Scale one named adapter's contribution. alpha/r is its own base scaling."""
for module in model.modules():
scaling = getattr(module, "scaling", None)
if isinstance(scaling, dict) and name in scaling:
if not hasattr(module, "_base_scaling"):
module._base_scaling = {}
module._base_scaling.setdefault(name, scaling[name])
scaling[name] = module._base_scaling[name] * float(w)
set_weight(model, "a", 1.0)
model.base_model.set_adapter(["a"]) # several adapters can be active at once
This sounds identical to a merge. An unmerged LoRA computes Wx + (B @ A)x * (alpha/r),
which is exactly what the merged weight W + B @ A * (alpha/r) would compute — the same
arithmetic, in a different order. You give up a small amount of inference speed and you keep the
ability to change the weight, stack several adapters, or turn one off. Nothing about the sound
changes.
If you are stacking adapters
Set each one's scaling separately and activate them together with
model.base_model.set_adapter([...]). Note that stacking is not free: in our own measurements a
deep stack held audio quality but destroyed intelligibility (word error 0.063 → 0.554). Add
adapters deliberately and measure.
If you maintain code that merges
A regex over module names is not enough — the reliable test is identity of storage. Group the
modules by weight.data_ptr() and refuse to merge into any group with more than one member.
lora_bank.py in LAION-AI/Humaneness-Voice-Demo-Server does this and asserts on the merge path.
One rank-4 LoRA per voice profile, trained on that voice's own takes that already sound like the voice (ECAPA speaker similarity to the voice's reference recording >= 0.60, held-out groups removed), and used to regenerate the takes of that voice that fell below the 0.40 similarity floor.
- rank 4, alpha 8, dropout 0.05, the same 23 target modules as the shipped per-voice adapters
- one stage, checkpointed and evaluated on held-out groups every epoch; the epoch shipped is the held-out-loss minimum, which varies by voice -- a fixed epoch count overfits some voices and underfits others
- merged for regeneration at scale 1.0
<voice>/report.json carries that voice's before/after numbers, the epoch curve that chose the
checkpoint, and the cost columns.
Two things that must travel with any result from these adapters
- The 0.40 floor is this project's ECAPA scale, not an established perceptual fact. An
independent WavLM-tbr embedder (
Orange/Speaker-wavLM-tbr, published same-speaker threshold 0.472) puts 75.4 % of the takes selected as "failures" above its own threshold. - Identity is bought, not free. In the reference study the identity gain cost -11 % target
emotion strength and -14 % blend. Every voice's
report.jsonreports its own cost columns.
And one about how to read a before/after
Production regenerates a single arm, so there is no per-voice control. The reference study
(emolia_c1699) measured that simply re-rolling these takes with no voice adapter lifts them
from 0.2 % to 14.0 % above the floor -- they are the unlucky tail of their groups, so regression
to the mean alone moves them. The adapter took the same population to 43.7 %. A per-voice
vs_orig delta in these reports therefore contains that re-roll component and is not the
adapter's effect on its own.
- Downloads last month
- -