Feedback Shortcut checkpoints
The trained models of Feedback Shortcut: Speculative Decoding for Latent Feedback Transformers: 36 runs, six settings at 135M and 780M parameters with three seeds each. Code, configs and the measurements that read these folders: ifm-ai/LoRM.
| Setting | Model | Runs |
|---|---|---|
transformer-mtp |
Transformer with MTP heads | transformer-mtp-135m, transformer-mtp-780m (+ -s1, -s2) |
feedback |
Latent feedback Transformer with MTP heads (zeros and MTP-state stand-ins) | feedback-135m, feedback-780m (+ -s1, -s2) |
state-alignment |
State alignment | state-alignment-135m, state-alignment-780m (+ -s1, -s2) |
in-place |
In-place substitution | in-place-135m, in-place-780m (+ -s1, -s2) |
in-place-half |
In-place substitution in half the sequences | in-place-half-135m, in-place-half-780m (+ -s1, -s2) |
in-place-decay |
In-place substitution only during the learning-rate decay | in-place-decay-135m, in-place-decay-780m (+ -s1, -s2) |
Each folder holds:
checkpoint.pt: the fp32 model weights ({"model": state_dict}), loaded bylorm.eval.run.load_run.config.yaml: the run's resolved config, asworks/feedback_shortcut/make_config.pywrites it.metrics.jsonl: its training and validation log.
hf download IFM/feedback-shortcut-checkpoints --local-dir runs # or --include 'state-alignment-135m/*'
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐ Ask for provider support