Feedback Shortcut checkpoints

The trained models of Feedback Shortcut: Speculative Decoding for Latent Feedback Transformers: 36 runs, six settings at 135M and 780M parameters with three seeds each. Code, configs and the measurements that read these folders: ifm-ai/LoRM.

Setting Model Runs
transformer-mtp Transformer with MTP heads transformer-mtp-135m, transformer-mtp-780m (+ -s1, -s2)
feedback Latent feedback Transformer with MTP heads (zeros and MTP-state stand-ins) feedback-135m, feedback-780m (+ -s1, -s2)
state-alignment State alignment state-alignment-135m, state-alignment-780m (+ -s1, -s2)
in-place In-place substitution in-place-135m, in-place-780m (+ -s1, -s2)
in-place-half In-place substitution in half the sequences in-place-half-135m, in-place-half-780m (+ -s1, -s2)
in-place-decay In-place substitution only during the learning-rate decay in-place-decay-135m, in-place-decay-780m (+ -s1, -s2)

Each folder holds:

  • checkpoint.pt: the fp32 model weights ({"model": state_dict}), loaded by lorm.eval.run.load_run.
  • config.yaml: the run's resolved config, as works/feedback_shortcut/make_config.py writes it.
  • metrics.jsonl: its training and validation log.
hf download IFM/feedback-shortcut-checkpoints --local-dir runs   # or --include 'state-alignment-135m/*'
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support