Add T5Gemma SentencePiece tokenizer.model

#8
by shawnbro12345 - opened

Adds the SentencePiece tokenizer.model for the T5Gemma text encoder already published here under tflite/t5gemma/encoder_fp16.tflite.

Why: the encoder is here but the tokenizer that pairs with it is not, so anything consuming the TFLite text-conditioning path can't tokenise without going to google/t5gemma-b-b-ul2 directly β€” which is gated (manual approval) and 401s unauthenticated. That defeats the purpose of re-hosting these weights, which was so users wouldn't have to accept Google's licence on HF themselves.

Concretely this is blocking the Stable Audio plugin's release pipeline: it can build the on-device model installers from this repo except for the tokenizer, which currently has to be copied off a laptop.

Provenance: byte-identical to google/t5gemma-b-b-ul2's copy, verified before upload.

size:   4241003
sha256: 61a7b147390c64585d6c3543dd6fc636906c9af3865a5548f27f31aee1d4c8e2

Unmodified β€” no conversion, no re-export.

Licence: same class of redistribution as the encoder already in this repo. LICENSE_GEMMA.md and NOTICE are present and cover it, so no licensing changes needed.

Root rather than under tflite/, since the tokenizer is shared across the MLX, tensorRT, cpu-amx and TFLite backends β€” happy to move it if you'd rather it sat elsewhere.

mattricesound changed pull request status to merged
mattricesound deleted the refs/pr/8 ref

Sign up or log in to comment