Tessera-small · ONNX
16×16 pixel-art sprites from a few words, running right in the browser.
rubber duck |
pizza wearing sunglasses |
astronaut cat |
sleepy sloth |
ghost |
taco |
fire |
energy drink |
red mushroom |
frog wearing a crown |
rocket |
slice of watermelon |
coffee cup |
robot |
cactus |
treasure chest |
Real outputs from this build, first try, seed 7. Nothing cherry-picked.
115 MBint8 weights, cached after the first visit |
~1 sfor 4 sprites on WebGPU |
16×16512-colour palette with transparency |
0 serversruns on the user's device |
Browser · Python · How it works · Quantization · License
Run it in the browser
Everything you need is in one file, web/tessera.js. It has no dependencies and no build step: it loads onnxruntime-web from a CDN and the model files from this repo.
1. Copy it into your project
curl -LO https://huggingface.co/cfahlgren1/tessera-small-onnx/resolve/main/web/tessera.js
2. Paint something
<script type="module">
import { Tessera } from "./tessera.js";
const tessera = await Tessera.load(); // ~115 MB the first time, then cached
const sprites = await tessera.generate("rubber duck"); // 4 variations
for (const sprite of sprites) document.body.append(sprite.toCanvas(8)); // 128×128, crisp pixels
</script>
That's it. On a laptop with WebGPU the four sprites take about a second.
Watch the pixels fill in by passing
onStep. It's called after each of the 16 steps with the sprites so far:await tessera.generate("astronaut cat", { onStep: (step, sprites) => preview.replaceChildren(...sprites.map((s) => s.toCanvas(8))), });
A complete page with a prompt box and progress is in web/example.html.
Options
Tessera.load(options)
| Option | Default | |
|---|---|---|
device |
"auto" |
"auto" uses WebGPU if it passes a quick correctness check (see below), otherwise the CPU. You can also force "webgpu" or "wasm". |
onProgress |
– | ({ stage, loaded, total }) => …, where stage is "download", "load" or "check". |
cache |
true |
Keep the models in the browser's Cache API, so later visits skip the download. |
modelUrl |
this repo | Base URL for the model files, e.g. a pinned revision or your own CDN. |
tessera.generate(prompt, options) returns Promise<Sprite[]>
| Option | Default | |
|---|---|---|
count |
4 |
How many variations, painted together in one batch. |
seed |
random | The same prompt and seed give the same sprites on the same device. |
steps |
16 |
Denoising steps. |
guidance |
3 |
Classifier-free guidance. Higher sticks closer to the prompt. |
onStep |
– | (step, sprites, painted) => … after every step. |
Sprite
toCanvas(scale = 8) |
A canvas scaled up scale× with crisp pixels. |
toBlob(size = 128) |
A PNG Blob. 128×128 is the size Slack wants for custom emoji. |
toImageData() |
The raw 16×16 ImageData. |
rgba / tokens |
16×16×4 RGBA bytes / the 256 palette indices (index 0 is transparent). |
Keep the page responsive on the CPU
WebGPU runs off the main thread. On the CPU, though, each batch of 4 takes about 40 seconds in a browser and blocks the page while it runs. If you support the CPU path, run Tessera in a Web Worker. It works there unchanged:
// worker.js
import { Tessera } from "./tessera.js";
const tessera = await Tessera.load();
self.onmessage = async ({ data: prompt }) => {
const sprites = await tessera.generate(prompt);
self.postMessage(sprites.map((s) => s.rgba), sprites.map((s) => s.rgba.buffer));
};
// main.js
const worker = new Worker("./worker.js", { type: "module" });
worker.onmessage = ({ data }) => { /* 4 × Uint8ClampedArray, 16×16 RGBA */ };
worker.postMessage("rubber duck");
Run it in Python
python/tessera_onnx.py is the same sampler in numpy. It runs on the CPU with onnxruntime, at about 2.5 seconds for 4 sprites.
pip install onnxruntime numpy pillow tokenizers huggingface_hub
curl -LO https://huggingface.co/cfahlgren1/tessera-small-onnx/resolve/main/python/tessera_onnx.py
python tessera_onnx.py "rubber duck" --out duck.png
Or from code:
from PIL import Image
from tessera_onnx import Tessera
tessera = Tessera.load() # downloads from this repo, cached by huggingface_hub
for i, sprite in enumerate(tessera.generate("rubber duck", seed=0)):
sprite.resize((128, 128), Image.NEAREST).save(f"duck-{i}.png")
What's in this repo
| File | Size | |
|---|---|---|
tessera_w8.onnx |
51 MB | The denoiser: 47.5M parameters, int8 weights. |
clip_text_w8.onnx |
64 MB | CLIP ViT-B/32 text encoder, int8 weights. |
tokenizer.json |
2.2 MB | The CLIP tokenizer. |
palette.json |
8 KB | The 512 RGBA colours that the model's tokens map to. Index 0 is transparent. |
selftest.json |
2.5 KB | Reference output for one fixed prompt, used to check that a GPU computes the model correctly. |
web/ |
tessera.js and example.html. |
|
python/ |
tessera_onnx.py. |
How it works
Tessera paints a sprite as a 16×16 grid of palette tokens, not RGB pixels. It starts with every pixel masked and unmasks them over 16 steps:
- Text. The prompt goes through the CLIP text encoder, padded to 24 tokens.
- Predict. At each step the denoiser predicts a distribution over the 512 colours for every pixel. It sees the pixels painted so far, its own predictions from the previous step (self-conditioning), and how much is still masked.
- Guide. Each sprite runs twice in the batch, with and without the prompt, and the two are mixed with classifier-free guidance (scale 3).
- Commit. Each pixel samples a colour. The most confident ones are kept, following a cosine schedule, with annealed Gumbel noise and a 3×3 non-maximum suppression so that confident pixels spread out across the grid.
All of this is in generate() in either file, in about 120 lines, if you want to write your own sampler.
Model inputs and outputs
tessera_w8.onnx (B = 2 × the number of sprites: a with-prompt row and a without-prompt row for each)
| Input | Type | Shape | |
|---|---|---|---|
x |
int64 | [B, 256] |
Palette index per pixel, or 512 for "still masked". |
text_states |
float32 | [B, 24, 512] |
CLIP last_hidden_state. |
text_pooled |
float32 | [B, 512] |
CLIP pooler_output. |
mask_ratio |
float32 | [B] |
Fraction of pixels still masked. |
sc_probs |
float32 | [B, 256, 512] |
Last step's probabilities (zeros on the first step). |
drop_text |
bool | [B] |
true for the without-prompt rows. |
Output logits |
float32 | [B, 256, 512] |
clip_text_w8.onnx: input_ids int64 [1, 24] gives last_hidden_state [1, 24, 512] and pooler_output [1, 512].
Quantization
The original fp16 export used about 2.4 GB of browser memory while loading, which is enough for phones to kill the tab. This build fixes that:
- int8 weights, fp32 math. Each weight has one scale per output channel and is turned back into fp32 as the model runs.
- Weights stay int8 in memory. By default onnxruntime folds constants at load time, which would expand every weight back to fp32.
tessera.jsturns that step off. - The convolution weights stay fp16. With per-channel int8, WebGPU in onnxruntime-web 1.22 gave wrong results for the 4D conv weights. They're only 0.3M of the 47.5M parameters.
x^2is written asx*x. The original graph usedPow(x, 2), which can return NaN on some GPUs for negative inputs.
| fp16 export | This build | |
|---|---|---|
| Download | 224 MB | 115 MB |
| Peak memory in Chrome while loading | 2.4 GB | 1.3 GB |
| 4 sprites on WebGPU | 1.1 s | 1.0 s |
Quality is very close to fp16. On 8 test prompts, the colour distribution per pixel differs by a KL divergence under 0.007, and 91–99% of pixels pick the same top colour.
The WebGPU self-test
Some GPUs and drivers run the model without any error but return NaN or garbage, which shows up as solid squares or random noise. So when device is "auto", Tessera.load() runs one fixed prompt and compares the output with selftest.json, which was computed on the CPU. If the output doesn't match closely (fewer than 80% of pixels picking the same top colour, or the logits off by more than 0.5 on average), it releases the GPU sessions and uses the CPU instead, and you get a warning in the console.
License and credits
- Tessera-small is by Jagrat Patel (TheJagStudio) under CC BY-NC 4.0, so this build is non-commercial too. Please credit the original model.
- CLIP text encoder: openai/clip-vit-base-patch32 (MIT).
- The ONNX conversion, int8 quantization,
tessera.jsandtessera_onnx.pyare from Slack Moji.
Model tree for cfahlgren1/tessera-small-onnx
Base model
thejagstudio/tessera-small