Tessera-small · ONNX

16×16 pixel-art sprites from a few words, running right in the browser.

Live demo: Slack Moji Base model: Tessera-small

WebGPU: about 1 second for 4 sprites Download: 115 MB Weights: int8 License: CC BY-NC 4.0

rubber duck
rubber duck
pizza wearing sunglasses
pizza wearing sunglasses
astronaut cat
astronaut cat
sleepy sloth
sleepy sloth
ghost
ghost
taco
taco
fire
fire
energy drink
energy drink
red mushroom
red mushroom
frog wearing a crown
frog wearing a crown
rocket
rocket
slice of watermelon
slice of watermelon
coffee cup
coffee cup
robot
robot
cactus
cactus
treasure chest
treasure chest

Real outputs from this build, first try, seed 7. Nothing cherry-picked.



115 MB

int8 weights, cached after the first visit

~1 s

for 4 sprites on WebGPU

16×16

512-colour palette with transparency

0 servers

runs on the user's device

Browser · Python · How it works · Quantization · License


Run it in the browser

Everything you need is in one file, web/tessera.js. It has no dependencies and no build step: it loads onnxruntime-web from a CDN and the model files from this repo.

1. Copy it into your project

curl -LO https://huggingface.co/cfahlgren1/tessera-small-onnx/resolve/main/web/tessera.js

2. Paint something

<script type="module">
  import { Tessera } from "./tessera.js";

  const tessera = await Tessera.load();                  // ~115 MB the first time, then cached
  const sprites = await tessera.generate("rubber duck"); // 4 variations

  for (const sprite of sprites) document.body.append(sprite.toCanvas(8)); // 128×128, crisp pixels
</script>

That's it. On a laptop with WebGPU the four sprites take about a second.

Watch the pixels fill in by passing onStep. It's called after each of the 16 steps with the sprites so far:

await tessera.generate("astronaut cat", {
  onStep: (step, sprites) => preview.replaceChildren(...sprites.map((s) => s.toCanvas(8))),
});

A complete page with a prompt box and progress is in web/example.html.

Options

Tessera.load(options)

Option Default
device "auto" "auto" uses WebGPU if it passes a quick correctness check (see below), otherwise the CPU. You can also force "webgpu" or "wasm".
onProgress – ({ stage, loaded, total }) => …, where stage is "download", "load" or "check".
cache true Keep the models in the browser's Cache API, so later visits skip the download.
modelUrl this repo Base URL for the model files, e.g. a pinned revision or your own CDN.

tessera.generate(prompt, options) returns Promise<Sprite[]>

Option Default
count 4 How many variations, painted together in one batch.
seed random The same prompt and seed give the same sprites on the same device.
steps 16 Denoising steps.
guidance 3 Classifier-free guidance. Higher sticks closer to the prompt.
onStep – (step, sprites, painted) => … after every step.

Sprite

toCanvas(scale = 8) A canvas scaled up scale× with crisp pixels.
toBlob(size = 128) A PNG Blob. 128×128 is the size Slack wants for custom emoji.
toImageData() The raw 16×16 ImageData.
rgba / tokens 16×16×4 RGBA bytes / the 256 palette indices (index 0 is transparent).

Keep the page responsive on the CPU

WebGPU runs off the main thread. On the CPU, though, each batch of 4 takes about 40 seconds in a browser and blocks the page while it runs. If you support the CPU path, run Tessera in a Web Worker. It works there unchanged:

// worker.js
import { Tessera } from "./tessera.js";
const tessera = await Tessera.load();
self.onmessage = async ({ data: prompt }) => {
  const sprites = await tessera.generate(prompt);
  self.postMessage(sprites.map((s) => s.rgba), sprites.map((s) => s.rgba.buffer));
};
// main.js
const worker = new Worker("./worker.js", { type: "module" });
worker.onmessage = ({ data }) => { /* 4 × Uint8ClampedArray, 16×16 RGBA */ };
worker.postMessage("rubber duck");

Run it in Python

python/tessera_onnx.py is the same sampler in numpy. It runs on the CPU with onnxruntime, at about 2.5 seconds for 4 sprites.

pip install onnxruntime numpy pillow tokenizers huggingface_hub
curl -LO https://huggingface.co/cfahlgren1/tessera-small-onnx/resolve/main/python/tessera_onnx.py

python tessera_onnx.py "rubber duck" --out duck.png

Or from code:

from PIL import Image
from tessera_onnx import Tessera

tessera = Tessera.load()  # downloads from this repo, cached by huggingface_hub
for i, sprite in enumerate(tessera.generate("rubber duck", seed=0)):
    sprite.resize((128, 128), Image.NEAREST).save(f"duck-{i}.png")

What's in this repo

File Size
tessera_w8.onnx 51 MB The denoiser: 47.5M parameters, int8 weights.
clip_text_w8.onnx 64 MB CLIP ViT-B/32 text encoder, int8 weights.
tokenizer.json 2.2 MB The CLIP tokenizer.
palette.json 8 KB The 512 RGBA colours that the model's tokens map to. Index 0 is transparent.
selftest.json 2.5 KB Reference output for one fixed prompt, used to check that a GPU computes the model correctly.
web/ tessera.js and example.html.
python/ tessera_onnx.py.

How it works

Tessera paints a sprite as a 16×16 grid of palette tokens, not RGB pixels. It starts with every pixel masked and unmasks them over 16 steps:

  1. Text. The prompt goes through the CLIP text encoder, padded to 24 tokens.
  2. Predict. At each step the denoiser predicts a distribution over the 512 colours for every pixel. It sees the pixels painted so far, its own predictions from the previous step (self-conditioning), and how much is still masked.
  3. Guide. Each sprite runs twice in the batch, with and without the prompt, and the two are mixed with classifier-free guidance (scale 3).
  4. Commit. Each pixel samples a colour. The most confident ones are kept, following a cosine schedule, with annealed Gumbel noise and a 3×3 non-maximum suppression so that confident pixels spread out across the grid.

All of this is in generate() in either file, in about 120 lines, if you want to write your own sampler.

Model inputs and outputs

tessera_w8.onnx (B = 2 × the number of sprites: a with-prompt row and a without-prompt row for each)

Input Type Shape
x int64 [B, 256] Palette index per pixel, or 512 for "still masked".
text_states float32 [B, 24, 512] CLIP last_hidden_state.
text_pooled float32 [B, 512] CLIP pooler_output.
mask_ratio float32 [B] Fraction of pixels still masked.
sc_probs float32 [B, 256, 512] Last step's probabilities (zeros on the first step).
drop_text bool [B] true for the without-prompt rows.
Output logits float32 [B, 256, 512]

clip_text_w8.onnx: input_ids int64 [1, 24] gives last_hidden_state [1, 24, 512] and pooler_output [1, 512].

Quantization

The original fp16 export used about 2.4 GB of browser memory while loading, which is enough for phones to kill the tab. This build fixes that:

  • int8 weights, fp32 math. Each weight has one scale per output channel and is turned back into fp32 as the model runs.
  • Weights stay int8 in memory. By default onnxruntime folds constants at load time, which would expand every weight back to fp32. tessera.js turns that step off.
  • The convolution weights stay fp16. With per-channel int8, WebGPU in onnxruntime-web 1.22 gave wrong results for the 4D conv weights. They're only 0.3M of the 47.5M parameters.
  • x^2 is written as x*x. The original graph used Pow(x, 2), which can return NaN on some GPUs for negative inputs.
fp16 export This build
Download 224 MB 115 MB
Peak memory in Chrome while loading 2.4 GB 1.3 GB
4 sprites on WebGPU 1.1 s 1.0 s

Quality is very close to fp16. On 8 test prompts, the colour distribution per pixel differs by a KL divergence under 0.007, and 91–99% of pixels pick the same top colour.

The WebGPU self-test

Some GPUs and drivers run the model without any error but return NaN or garbage, which shows up as solid squares or random noise. So when device is "auto", Tessera.load() runs one fixed prompt and compares the output with selftest.json, which was computed on the CPU. If the output doesn't match closely (fewer than 80% of pixels picking the same top colour, or the logits off by more than 0.5 on average), it releases the GPU sessions and uses the CPU instead, and you get a warning in the console.

License and credits

  • Tessera-small is by Jagrat Patel (TheJagStudio) under CC BY-NC 4.0, so this build is non-commercial too. Please credit the original model.
  • CLIP text encoder: openai/clip-vit-base-patch32 (MIT).
  • The ONNX conversion, int8 quantization, tessera.js and tessera_onnx.py are from Slack Moji.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for cfahlgren1/tessera-small-onnx

Quantized
(1)
this model

Spaces using cfahlgren1/tessera-small-onnx 4