Add model card: download + build + test (--slot8) instructions

README.md

@@ -0,0 +1,96 @@
+---
+license: other
+license_name: glm-5.2
+tags:
+- gguf
+- moe
+- glm
+- flash-moe
+- llama.cpp
+- metal
+library_name: llama.cpp
+---
+
+# GLM-5.2 Flash-MoE Sidecar (UD-IQ1_M)
+
+SSD-streamed Mixture-of-Experts **expert sidecar** for **GLM-5.2** (Unsloth Dynamic `UD-IQ1_M`),
+built for the **Flash-MoE slot-bank** runtime in the
+[`anemll/flash-llama.cpp`](https://github.com/Anemll/anemll-flash-llama.cpp/tree/GLM-5.2-Moe) fork.
+
+The routed experts are stored as per-layer `layer_*.bin` files and streamed from SSD on demand into
+a small resident **slot bank** during decode, so the full MoE runs on a unified-memory Mac without
+keeping every expert in RAM. The dense / shared weights live in a separate small GGUF.
+
+## What's in this repo
+
+| Path | Size | Description |
+|------|------|-------------|
+| `dense/model-dense.gguf` | ~15.5 GB | Dense + shared weights, router, attention (the model you pass to `-m`) |
+| `dense/flashmoe-package.json` | — | Flash-MoE package descriptor |
+| `layer_003.bin` … `layer_NNN.bin` | ~213 GB total | Per-layer routed-expert tensors (IQ1_M), streamed on demand |
+| `manifest.json` | — | Sidecar manifest (tensor map, quant types, byte offsets) |
+
+**Model facts:** arch `glm-dsa`, 256 routed experts, top-8 per token, 3 leading dense layers,
+`n_embd = 6144`, routed `n_ff = 2048`, experts quantized `IQ1_M`. Layout: `layer_major_whole_tensor`.
+
+> Total download is ~**213 GB**. You need a fast SSD; decode is I/O-bound on expert streaming.
+
+## Download
+
+```bash
+hf download anemll/GLM-5.2-sidecar --repo-type model --local-dir ~/Models/GLM-5.2-sidecar
+```
+
+## Build the runtime (Apple Metal)
+
+This sidecar requires the Flash-MoE fork on the **`GLM-5.2-Moe`** branch:
+
+```bash
+git clone -b GLM-5.2-Moe https://github.com/Anemll/anemll-flash-llama.cpp
+cd anemll-flash-llama.cpp
+cmake -B build -DGGML_METAL=ON
+cmake --build build --config Release -j --target llama-cli
+```
+
+## Run / test
+
+```bash
+./build/bin/llama-cli --perf \
+  -m ~/Models/GLM-5.2-sidecar/dense/model-dense.gguf \
+  --moe-mode slot-bank \
+  --moe-sidecar ~/Models/GLM-5.2-sidecar/ \
+  --moe-verify-sidecar \
+  --moe-slot-bank 64 \
+  --moe-topk 8 \
+  --moe-cache-io-split 2 \
+  --moe-prefetch-temporal \
+  -fit on \
+  -ub 1 -b 64 \
+  -ngl 999 \
+  -c 512 \
+  --seed 123 --temp 0 \
+  -p "What is Apple Neural Engine? Answer in one sentence." \
+  -n 2000 -st \
+  --slot8
+```
+
+### `--slot8` (fused single-kernel routed FFN)
+
+This branch adds `--slot8`, which collapses the whole routed FFN — gate, up, SwiGLU, down, and the
+routed weighted-sum over all selected experts — into a **single fused op** (two Metal kernels,
+IQ1_M) for single-token decode. It reads the resident slot ids once at encode time, so the
+per-expert `mul_mat_id` decode replay / ICB cache is no longer used on that path. Output is
+validated byte-identical to the unfused reference path.
+
+Toggles:
+- `--slot8` / `--no-slot8` — enable/disable the fused path (only engages on eligible top-k decode layers).
+- `LLAMA_FLASH_MOE_SLOT8_REFERENCE=1` — force the `mul_mat` reference path (A/B comparison / fallback).
+- `LLAMA_FLASH_MOE_SLOT8_DEBUG=1` — log which layers take the fused path.
+
+> Tested on Apple M5 Max (128 GB). `--slot8` is a decode-only fast path; prefill and non-eligible
+> layers use the normal slot-bank route.
+
+## License
+
+Derived from **GLM-5.2** (Z.ai / Zhipu AI). Use is subject to the original GLM-5.2 model license;
+this sidecar only repackages those weights for SSD-streamed inference.