Clarify Standard One 8B usage and benchmark paths

README.md

@@ -27,29 +27,47 @@ tags:
 
 # Standard One 8B
 
-Decode-free typed decisions on Ministral 3 8B — one forward pass, native probabilities, ~26 ms per
-decision on one H200.
+Standard One scores a bounded set of answers for a supplied scenario and returns probabilities through
+`POST /v1/systemone`. It does not generate free-form response text. This repository contains the merged
+BF16 8B checkpoint and the server code.
+
+| If you need | Repository |
+|---|---|
+| Merged 8B checkpoint and server code | [StandardOne-8B](https://huggingface.co/StandardThinking/StandardOne-8B) (this repository) |
+| 8B adapter weights and merge recipe | [StandardOne-8B-LoRA](https://huggingface.co/StandardThinking/StandardOne-8B-LoRA) |
+| Smaller merged checkpoint | [StandardOne-3B](https://huggingface.co/StandardThinking/StandardOne-3B) |
+| Smaller adapter weights and merge recipe | [StandardOne-3B-LoRA](https://huggingface.co/StandardThinking/StandardOne-3B-LoRA) |
+
+In the reported served evaluations, 8B scores higher than 3B on the public standard and hard tiers;
+3B has a lower median latency on the measured short-request profile. See [Benchmarks](#benchmarks)
+for the measurement conditions and limitations.
 
 ![Standard One benchmark card: JevBench public tiers, held-out suites, stated-distribution probability, hard-tier calibration, latency and throughput for Standard One 8B, Standard One 3B and Jev 1.13.](docs/assets/00-benchmark-card.png)
 
-*Same items, same option order, raw probabilities unless noted; measured 24 September 2026.*
+*The figure combines results from different measurement paths. See [Benchmarks](#benchmarks) for
+served versus offline conditions; measured 24 September 2026.*
 
 ## At a glance
 
-- Turns `mistralai/Ministral-3-8B-Instruct-2512` into a typed-decision endpoint: send a piece of state
-  and a bounded rubric (`choice`, `noul`, or `score`), get back a native probability distribution over
-  every option — no decoding, one forward pass, ≈26 ms per decision on one H200.
-- Clear accuracy gains over the untuned base on five of the six suites measured; on the public hard
-  tier the untuned base currently scores higher than Standard One 8B (see Limitations).
+- Send a state and a bounded rubric to receive probabilities for the supplied labels:
+  `choice` selects among labeled options, `noul` is yes/no, and `score` uses an ordinal scale.
+  The endpoint scores the labels in one forward pass without decoding answer text.
+- In the same-run **offline** comparison with the untuned base, 8B improves on four of six suites,
+  ties on public easy, and declines on public hard. These are not served-endpoint results.
 - Probabilities are temperature-scaled and calibration-checked (hard-tier ECE, distribution
   total-variation) — see Benchmarks below.
 - Multilingual: English plus Japanese, Chinese, Spanish, French, German, Portuguese and Russian, with a
-  smaller Korean share. Image input is supported (the Pixtral vision tower is retained; images are
-  passed as data URLs).
+  smaller Korean share. The retained Pixtral vision tower accepts image data URLs; this card does
+  not report a separate image-input benchmark.
 - Apache-2.0 throughout: base model, adapter, merged weights and server code.
 
 ## Quick start
 
+Follow the setup in [QUICKSTART.md](QUICKSTART.md#standard-one-8b) first: on a CUDA-capable Linux
+host, clone this repository and create the SGLang and adapter virtual environments with `uv`.
+The commands below assume the working directory and installations from that guide. Run the engine
+and adapter in separate terminals.
+
 Engine (stock SGLang 0.5.20):
 
 ```bash
@@ -64,7 +82,7 @@ CUDA_VISIBLE_DEVICES=0 SGLANG_VLM_CACHE_SIZE_MB=0 .venv-sglang/bin/python -m sgl
 Adapter (`jev-adapter`, ships as `server/` in this repository):
 
 ```bash
-jev-adapter --engine-url http://127.0.0.1:30000 --model standard-one-8b --alias jev-latest \
+.venv-native/bin/jev-adapter --engine-url http://127.0.0.1:30000 --model standard-one-8b --alias jev-latest \
   --host 0.0.0.0 --port 30120 --max-concurrency 1 \
   --tokenizer-model mistralai/Ministral-3-8B-Instruct-2512-BF16 \
   --tokenizer-revision f6fae9795746f63c9be8344932f01275f3c63734 \
@@ -103,24 +121,48 @@ Response shape (illustrative values, default temperature applied):
 }
 ```
 
-More (client command, 3B variant, request format): see QUICKSTART.md.
+More (client command, 3B variant, request format): see [QUICKSTART.md](QUICKSTART.md).
 
 ## Benchmarks
 
-**(a) Accuracy by suite** — untuned base, offline no-system-prompt runner, native wording, single option order, T=1; Jev 1.13 measured on the same items through its own endpoint, raw probabilities, no temperature applied.
-
-| Suite | Untuned base | **Standard One 8B** | Jev 1.13 |
-|---|---|---|---|
-| JevBench public easy (48) | 100.00 % | **100.00 %** | 100.00 % |
-| JevBench public standard (72) | 79.17 % | **97.22 %** | 98.61 % |
-| JevBench public hard (111) | 60.36 % | **54.95 %** | 72.07 % |
-| judge proxy (600: routing + answer adequacy) | 79.33 % | **88.67 %** | 90.50 % |
-| realistic transfer set (600) | 72.83 % | **88.83 %** | 86.67 % |
-| stated-distribution probability (1,036) | 34.85 % | **81.08 %** | 72.97 % |
-
-The public hard tier is the one suite where the untuned base leads Standard One 8B; every other suite shows a double-digit gain over the base. Hard-tier ECE at served T is 0.178 (T=1.35) against Jev 1.13's 0.099 raw, and mean TV to the stated distributions is 0.127 at served T against Jev's 0.192 raw; full calibration table: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md#calibration).
-
-**(b) Speed** — serial, one H200, SGLang 0.5.20, 242-decision profile. Qwen checkpoints are untuned and shown for speed only; no accuracy comparison is implied.
+**Served endpoint results (the release configuration).** Merged BF16 weights through SGLang 0.5.20
+and `jev-adapter`, native wording, no system prompt, one option order, T=1.35. Jev 1.13 was
+measured on the same items through its hosted endpoint; its probabilities are raw, with no
+temperature applied. These are our measurements, not official sealed-set JevBench scores.
+
+| Suite | **Standard One 8B** | Jev 1.13 |
+|---|---|---|
+| JevBench public easy (48) | **100.00 %** | 100.00 % |
+| JevBench public standard (72) | **95.83 %** | 98.61 % |
+| JevBench public hard (111) | **54.05 %** | 72.07 % |
+| judge proxy (600: routing + answer adequacy) | **90.33 %** | 90.50 % |
+| realistic transfer set (600) | **89.50 %** | 86.67 % |
+| stated-distribution probability (1,036) | **81.56 %** | 72.97 % |
+| hard proxy (600) | **48.33 %** | 54.83 % |
+
+**Offline comparison with the untuned base.** This separate `transformers` runner used native
+wording, no system prompt, one option order, T=1. The base and tuned checkpoint were scored by the
+same offline path; these numbers are indicative of the base-model change, not the served scores above.
+
+| Suite | Untuned base | **Standard One 8B** |
+|---|---|---|
+| JevBench public easy (48) | 100.00 % | **100.00 %** |
+| JevBench public standard (72) | 79.17 % | **97.22 %** |
+| JevBench public hard (111) | 60.36 % | **54.95 %** |
+| judge proxy (600: routing + answer adequacy) | 79.33 % | **88.67 %** |
+| realistic transfer set (600) | 72.83 % | **88.83 %** |
+| stated-distribution probability (1,036) | 34.85 % | **81.08 %** |
+
+Against the untuned base, four suites improve, public easy ties, and public hard falls by 5.41
+percentage points on this offline run. Served and offline probabilities differ even on identical
+prompts, so use the served table for expected endpoint behavior. Hard-tier ECE at served T is 0.178
+(T=1.35) against Jev 1.13's 0.099 raw, and mean TV to the stated distributions is 0.127 at served T
+against Jev's 0.192 raw; full calibration table: [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md#calibration).
+
+**Speed** — raw serial latency on one H200 with SGLang 0.5.20, using a 242-decision profile averaging
+about 280 input tokens per decision. The 25.8 ms figure is p50 for this profile, not a latency
+guarantee for other request lengths, concurrency or hardware. Qwen checkpoints are untuned and shown
+for speed only; no accuracy comparison is implied.
 
 | Model | p50 | p95 | Input tokens/decision |
 |---|---|---|---|
@@ -158,8 +200,9 @@ An exact-text overlap audit against the public JevBench tiers found 0 exact scen
 
 ## Limitations
 
-- **Public hard tier: 54 %** for Standard One 8B, against **60 %** for the untuned base and **72 %** for Jev
-  1.13 — on this one suite the untuned base currently outperforms Standard One 8B.
+- **Public hard tier:** the served 8B score is **54.05 %**, versus **72.07 %** for Jev 1.13. In the
+  separate offline base comparison, Standard One 8B scores **54.95 %**, below the untuned base's
+  **60.36 %**.
 - Served probabilities are temperature-scaled by a single global value; if you apply this model to a
   materially different question distribution, re-fitting that temperature is advisable rather than
   assuming 1.35 transfers.
@@ -168,6 +211,7 @@ An exact-text overlap audit against the public JevBench tiers found 0 exact scen
 - Served and offline probabilities can differ on identical prompts (mean total-variation ≈0.07 on the
   hard tier); served numbers are treated as authoritative.
 - Korean is a small share of multilingual training relative to the other seven languages.
+- The card reports text benchmarks; it does not establish decision accuracy on image inputs.
 - Ten-way support triage (34–40 %) and RAG passage relevance (58–61 %) are weak zero-shot; fine-tune
   for those.