Configuration Parsing Warning:In adapter_config.json: "peft.task_type" must be a string

EvidenceGuard

A model that answers multiple-choice questions strictly from a supplied context, and says "insufficient information" when the context does not decide the answer.

It was trained on EvidenceGuard decision groups. Each group is a base question plus controlled variants: paraphrases, reordered options, irrelevant additions (the answer should not change) and decisive edits (the answer should change). A group counts as solved only if every variant in it is answered correctly.

Results

This model answers in a single forward pass, with no reasoning. Every model below was scored on exactly the same questions. "Groups fully correct" counts a group only if every variant in it is answered correctly.

EvidenceGuard test set: this model vs. decision APIs

EvidenceGuard test split

4,984 text questions in 878 groups.

Model Type Questions correct Groups fully correct
EvidenceGuard (this model) fine-tuned, single pass 86.5% 77.6%
Jev 1.14 decision API 84.9% 73.5%
GLM 5.3 LLM, with reasoning 85.5% 68.8%
DeepSeek V4 Pro LLM 78.4% 54.7%
GPT-6 Luna Decisions decision API 75.8% 50.9%
Perplexity Decider v1.2 decision API 72.7% 35.5%

New domains, never trained on

1,122 LLM-written questions in 187 groups, from four domains: legal, IT security, insurance and HR.

Model Type Questions correct Groups fully correct
GLM 5.3 LLM, with reasoning 94.0% 71.1%
EvidenceGuard (this model) fine-tuned, single pass 75.8% 31.0%
Perplexity Decider v1.2 decision API 75.1% 32.1%
Jev 1.14 decision API 71.4% 31.0%
GPT-6 Luna Decisions decision API 76.5% 28.3%
DeepSeek V4 Pro LLM 63.2% 11.8%

The model's lead does not carry over to new domains. There it ties the decision APIs, and a reasoning model is far ahead.

System1-Arena

System1-Arena is an outside decision benchmark that shares no data with EvidenceGuard. We used a 209-task sample covering its 13 non-agentic domains. Accuracy is averaged over tasks.

Model Type Accuracy
Jev 1.14 decision API 73.8%
EvidenceGuard (this model) fine-tuned, single pass 73.3%
Qwen 3.8 27B, untrained base model 71.0%
GPT-6 Luna Decisions decision API 68.0%

Fine-tuning did not cost general decision ability: the model scores 2.3 points above its untrained base.

Median latency is about 0.12 s per question at batch size 1 on one H200 GPU. API models were called through OpenRouter.

How it scores

There is no generation. The prompt lists the options as letters, and the model's logits at the first answer position are read for each letter (the max over the "A" and " A" tokens), then softmaxed over the K options.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen3.8-27B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "EvidenceGuard-Experiments/evidenceguard").eval()

SYSTEM = ("You answer multiple-choice questions strictly from the given context. If the context does not contain "
          "enough information to decide, choose the option that says so when one is offered. Reply with the letter only.")
LETTERS = "ABCDEFGH"
T = 1.9433390039063905  # calibration.json

def answer(context: str, question: str, options: list[str]) -> list[float]:
    opts = "\n".join(f"{LETTERS[i]}. {o}" for i, o in enumerate(options))
    user = f"Context:\n{context}\n\nQuestion: {question}\n\nOptions:\n{opts}\n\nAnswer with the letter of the correct option."
    msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": user}]
    prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False)
    enc = tok(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        last = model(**enc).logits[0, -1].float()
    ids = [sorted({tok.encode(L, add_special_tokens=False)[0], tok.encode(" " + L, add_special_tokens=False)[-1]})
           for L in LETTERS[: len(options)]]
    z = torch.stack([last[i].max() for i in ids])
    return torch.softmax(z / T, -1).tolist()

Limitations

  • Trained and tested on English multiple-choice questions with 2–8 options; not a general chat model.
  • Evaluation labels for the new domains come from three reasoning models and are still being checked by hand.
  • The training data is not public yet.
Downloads last month
35
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for EvidenceGuard-Experiments/evidenceguard

Base model

Qwen/Qwen3.8-27B
Adapter
(378)
this model