There is no single refusal-rate league table for open-weight models. As of November 13, 2026, the published figures come from three kinds of source: vendor model cards, academic benchmarks and independent studies. Each uses its own prompts, judge and metric. This page collects what each source reports. The figures in one table cannot be compared with those in another, and most vendor cards we read give no numbers at all.
What Vendor Model Cards Report
| Model | What the card publishes |
|---|---|
| gpt-oss-120b and gpt-oss-20b (OpenAI, August 2025) | Per-category scores on three internal evaluations, shown below |
| Gemma 4 31B (Google) | No figures. The card says the models produced "minimal policy violations" and improved on Gemma 3 while keeping unjustified refusals low |
| Qwen3.8-27B (Alibaba) | No refusal or safety evaluation. We found no mention of refusals, safety or jailbreaks in the card |
| GLM-5.3 (Z.ai) | No refusal figures. The card reports a capability score of 84.5 on CyberGym and describes cyber capability that grew faster than the developer expected |
OpenAI's card for gpt-oss scores the share of responses that were not unsafe under its own policy, so higher is safer.
| Evaluation and category | gpt-oss-120b | gpt-oss-20b | o4-mini |
|---|---|---|---|
| Standard disallowed content, illicit non-violent | 1.000 | 0.983 | 0.991 |
| Standard disallowed content, personal data (semi-restrictive) | 0.967 | 0.947 | 0.975 |
| Production benchmarks, illicit non-violent | 0.692 | 0.656 | 0.658 |
| Production benchmarks, illicit violent | 0.817 | 0.744 | 0.845 |
| Production benchmarks, harassment or threatening | 0.832 | 0.819 | 0.695 |
| Jailbreak evaluation, violence prompts | 0.983 | 0.979 | 0.991 |
| Jailbreak evaluation, illicit non-violent crime prompts | 0.979 | 0.960 | 0.980 |
The card calls its standard evaluation saturated. The production set, built from multi-turn conversations, gives much lower scores for the same models, which shows how far the choice of test moves the number.
An Academic Benchmark: SORRY-Bench
SORRY-Bench (ICLR 2025) tests 440 unsafe instructions across 44 topics and reports a fulfilment rate: the share of requests a model complied with, as judged by a fine-tuned 7B model. Lower means more refusal. The paper covers more than 50 models. Figures for some open-weight models:
| Model | Fulfilment rate |
|---|---|
| Llama-2-70b-chat | 12.27% |
| Llama-2-7b-chat | 13.86% |
| Gemma-7b-it | 18% |
| Llama-3-8b-instruct | 22% |
| Vicuna-7b-v1.5 | 34.55% |
| Llama-3-70b-instruct | 35% |
| Mistral-7b-instruct-v0.2 | 67% |
| OpenChat-3.5-0106 | 68% |
These are 2024 models. The same paper puts GPT-4o at 30% and Claude 2.1 at 5.23%, so the spread between vendors was wider than the gap between open and closed models.
An Independent Study of 2026 Models
A study published by Ubicloud on July 14, 2026, run by Superagent, tested three open-weight models against a closed baseline. Scores run from 0 to 100, higher is safer.
| Model | Harmful-content refusal, standard benchmarks | Harmful-content refusal, adaptive red-teaming |
|---|---|---|
| Claude Opus 4.8 (closed baseline) | 98.54 | 91.7 |
| Kimi K2.6 | 98.12 | 50.0 |
| Nemotron 3 Ultra | 98.04 | 33.3 |
| DeepSeek V4 Pro | 84.42 | 25.0 |
The standard column averages StrongREJECT (60 prompts) and HarmBench (400 prompts). The adaptive column rests on about 12 attacker-chosen attempts per cell, and the authors call it indicative only. Their conclusion was that open models can match the baseline on fixed benchmarks and fall behind once the test adapts. This is one commissioned study with single runs and no confidence intervals.
After the Weights Are Modified
For open weights, the refusal rate of the released model is not the whole picture, because anyone can change the weights. Research reports large shifts.
| Source | What was measured | Reported result |
|---|---|---|
| Arditi et al., NeurIPS 2024 | HarmBench attack success after editing out the refusal direction | From 4.4% to 84.3% across six Llama-2 and Qwen models, depending heavily on the model and on whether a system prompt was present |
| Lermen et al., 2023 | Refusal rate of Llama 2-Chat 70B after low-cost fine-tuning | About 1% on two refusal benchmarks, for under $200 |
| Abu Shairah et al., 2025 | Drop in refusal rate after abliteration | 70 to 80% in baseline models, at most 10% in models trained with the authors' defence |
| Kuo et al., 2026 | Attack success against models with tamper defences, using two gradient-free methods | From under 10% to between 16% and 96% |
| TamperBench, 2026 | 21 open-weight models against nine tampering threats | Current alignment-stage defences mostly fail under attack sweeps |
The distinction between weight edits, fine-tunes and prompt attacks is set out in abliterated vs uncensored fine-tune vs jailbreak.
Why the Numbers Cannot Be Compared
- Different metrics. "Not unsafe", fulfilment rate, safe rate and attack success rate measure different things, and some run in opposite directions.
- Different judges. Some use keyword matching, some a classifier, some another language model. The Arditi paper notes that its keyword-based refusal score does not check whether the answer was coherent or harmful.
- Different prompts and settings. System prompts, multi-turn context and provider-side filters all move the result.
- Vendor figures are self-reported. They use internal test sets that outsiders cannot rerun.
- Refusing more is not the same as being safer or better. A model can refuse harmless requests too, and benchmarks that count only refusals of harmful prompts do not measure that.
Methodology
We read each source directly on November 13, 2026: the gpt-oss model card (tables 4 to 6), the model cards for Gemma 4 31B, Qwen3.8-27B and GLM-5.3 on Hugging Face, the SORRY-Bench paper (tables 2 and 4 of version 2), the Ubicloud study and the abstracts of the research papers linked above. Figures are quoted as published and were not recalculated or rerun. The Arditi range is the lowest and highest value in that paper's HarmBench table for its own method. "No figures" for a model card means we searched the card text for refusal, safety, harm and jailbreak terms on that date. A developer may publish safety results elsewhere.