BlueBench
BlueBench is Cotool's benchmark for AI in defensive security work, run in environments we built around real and simulated intrusions. Frontier and open-weight models investigate each one and are scored on incident response, threat hunting, and detection engineering.
- Claude Opus 5.5 · score 83.2 · cost $3.54 (Pareto frontier)
- GPT-6 Astra · score 80.3 · cost $9.60
- Gemini 4 Argon · score 79.9 · cost $6.76
- Grok 4.6 · score 79.2 · cost $3.64
- Claude Sonnet 5.5 · score 77.5 · cost $1.01 (Pareto frontier)
- GPT-5.6 Sol · score 77.2 · cost $2.89
- GPT-6.1 Sol · score 76.7 · cost $1.30
- Claude Opus 4.8 · score 74.9 · cost $1.56
- GPT-6 Sol · score 71.1 · cost $1.74
- GLM-5.3 · score 70.9 · cost $3.27
- GLM-5.3 Flash · score 70.6 · cost $0.11 (Pareto frontier)
- GPT-5.6 Terra · score 69.9 · cost $1.01
- Claude Haiku 5.5 · score 68.5 · cost $0.59
- Kimi K3 · score 67.3 · cost $2.62
- GLM-5.2 · score 65.5 · cost $1.03
- GPT-6 Luna · score 64.1 · cost $0.26
- Gemini 3.6 Flash · score 62.7 · cost $3.01
- GPT-5.6 Luna · score 62.4 · cost $0.58
- DeepSeek V4 Flash · score 58.6 · cost $0.17
- MiniMax M3 · score 56.4 · cost $0.90
- DeepSeek V4 Pro · score 53.6 · cost $0.91
- Claude Opus 5 · score 50.5 · cost $4.44 (cyber refused)
- Gemini 3.1 Pro · score 45.3 · cost $4.63
- Claude Sonnet 5 · score 44.8 · cost $1.77 (cyber refused)
About BlueBench
BlueBench measures how well AI models do real defensive security work. The suite contains 74 tasks evaluated in a common harness, with task-specific prompts and tools over the raw telemetry. Six benchmarks in two series feed four tracks: incident response, threat hunting, detection engineering, and malware analysis. Those roll up into the capability, reliability, and cost leaderboards.
- Claude Opus 5.5 · 83.2
- GPT-6 Astra · 80.3
- Gemini 4 Argon · 79.9
- Grok 4.6 · 79.2
- Claude Sonnet 5.5 · 77.5
- GPT-5.6 Sol · 77.2
- GPT-6.1 Sol · 76.7
- Claude Opus 4.8 · 74.9
- GPT-6 Sol · 71.1
- GLM-5.3 · 70.9
- GLM-5.3 Flash · 70.6
- GPT-5.6 Terra · 69.9
- Claude Haiku 5.5 · 68.5
- Kimi K3 · 67.3
- GLM-5.2 · 65.5
- GPT-6 Luna · 64.1
- Gemini 3.6 Flash · 62.7
- GPT-5.6 Luna · 62.4
- DeepSeek V4 Flash · 58.6
- MiniMax M3 · 56.4
- DeepSeek V4 Pro · 53.6
- Claude Opus 5 · 50.5 (cyber refused)
- Gemini 3.1 Pro · 45.3
- Claude Sonnet 5 · 44.8 (cyber refused)
- Claude Opus 5.5Claude Opus 5.5
- GPT-6 AstraGPT-6 Astra
- Gemini 4 ArgonGemini 4 Argon
- Grok 4.6Grok 4.6
- Claude Sonnet 5.5Claude Sonnet 5.5
- GPT-5.6 SolGPT-5.6 Sol
- GPT-6.1 SolGPT-6.1 Sol
- Claude Opus 4.8Claude Opus 4.8
- GPT-6 SolGPT-6 Sol
- GLM-5.3GLM-5.3
- GLM-5.3 FlashGLM-5.3 Flash
- GPT-5.6 TerraGPT-5.6 Terra
- Claude Haiku 5.5Claude Haiku 5.5
- Kimi K3Kimi K3
- GLM-5.2GLM-5.2
- GPT-6 LunaGPT-6 Luna
- Gemini 3.6 FlashGemini 3.6 Flash
- GPT-5.6 LunaGPT-5.6 Luna
- DeepSeek V4 FlashDeepSeek V4 Flash
- MiniMax M3MiniMax M3
- DeepSeek V4 ProDeepSeek V4 Pro
- Claude Opus 5Claude Opus 5
- Gemini 3.1 ProGemini 3.1 Pro
- Claude Sonnet 5Claude Sonnet 5
BlueBench-Intrusion
Datasets of real intrusion logs. Each benchmark is a genuine hands-on-keyboard intrusion captured in production-grade telemetry: EDR, host and cloud audit logs, Zeek network metadata, and the alerts a SOC would actually see. Nothing is synthesized.
BlueBench-Simulation
Every environment is a fully generated enterprise: routine baseline activity, a simulated intrusion, and benign lookalikes that punish pattern-matching without verification. Because we author the environment, the ground truth is complete and none of the data can have leaked into training sets.
Leaderboards
The suite rolls up into three leaderboards. Each one has its own explorer.
Incident Response
Investigate an alert end to end and write the incident report: scope, affected hosts and accounts, attacker tradecraft, impact, and the response actions warranted. Graded against a hidden rubric.
- 1GPT-6 Astra88.0
- 2Claude Opus 5.587.8
- 3Gemini 4 Argon86.8
- 4GPT-6.1 Sol86.4
- 5Claude Sonnet 5.583.9
- 6GPT-5.6 Sol82.6
- 7Grok 4.680.8
- 8GPT-6 Sol77.9
- 9GLM-5.377.2
- 10GLM-5.3 Flash72.6
Detection Engineering
Write a detection rule that surfaces a specific behavior. Rules are re-executed against the evidence and scored on precision and recall.
- GPT-6 Astra · 87
- Claude Opus 4.8 · 85
- Claude Opus 5.5 · 84
- GPT-5.6 Sol · 83
- Grok 4.6 · 83
- Gemini 3.6 Flash · 81
- Gemini 4 Argon · 81
- Claude Haiku 5.5 · 81
- GPT-6.1 Sol · 79
- GPT-6 Sol · 78
- 87
- 85
- 84
- 83
- 83
- 81
- 81
- 81
- 79
- 78
Threat Hunting
Start from a weak lead, hunt across raw multi-source telemetry without assuming compromise, and report what is actually happening. Graded against a hidden rubric.
- 1Claude Opus 5.576.1
- 2Gemini 4 Argon74.5
- 3Grok 4.672.6
- 4Claude Opus 4.868.6
- 5GLM-5.368.2
- 6Claude Sonnet 5.566.9
- 7Kimi K364.3
- 8GLM-5.3 Flash63.2
- 9GPT-6 Astra62.8
- 10GPT-5.6 Sol62.7
AI for the blue team.
Run Cotool's harness in your environment to get real security work done