LLM Model Survey for Offensive Security Research

Key Findings

• Research tool access is the #1 quality differentiator — models with knowledge base search and web research capabilities scored up to 80% higher than the same model without tools
• Model size does not predict quality — a 40B-active MoE model outperformed a 671B model by 75%
• No free or low-cost model can replace frontier models for validation — the gap is structural, not incremental
• Knowledge base curation quality sets the ceiling for all downstream model work, regardless of model capability

Motivation

Offensive security research increasingly relies on LLM-assisted workflows for vulnerability analysis, exploit chain design, shellcode development, and technical reporting. As a Service-Disabled Veteran-Owned Small Business (SDVOSB) conducting authorized security assessments, we needed to answer:

1. Which models can support our research workflows at each cost tier?
2. How much does research tool access (knowledge bases, web search) improve output quality?
3. Can lower-cost models handle tasks currently requiring frontier models?
4. What are the structural limitations no amount of prompting can overcome?

All work described in this report was conducted under authorized Rules of Engagement against lab-owned hardware.

Methodology

Task Design

We designed a standardized exploit chain design task targeting a well-documented, fully-patched embedded system vulnerability (CVE-2010-2965, disclosed 2010). The task requires:

1. Architecture-specific shellcode design — MIPS32 Big Endian assembly using RTOS-specific APIs
2. Protocol-level exploitation steps — ordered sequence of debug agent commands
3. Hardware-level technical detail — CPU cache coherency constraints specific to the target architecture
4. Alternative approach design — achieving the same objective through a fundamentally different technique
5. Comparative analysis — multi-dimensional comparison of approaches with engineering tradeoffs

This task was chosen because it requires depth across multiple domains (assembly language, RTOS internals, network protocols, hardware architecture) and has well-established correct answers that can be objectively verified.

Scoring Rubric

Each output was scored by Claude Opus 4.6 across 4 dimensions (/10 each, 40 total):

Research Access

Models tested in two configurations:

• Raw — prompt only, no external tools
• With research tools — access to a curated knowledge base (hybrid vector + keyword search), web search, and web page fetching via MCP (Model Context Protocol) tool integration

Results

Full Leaderboard

Dimension Analysis

Cache Coherency is universally strong (6-8/10). Split I/D cache architecture and the need for explicit synchronization are well-represented in LLM training data. Every model that scored above 14/40 correctly explained the fundamental problem and the two-step solution (data cache writeback + instruction cache invalidation).

Shellcode Design caps at 6-7/10 for non-frontier models. Every non-frontier model produced functionally incomplete code — implementing data echo loops rather than actual command execution capability. This pattern persisted regardless of research quality, suggesting a training data gap in RTOS-specific exploitation techniques.

Exploitation Steps separate the top tier. Only GLM-5 and the frontier model scored 8+/10 on this dimension. The differentiator was protocol wire format detail — RPC framing, XDR encoding rules, and procedure-specific payload structure.

Alternative Approach reveals reasoning depth. Models that scored 6-7/10 demonstrated genuine architectural reasoning — identifying that the alternative technique eliminates the cache coherency requirement entirely. Models at 2-3/10 simply restated the primary approach with minor variations.

Finding 1: Research Tools Are the #1 Multiplier

The same model tested with and without research tool access showed dramatic score differences:

M2.7's improvement came primarily from Shellcode Design (+5) and Alternative Approach (+4) — dimensions where correct API signatures and protocol details from research replaced fabricated values.

However, more research does not always mean better results. M2.7 with 8 targeted tool calls scored 27/40, while the same model with 16 broader calls scored only 23/40. Targeted, specific queries outperform broad queries.

This finding informed our development of a structured question decomposition framework (TMIV — Target, Mechanism, Implementation, Validation) adapted from the medical research PICO framework.

Finding 2: Model Size Does Not Predict Quality

DeepSeek V3.1 (671B) scored lower than GLM-5 (40B active). The largest model produced only prose descriptions with no actual assembly code, while the smaller model produced protocol wire format detail that matched authoritative specifications.

Finding 3: The Frontier Gap Is Structural

No combination of model selection, research tools, or prompt engineering closed the gap between the best non-frontier model (28/40) and the frontier model (39-40/40). The missing capabilities are:

These are not research gaps — they're reasoning capabilities tied to training data depth on binary formats and low-level systems internals. Frontier models remain essential for validation even when lower-cost models handle research and initial generation.

Finding 4: Knowledge Base Quality Is the Ceiling

In a controlled test, the frontier model scored 39-40/40 when working from its own training data, but dropped to 32/40 when given access to a knowledge base containing prior model outputs from earlier benchmark runs.

The degradation occurred because the KB contained derivative outputs from lower-quality models — specifically, outputs that implemented incomplete functionality but were labeled with the correct terminology.

Implications for AI-assisted security research:

• Knowledge bases must distinguish between authoritative sources (vendor documentation, protocol specifications) and derivative work (model outputs, internal analysis)
• Research queries should filter by source authority level, defaulting to authoritative sources only
• Unvalidated model outputs should never be stored alongside reference material without clear labeling
• Curation quality directly constrains output quality for ALL models, including frontier

Finding 5: Consistency Matters for Production Workflows

GLM-5 has a higher peak (28) but M2.7 is more consistent (4-point range vs 7). For production research workflows where reliability matters more than occasional brilliance, M2.7's tighter variance makes it the safer choice as a primary workhorse model.

The TMIV Framework

Based on these findings, we developed a structured question decomposition framework for technical security research, adapted from the medical research PICO framework:

Recommendations

For Security Teams Evaluating LLM Tooling

1. Don't choose based on parameter count or benchmarks alone. Test on YOUR domain tasks.
2. Invest in research tool integration. The difference between a model with and without KB/web search access was larger than the difference between most model pairs.
3. Keep frontier models for validation, not generation. Use lower-cost models for research and initial drafting, then validate critical outputs with a frontier model.
4. Curate your knowledge base like it's production code. Bad data in your KB degrades ALL models, including frontier.
5. Test consistency, not just peak performance. A model that scores 25 every time is more useful than one that scores 30 sometimes and 18 other times.

For LLM Providers

1. Tool-calling optimization matters for specialized domains. Models fine-tuned for agentic tool use dramatically outperformed larger models without this optimization.
2. Content filtering on security research needs nuance. PII detection that treats CPU register names as person identifiers renders models unusable for legitimate security research.
3. Training data depth on embedded systems and RTOS internals is a differentiator. The gap between models correlates with VxWorks/MIPS-specific knowledge, not general reasoning capability.

Methodology Notes

• All testing conducted against fully-patched, lab-owned hardware under authorized Rules of Engagement
• CVE-2010-2965 was disclosed in 2010 and affects end-of-life hardware — no zero-day or novel vulnerability information is presented
• Exploit chain designs were evaluated for technical accuracy, not weaponized for operational use
• Scoring was automated via frontier model evaluation to ensure consistency across 18 test runs
• All models accessed via their respective cloud APIs or local inference; no model weights were modified