NORN

The problem

Too many LLM audits are built on anecdotes: “I got it to say something it shouldn’t”. Without a taxonomy and metrics, two audits cannot be compared, there is no way to tell whether a change (a filter, a stricter system prompt) actually improves anything, and the final report cannot be reproduced.

Approach

NORN is a CLI that runs structured adversarial campaigns against models, RAG systems and agents, with three design decisions:

• Three-layer taxonomy: L1 standalone model (prompt injection, jailbreak, adversarial suffixes, semantic evasion, system-prompt leakage, training-data extraction), L2 RAG systems (indirect injection, index poisoning, exfiltration through poisoned retrieval) and L3 tool-using agents (tool abuse, privilege escalation, cross-tool exfiltration).
• 16 encoded techniques (L1_AT_01–10, L2_AT_01–03, L3_AT_01–03), mapped to the OWASP Top 10 for LLM, MITRE ATLAS and NIST AI RMF.
• Measure, don’t impress: heuristic scoring plus an LLM judge, and attack-success and error metrics — ASR, FAR/FRR, PSR@K, TDS, UAR, CTER and KCCR — exported to JSON, CSV, Excel and HTML so the report comes out of the tool, not out of a spreadsheet.

Results

Across four models (Gemma 4 31B, MiniMax M2.5, Nemotron 3 Super and Qwen3 Coder Next): 48 campaigns, 924 cases and 4,470 replicas.

• Gemma 4 31B was the most vulnerable at baseline (ASR = 0.2037).
• L1 hardening lowers ASR on Gemma 4 but raises it on MiniMax (from 0.0530 to 0.1230).
• L3 hardening did not reduce UAR: reviewing the cases, the failure is not in the textual output but in the decision to invoke the tool.
• KCCR produced no significant values and needs end-to-end validation.

I publish what did not work as well, because that is what changes decisions: if hardening raises ASR on one of four models, that matters more than any table of successes.

How to reproduce

• Master’s thesis (80 pages): methodology, full taxonomy, campaign configuration, metric definitions and results analysis. Linked above.
• Repository: the CLI with campaign execution, scoring and report export.

Limits

An ASR measures a specific attack under specific conditions, not the security of the whole system. The taxonomy covers text: multimodal and fine-tuning are out of scope. The LLM judge adds cost and its own variance to the scoring. And KCCR, as implemented, has not proven to be a useful metric.