Works with the AI Agents you already use
Detection Rule Benchmark
Measures your security detection rules against labelled attacks and benign lookalikes, then reports detection rate, false positives, error bars, and missed cases, with a pass or fail gate for CI.
$12
Detection Rule Benchmark
Example session with this skill installed
Here are our test scenarios and the alerts from last night's rule run. How good are these detections really, and what should we fix first?
- Read your context and instructions
- Compiled the detection rule benchmark
This uses
examples/, which ships with the skill, with--known-ruleslisting a fourth rule. The numbers are real output from runningscore_detections.pyon that sample data, so you can reproduce them.
Summary
Status: fail. Detection is 5 of 6 and false positives are 1 of 5, so both targets are missed. The set is also too small to support a pass.
Results
| Metric | Result | 95% interval | Target |
|---|---|---|---|
| Detection rate | 83.3% (5 of 6) | 43.7% to 97.0% | at least 90% |
| False positive rate | 20.0% (1 of 5) | 3.6% to 62.5% | at most 10% |
| Hard negative false alarms | 33.3% (1 of 3) | not gated | |
| Technique recall | 62.5% | not gated | |
| Technique precision | 100% | not gated |
These intervals are wide because the sample is small. Treat the rates as rough.
What to fix first
DNS tunnelling is never detected. dns-tunnel-01 was missed, and the dns-long-labels rule never fired, so check that rule and its data source first.
2.
powershell-bypass fires on a developer build. Scenario dev-build-01 is a false alarm. It needs to separate build tools from attacker behaviour.
3.
The PowerShell rule doesn't report obfuscation. Both encoded-ps scenarios expect T1027.010, but the alerts only report T1059.001, which is part of why technique recall is 62.5%.
Per family
| Family | Detected |
|---|---|
| brute-force | 2 of 2 |
| script-execution | 2 of 2 |
| ransomware | 1 of 1 |
| dns-tunnel | 0 of 1 |
What the data can't support
The script also reports these shortfalls, so fixing the rules alone won't produce a pass:
{
"status": "fail",
"problems": [
"detection rate 83.3% is below 90.0%",
"false positive rate 20.0% is above 10.0%",
"family dns-tunnel was never detected"
],
"shortfalls": [
"6 attack scenarios, need 20",
"5 benign scenarios, need 20",
"3 hard negatives, need 5"
]
}
Next steps
- Fix the DNS data source and the PowerShell rule, using tuning scenarios only.
- Grow the set to at least 20 attacks, 20 benign scenarios, and 5 hard negatives, with a holdout.
- Save the new report as a baseline and add the script to CI.
Connects securely to your tools. The creator never sees your data.
About this skill
"100% detection" means little if the test was five easy attacks. A low false positive rate means little if the benign data looked nothing like an attack. This skill gives your agent a fairer way to test detection rules, and a script that reports the result with its uncertainty.
What it does
- Tests against hard negatives. Benign activity built to look like an attack, such as an admin creating accounts or a developer running PowerShell with
-ExecutionPolicy Bypass, is tracked and scored separately. - Reports error bars. Every rate comes with a 95% Wilson interval, so 20 of 20 is reported as about 84% to 100%, not as 100%.
- Won't pass on thin data. Fewer than 20 attacks, 20 benign scenarios, or 5 hard negatives returns
insufficient, even with perfect scores. - Finds gaps that averages hide. An attack family that is never detected fails the run, even when the overall rate meets the target.
- Keeps a holdout. Mark scenarios as holdout and the gate scores only those, which guards against tuning to the test.
- Catches regressions. Compare with an earlier report, and any newly missed attack or new false alarm fails the run.
- Names what to fix. Missed scenarios, the rule behind each false alarm, rules that never fired, and alerts that matched nothing.
What's in the package
SKILL.md: the routine your agent followsscripts/score_detections.py: the scorer, in Python standard library only, with no network callsreferences/scenario-design.md: how to vary attacks, hard negatives to include, and how much data you needreferences/file-formats.md: input formats, options, and report fieldsexamples/: a small scenario set you can run
Good fit
- Writing or tuning SIEM, Sigma, KQL, Splunk, YARA-L, or Elastic rules
- A CI check on a detection-as-code repository
- Giving leadership detection numbers that hold up
Known limitations
- It doesn't run your rules. You run them on your own engine and export the alerts. It works with any engine that can do that, but it has no connectors or rule converters.
- It doesn't include attack data. You supply recorded or synthetic telemetry. The guide explains how to design it.
- A pass applies to these scenarios only. It doesn't mean you would catch a real attacker.
- It can't detect made-up alerts or labels. The skill tells the agent not to work out matches by reading rules, but the score is only as good as its input.
This is a tool for defensive measurement. It contains no attack tooling.
Where it comes from
The approach comes from the evaluation of Aegis, my AI security operations platform, which is benchmarked against synthetic attacks and benign lookalikes. The results published for Aegis measure that system, not this skill.
Requires Python 3.9+. Doesn't run your rules, convert formats, or include telemetry. A pass applies to the tested scenarios only. It can't detect made-up alerts or labels.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 15 days ago
Requires
Permissions
Terminal runs scripts/score_detections.py. Read covers your scenario and alert files. Write covers saved reports. The script makes no network calls.