SOC Incident Response Agent — RL Benchmark
An OpenEnv-compliant environment that simulates a Security Operations Centre so LLM agents can be benchmarked on real incident-response tradeoffs.
The problem
Everyone claims their agent can do SOC work. Nobody could show it under a grader that penalises the wrong tradeoff between containment, business continuity, and compliance.
How it works
Built with friends at a hackathon: an OpenEnv-compliant environment exposing three escalating scenarios — alert triage, attack-chain reconstruction through MITRE ATT&CK correlation, and constrained incident response where every containment action has a business cost.
Grading is deterministic by design. Each scenario returns a transparent score in [0.0, 1.0] from explicit rules rather than a model's opinion, so two runs of the same agent are comparable and a failure can be traced to the decision that caused it.
Outcomes
- Three escalating scenarios covering triage, correlation, and response under constraint.
- Reproducible agent comparison — deterministic graders, no judge variance.
- My contribution centred on the evaluation test suite and fixing correctness bugs in the graders.