bluesec-soc-agent+

An LLM SOC-analyst agent for BlueSec: 60+ experiments in five days, $0.40 for the graded run, 25th of 76

● 2026-10-10 · 6 min readdraft

On October 10 the online stage of BlueSec closed. It’s a competition run by Positive Technologies and the FalsePositive community for AI agents that investigate security incidents. My agent completed the graded queue 40/40, scored 19.87236 and finished 25th of 76 (the leader scored 37.59360). The whole graded run cost $0.40 and took 56 minutes.

Here is what turned out to matter. Short version: the model sets the ceiling; the harness around it and measurement discipline decide how much of that ceiling you actually get.

The task

The agent receives an alert — JSON with the initial entities and relations — and walks an evidence graph through the platform’s tools: processes, files, network connections, registry, AD objects. It ends with a verdict:

  • malicious — plus the response: which hosts to isolate, which accounts to rotate, which files to remove;
  • benign — plus evidence that the activity is legitimate.

Scenarios contain legitimate noise and false leads. As I only learned at the very end, a task is scored Q · (0.6 + 0.4 · eff), where Q is answer quality and eff is call efficiency; the run score is the sum over tasks. Every extra tool call lowers the score.

Starting point

I started from the organisers’ reference agent (a ReAct/SGR loop). The first runs were instructive:

  • exp-000: the free model didn’t support json_schema — all 6 tasks failed with HTTP 400, zero tool calls. The agent burned through the queue in seconds.
  • exp-001: the reference agent on another model had to be stopped on budget: 14,537,750 tokens over 113 steps, up to 306,107 tokens in a single step, 49 failed calls out of 113.

The main cause: the reference loop re-sent the full history and the whole tool catalogue to the model on every step. On the same free model (partial run), after cleaning up the context and adding a local guard, the median tokens per step fell from 116,434 to 17,343, and tokens per task by about 4.8× (2.0M → 0.42M).

What worked

1. The harness around the model. The model mostly picks the next step; checks, cleanup and completing the answer are done in code:

  • a call guard answers repeats and calls with the wrong id type locally. On its own it wasn’t enough: a weak model still got stuck — 82 repeated calls, up to 40 in a row — while writing “stuck in a loop” in its own reasoning. The guard saved platform calls, but not time or tokens; only a hard repeat limit broke the loop;
  • expand_entity — a custom tool: “entity + neighbours in one call”. A hint in the prompt didn’t change behaviour (5 searches vs 64 one-by-one relation lookups); a dedicated tool did;
  • a conclude step: before closing an alert as a false positive, the model fills in a structured conclusion, and code checks against the call log that the required checks were actually done — “never from the model’s self-report”. There are three checks: what else happened on the host around the alert time, who is on the other end of inbound connections, and whether the process’s effects were read;
  • answer review before submitting, plus an answer policy: clean the rows, at most 8 rows per answer, and fetch the account and the source host if the agent never looked them up.

2. Precise beats more. The strongest changes were about removing things: telemetry rows from the answer, citations from the benign justification. Even one accurate root-cause citation instead of an empty justification lowered benign answer quality (≈0.55 → 0.46, one run).

3. One variable per experiment. Every hypothesis went into the ledger before the run, with the expected effect and a commit. The ledger has 60+ experiments: from exp-000 on October 6 to exp-061 on October 10. A rule from the ledger: “Deltas within the noise floor do not count”. Repeats of the final configurations (2–3 runs each) differed by 0.48–2.4 points — so many “improvements” turned out to be noise.

4. Not losing runs. The platform closes a run after 15 minutes without requests, and a closed run can’t be resumed. I lost runs to a sleeping laptop, the network and — ironically — my own guard: by saving calls it created an 18-minute silence. The fix: a deadline per model step, a forced verdict before the timeout, retries for platform calls, and caffeinate.

Models

The main model was xiaomi/mimo-v2.6-flash: a 30-task run cost about $0.3. According to the ledger on October 7, switching to it from a weak free model was the largest jump up to that point.

I also tried expensive models (prices as of October 9–10, 2026):

  • Claude Sonnet — in one run on the harness tuned for mimo it was no better and raised 5 false alarms out of 21. After calibrating the prompt for Claude, false alarms dropped to 1 and the score matched mimo — at about $1 per task vs ≈$0.01 for mimo;
  • Claude Opus (a probe on 5 malicious tasks, a different scenario set, benign untested) — about +0.04 per task by my estimate, at ≈181× the cost ($1.85 vs $0.0102 per task). I didn’t run it on the graded queue.

A frontier model was not a “drop-in win”: a harness tuned to one model doesn’t transfer to another for free.

The graded run

I chose the final configuration (final-v4) from repeats: v3 scored 17.45 / 16.81 / 15.03, v4 17.30 / 16.62. The difference in means (≈+0.53) is within noise, so v4 was chosen for its consistently better malicious answers, not for the score.

The first graded attempt broke off after 19 of 40 tasks — a platform-side failure, confirmed by the organisers. The relaunch went 40/40: no aborted tasks, no forced finishes, the run closed cleanly.

Per-task breakdown:

  • malicious: 28 of 28 verdicts correct, but answer completeness was only Q 0.574;
  • benign: 12 verdicts, 4 of them — by my reading — missed attacks (quality 0). All four share one shape: the alert pointed at an ordinary long-running process with no actions of its own, and the agent concluded that if the process did nothing, all was clean;
  • call efficiency cost only 0.68 points.

The main headroom isn’t the verdict or the number of calls — it’s the completeness of the response to an attack.

How it was built: one human, two agents

In one terminal pane a developer agent ran the experiments, read the results and kept the ledger. I delegated launch decisions to it. In the next pane a chronicler agent read the ledger and the raw run results every half hour and built the dossier for this post. There was also a “SOC expert” subagent — its advice was treated as a hypothesis and tested with a run.

The chronicler turned out more useful than I expected: cross-checking the ledger against the data, it caught errors — a wrong mean over repeats, “~240×” instead of the real ≈181×, an undercounted cost in the summary. The last one came after the graded run: the developer agent and I pulled the exact scoring formula from the run page, and the chronicler checked it against the cabinet scores of three dev runs (14.377, 11.996, 10.45) — they matched. So my score estimate since October 7 had been just a good fit.

The main takeaway: a developer agent needs a second agent that doesn’t write code but checks the numbers.