The project began with a basic question: how do authentication, endpoint, and network events become something a security analyst can actually investigate? The first version focused on normalization, detections, correlation, and reporting. The newer work asks the next question: once a finding exists, how can an investigation collect useful context without letting an AI model invent or control the evidence?
1. Deterministic detection stays underneath everything
SignalF0rge normalizes native JSONL plus selected Windows Security and Sysmon events into a common event model. Twenty-four native rules cover authentication, endpoint, network, behavioral, and cross-source activity.
raw telemetry → normalize → detect → correlate → score → finding
Rules include thresholds, distinct counts, ordered sequences, multi-step sequences, and cross-source correlation. Findings preserve their evidence, severity, entity, timestamps, and MITRE ATT&CK mappings.
2. Correlation turns alerts into a story
A failed login, PowerShell execution, credential access event, and network connection are individually useful signals. When they share an entity and occur in a meaningful order, they can describe a much stronger behavioral chain.
authentication → suspicious execution → credential access → network activity
deterministic output: correlated finding + preserved evidence
This remains the foundation of the project. The investigation agent begins after these findings already exist.
3. Build an investigation case around an entity
The agent groups findings into entity-centric cases. A case summarizes the related findings, severity, confidence, ATT&CK techniques, time range, evidence count, and recommended next actions.
This creates a boundary between detection and investigation: detections say what matched; the case gives the investigation a focused starting point.
4. Give the investigator tools, not unrestricted access
The investigation layer exposes a deliberately small set of read-only tools. It can retrieve related findings, map ATT&CK techniques, search matching events, inspect direct entity relationships, and perform bounded multi-hop traversal across users, hosts, and IP addresses.
case entity ↓ related findings ↓ entity links / bounded graph pivots ↓ matching telemetry ↓ ATT&CK context ↓ auditable investigation report
The tools do not modify telemetry, execute shell commands, or run arbitrary Python. Graph traversal is bounded so an investigation cannot expand indefinitely through every connected entity.
5. The LLM is a planner, not the detective database
SignalF0rge has an optional OpenAI-compatible planner. Its job is narrow: look at the current investigation context and choose which approved read-only tool should be called next.
Every requested tool call is validated against an allowlist before execution. Severity, findings, and evidence continue to come from deterministic SignalF0rge logic.
6. Treat AI input as hostile data
Security telemetry itself can contain attacker-controlled strings. A command line, event message, or previous tool result could include text that looks like an instruction to the model.
The planner therefore explicitly treats telemetry, finding text, commands, prior tool output, and other evidence as untrusted data. It is instructed not to follow directions found inside that evidence. Requested arguments are also validated before a tool can run.
This matters because adding an LLM to a security workflow creates a new attack surface. Prompt injection is not just a chatbot problem when the model is reading attacker-influenced logs.
7. Bound the agent loop
An agent that repeatedly asks for the same information is not useful. The investigation runner limits the number of steps and stops repeated identical tool requests. That makes the execution easier to audit and prevents a planner from looping indefinitely.
8. Reuse prior cases without turning memory into evidence
Bounded case memory lets a future investigation know that the same entity has been investigated before without storing unrestricted raw telemetry. The history contains compact report metadata, uses retention limits, and only recent cases for the current entity are exposed to the planner.
Historical cases are explicitly untrusted context. An old critical or escalate verdict cannot override the severity or verdict produced from the current deterministic findings. This gives the planner continuity without turning memory into proof.
9. Measure the system against harder benign activity
The evaluation layer now separates detection quality from investigation behavior. The adversarial benchmark contains 3,000 ordinary benign events, 19 legitimate events deliberately shaped like attack signals, and 48 attack events across 21 labeled scenarios: 3,067 events total.
31 true positives · 8 false positives · 0 false negatives
precision 79.49% · recall 100.00% · F1 88.57%
The false positives are useful results rather than something to hide: approved PowerShell retrieval, scheduled-task and service creation, vulnerability scanning, and an expired-password login burst all resemble behaviors the rules are designed to catch. CI pins the exact known false-positive set, so a new unexplained false positive fails the regression suite.
Investigation quality is evaluated separately through useful evidence pivots, redundant tool calls, tool-budget compliance, and deterministic-verdict grounding. Misleading historical case memory is also tested to verify that prior verdicts cannot override current evidence.
What this project taught me
The AI part is not valuable because it can generate a confident paragraph. It is valuable only if the system controls what evidence the model can see, what actions it can request, and which parts of the result remain deterministic. The realism benchmark reinforced a second lesson: a detector can look excellent on clean demonstrations and still produce understandable false positives when legitimate administration resembles attacker behavior.
That shifted the project from “add an LLM to a security tool” toward a more useful question: what is the smallest safe role an LLM can play in an investigation while the security engine remains authoritative?
View SignalF0rge on GitHub ↗