When the Sensor Starts Thinking: SnortML, Agentic AI, and the Evolving Architecture of Intrusion Detection

When the Sensor Starts Thinking: SnortML, Agentic AI, and the Evolving Architecture of Intrusion Detection - Stack Overflow

Stack Overflow Business Stack Internal: the knowledge intelligence layer that powers enterprise AI.Stack Data Licensing: decades of verified, technical knowledge to boost AI performance and trust.Stack Ads: engage developers where it matters — in their daily workflow.Every IDS deployment has a gap. Anyone who has run one long enough eventually finds it, usually at the worst possible time. The gap sits between what you wrote rules for and what the attacker chose to do instead. Classic Snort signatures are genuinely impressive instruments. A well-crafted rule can catch a known exploit with near-zero false positives and overhead that barely registers on a profiler. That precision comes from specificity, and specificity is the whole problem. Write a rule for CVE-2024-12345 and you have coverage for that CVE. A modified payload that clears the same vulnerable code path by a slightly different route? Nothing fires.That is not a criticism of the signature model. It works exactly as designed. Signatures encode specific, verifiable knowledge about what an attack looks like at the wire level, and the low false positive rate is a direct product of that specificity. The real constraint is something harder to solve: exposure time. Between the moment a novel exploit surfaces in the wild and the moment a researcher captures it, reverse-engineers it, writes a rule, validates that rule against a test corpus, and ships it through an update channel, days or weeks can pass. For actively exploited vulnerabilities in common software, that window is not hypothetical.Cisco Talos addressed this directly in March 2024 with SnortML, a machine learning detection engine running natively inside Snort 3. Around the same time, a broader transformation in security operations started gaining serious traction: agentic AI entering network defense. These two developments operate at different layers of the same shift, and examining them together reveals something that looking at either one in isolation does not.Part 1: What SnortML Actually DoesBefore architecture and implications, the mechanics matter. SnortML is not a generic anomaly scorer tacked onto Snort's alert output, and it is not a cloud reputation service that phones home on every request. The inference happens entirely on the local device, inside the same processing pipeline as normal rule evaluation, and it produces a verdict in under a millisecond.Two components make this work. The snort_ml_engine module handles model loading at startup, bringing pre-trained TensorFlow models into memory and making them available as classifiers throughout the session. The snort_ml inspector then subscribes to data feeds from Snort's existing service inspectors through the publish/subscribe interface Snort 3 already uses internally. When the HTTP inspector finishes parsing a request, it publishes the URI query string and POST body to the event bus. The SnortML inspector picks that up, runs it through the classifier, and returns a float representing the probability that the content contains an exploit attempt.The neural network does not need to have seen a specific attack to flag it. It has learned the shape of what SQL injection attempts look like at the byte level, and that shape holds across a wide range of syntactic variations.The model architecture is an LSTM preceded by an embedding layer. The embedding layer maps raw byte values to learned vector representations, which captures relationships between bytes in a way that pure frequency analysis cannot. Think of it as analogous to word embeddings in NLP, except the tokens are bytes rather than words. A byte value of 0x27 (apostrophe) sitting next to 0x4F 0x52 (OR) carries learned context about SQL injection patterns, and the embedding layer encodes that. The LSTM then processes these sequences and captures temporal structure: the ordering of bytes matters, and attack payloads tend to have characteristic orderings that distinguish them from legitimate query strings.A final dense layer collapses the LSTM's output to a single probability float. LibML, the inference library shipped with SnortML, uses XNNPACK for hardware-accelerated matrix operations that keep inference time predictable under load. On a 4.7 GHz AMD processor, a single classification pass runs in roughly 350 microseconds. One practical detail worth knowing: from Secure Firewall 10.0.0 onward, SnortML automatically selects between models sized for 256, 512, or 1024 byte inputs based on the actual query length. Short queries get the lighter model. Only the longer, more complex requests go through full-sized inference. For queries exceeding 1024 bytes, the input gets truncated to that boundary before classification, which is a behavior worth keeping in mind when working with applications that generate unusually long parameter strings.The initial release targeted SQL injection detection. By late 2025, coverage expanded to include XSS and command injection attack classes. Model updates arrive through Snort's Lightweight Security Package system, using the same update channel as rule content, which means SnortML stays current without requiring a separate update workflow.TECHNICAL NOTE: ADAPTIVE MODEL SELECTIONThe 256/512/1024-byte model selection is not just a performance optimization. Each model was trained on a distribution of inputs at that length range, so the smaller model is genuinely calibrated for short queries rather than being a truncated version of the full model. This matters when reasoning about false positive behavior: a 200-byte legitimate query that looks slightly injection-like will be scored by the 256-byte model, which has seen a more concentrated distribution of short query traffic during training. Understanding which model variant fires on a given alert helps when tuning threshold behavior.The diagram below shows how this sits inside Snort's packet processing pipeline. The SnortML classifier runs in parallel with traditional signature matching. Pay attention to where the two paths diverge from the inspector dispatch and where they converge at the verdict stage: either path can independently trigger an alert, and a detection where both paths fire simultaneously carries meaningfully higher confidence than one triggered by ML alone.FIGURE 1: SNORTML INFERENCE PIPELINE INSIDE SNORT 3Why the parallel architecture matters?SnortML does not replace signature evaluation. Running both in parallel is a deliberate engineering decision, not a transitional compromise. A neural network trained on a vulnerability class will occasionally misfire on legitimate traffic that resembles attack syntax. URL-encoded special characters in a legitimate database query are a common case. Running signatures alongside the ML model means the two mechanisms provide independent coverage with different error profiles: the ML catches novel variants that no signature exists for, while classical matching provides a low-noise floor for known patterns. When both fire on the same payload, that correlation is meaningful signal for downstream systems.Latency impact was tested carefully. The 350-microsecond overhead is real and needs to be understood in context. A high-throughput Snort deployment on a current Cisco Secure Firewall appliance operates with a per-packet processing budget ranging from low hundreds of microseconds to a few milliseconds, depending heavily on ruleset size and protocol complexity. Adding 350 microseconds is not negligible. That is precisely why XNNPACK acceleration matters: it keeps the ML overhead predictable and bounded rather than variable under load.Part 2: The Limits of Embedded ML and Why Agents Come NextSnortML is a focused engineering solution to a specific problem. That focus is a strength and a limitation at the same time. It catches zero-day exploit variants within known vulnerability classes, on-device, with no external dependency. Within that scope, it works well. The scope itself is where things get interesting.The classifier operates on individual HTTP parameters. A single URI query string or POST body comes in, gets scored, and either triggers or does not. What the model has no visibility into is what came before that request, or after it, or from the same source IP over the previous twenty minutes. Consider a simple three-request sequence: a probe request to map the application's input validation behavior, an enumeration request to identify injectable parameters, and finally the exploit attempt tailored to what the first two probes revealed. Each request, looked at individually, might score below threshold. The third request in that chain is more dangerous precisely because of the first two, and SnortML cannot see that relationship.The same boundary applies to anything outside HTTP parameter space. DNS tunneling exfiltration, TLS-layer protocol attacks, SMB exploitation, timing-based covert channels, protocol behavioral anomalies in non-HTTP services: none of these pass through the HTTP inspector publish path, so SnortML never sees them. The architecture could accommodate other inspectors, and the publish/subscribe interface is generic enough for that. But trained models for those data sources do not exist yet, and assembling the labeled training corpora for less studied protocols is considerably harder than it was for HTTP.None of this is a bug. These are the natural constraints of any detection system that operates at the per-packet, per-parameter level. Getting past them requires a different kind of reasoning: one that holds context across time, correlates signals from multiple observation points, and does not wait for a human to read a report before acting. That is what agentic AI in security operations is built to do.The diagram below shows how different detection approaches cover different layers of the attack surface. SnortML sits at the wire level, extending Snort's reach from known patterns into zero day variants. Agentic reasoning operates at a higher abstraction