The firewall market has rebranded itself around machine learning, and security teams are being asked to evaluate claims they have no easy way to test. Having sat on both sides of those evaluations — as a buyer and as the person later reconstructing why a detection did not fire — my position is straightforward: AI-driven inspection is a real capability with real limits, and the vendors least willing to explain their limits are the ones to walk away from.
What the Models Can Actually See
With most traffic now encrypted end to end, deep packet inspection in the classic sense inspects very little payload. What modern engines analyze instead is metadata and behavior: TLS handshake fingerprints, packet size and timing distributions, connection graphs, DNS patterns, and the entropy characteristics of what payload remains visible. Machine learning genuinely helps here, because these signals are statistical by nature — a human cannot write a crisp rule for “this session’s timing rhythm resembles interactive command and control,” but a model trained on labeled flows can score it.
That is the honest pitch. The dishonest version implies the appliance somehow reads encrypted content without decrypting it. It does not. If a vendor’s detection story depends on payload visibility, ask precisely where decryption happens, what the key management story is, and what traffic classes — medical, financial, privileged credentials — you will be legally or ethically obliged to exempt. Those exemptions become your blind spots, and adversaries who profile your environment will find them.
Failure Modes Worth Interrogating
Statistical classifiers fail differently from signatures, and your evaluation should target those differences deliberately.
- Base rates: even an impressive-sounding accuracy figure produces overwhelming false positives when malicious flows are a vanishing fraction of total traffic. Ask for precision at your traffic volume, not accuracy in a lab.
- Drift: models trained on last year’s traffic mix degrade quietly as your applications change. Ask how retraining works and who validates it.
- Adversarial adaptation: attackers can pad packets, jitter timing, and mimic benign fingerprints. Ask what happens when the statistical disguise is competent.
- Explainability: when the model blocks a flow at 3 a.m., your on-call engineer needs a reason, not a confidence score.
A detection you cannot explain is a detection you cannot tune, and a detection you cannot tune will eventually be disabled.
That last point deserves emphasis because it is how these products actually die in production. The appliance ships, alerts flood the SOC, nobody can articulate why individual verdicts were reached, thresholds get raised to quiet the noise, and within two quarters the “AI engine” is effectively a very expensive NetFlow collector. The failure was not the model. It was deploying a probabilistic control without an operational plan for probabilistic output.
Running an Evaluation That Means Something
Vendor bake-offs default to replaying canned malicious PCAPs, which every product on the shortlist has already been tuned against. A meaningful evaluation instead mirrors your own traffic — a span port or packet broker feeding your real, messy, encrypted production mix — and layers controlled red-team activity on top: a legitimate remote-access tool used maliciously, data staged and exfiltrated over an approved SaaS channel, beaconing tuned to blend into your actual traffic rhythms. Score the product on what it catches, what it misses, and critically, on the analyst time each alert consumes.
Insist on measuring time to explanation: from alert to a human understanding what happened and why. Measure it for true positives and false positives alike, because the second number determines whether your team will still trust the console in six months.
Where This Belongs in the Stack
AI-driven inspection earns its place as one detection layer among several — strongest against novel, behavior-shaped threats, weakest at providing courtroom-grade certainty. Keep signature-based controls for the known-bad, keep identity and endpoint telemetry as your primary investigative substrate, and let the statistical layer do what it is actually good at: surfacing the anomalies your rules never anticipated. Buy the model, but budget for the humans who will live with its output. The product is the easy half of the deployment.