Summarize the Content of the Blog
Why Splunk ES becomes too noisy
Splunk Enterprise Security ships with hundreds of out-of-the-box correlation searches. Most of them are enabled by default during initial deployment. They are calibrated against generic baselines, not the environment they end up running in.
The result is predictable. A typical mid-size ES environment generates 800 to 3,000 notable events per day after the first 30 days of operation. A senior SOC analyst can meaningfully triage about 40 to 60 events in an eight-hour shift. The arithmetic does not work.
The Splunk community has documented this pattern for years. The root causes consistently fall into four categories: too many correlation searches enabled, thresholds set too sensitively, throttling not configured, and notable event titles not deduplicated for repeat events. A senior bitsIO consultant working a typical ES tune-up engagement walks through the same six-step sequence below.
For broader context on alert-volume problems across the Splunk stack, see strategies for reducing Splunk alert fatigue.
Step 1: Audit which correlation searches are firing
Open the Incident Review dashboard and sort by correlation search name. The 20 percent of searches generating 80 percent of the volume are the tuning targets.
For each high-volume search, capture three numbers: events per day, true-positive rate (events that were investigated and confirmed as real), and the time-to-close on confirmed events. A correlation search firing 200 events a day with a 1 percent true-positive rate is producing 198 distractions and 2 useful signals daily. The math determines the next step.
Run the audit over a representative two-week window. One week is too short to capture weekly cycles (Monday morning logon spikes, end-of-month batch jobs). Four weeks is too long to wait before action.
Step 2: Disable the correlation searches you do not need
This is the most common missed step. Splunk ES correlation searches are documented in the Splunk Community as the single highest source of preventable noise. The Splunk Lantern guidance is direct: only enable correlation searches that match a defined use case for the environment.
For each high-volume search with a true-positive rate below 5 percent, ask one question: does this search map to a documented threat the security program cares about? If the answer is no, disable it. If the answer is yes, move it to Step 3.
A typical first pass disables 30 to 50 percent of enabled correlation searches without losing meaningful detection coverage. The savings on search head load, SOC analyst hours, and notable event volume are immediate.
Step 3: Tune thresholds on the searches you keep
For each correlation search in scope after Step 2, look at the threshold logic. ES correlation searches typically use a where clause with a count, time window, or statistical threshold. The thresholds are calibrated against Splunk’s reference environment, which is almost never the customer’s environment.
A practical approach: pull the search’s actual data over the audit window, build a histogram of the metric in the threshold (count of logins per user per hour, count of failed authentications per source IP, bytes transferred per session), then set the threshold at the 95th or 99th percentile of the observed distribution. This shifts the search from “find anything unusual” to “find what is clearly unusual for this environment.”
The Splunk Community has documented this calibration approach across multiple threads. The pattern is consistent: thresholds set against actual environmental baselines reduce notable event volume by 50 to 80 percent without losing high-fidelity detections.
Step 4: Add suppression and throttling
Throttling in Splunk ES correlation searches prevents repeated notable events for the same condition firing in a short window. Without throttling, a single attacker brute-forcing an authentication endpoint can generate one notable per failed attempt - 200 notables in three minutes for a single incident.
Configure throttling on every correlation search that has a high risk of repeat firing. The throttling window should be longer than the typical analyst triage time for that event class. For most authentication and network alerts, a 60-minute throttle window is reasonable. For high-volume reconnaissance alerts, 4 to 8 hours is reasonable.
Suppression is a separate mechanism that lets the analyst mark known-good conditions to be excluded from future correlation matches. Most environments have a small set of known-good patterns (specific admin accounts, scheduled service-to-service communication, vulnerability scanner traffic) that account for a disproportionate share of repeat notable events. Capture these as suppression rules during the tune-up.
For deeper SOC-side context on what happens after correlation search tuning, see the full threat-detection to automated-response pipeline.
Step 5: Map to Risk-Based Alerting (ES 7.x and 8.x)
Risk-Based Alerting is the most consequential change in Splunk ES of the last three release cycles. Instead of each correlation search creating an immediate notable event, correlation searches contribute a risk score to entities (users, systems, sessions). Notable events fire only when the cumulative risk score crosses a threshold over a defined time window.
In practice, RBA collapses the notable event volume problem at the architectural level. Ten correlation searches that each contributed a low-fidelity notable event per hour now contribute risk-score increments. The single notable event fires only when an entity has accumulated enough cross-correlated signals to warrant attention.
Moving an environment from rule-based correlation to risk-based correlation is a multi-week effort. The reward is consistent across Splunk Community case studies: 70 to 90 percent reduction in raw notable event volume, with no loss of detection coverage. ES 8.x extends this with refined risk scoring and AI-augmented detection, covered in what changed in Splunk ES 8.2 for alert workflows.
Step 6: Consolidate to notable events that matter
The final step is the editorial layer. Splunk ES notable events should reach the SOC analyst with three properties: clear title (the event type plus the affected entity), correct severity (informational, low, medium, high, critical, mapped to actual operational impact), and a defined next action (the analyst should know what to do within 30 seconds of opening the event).
Most notable event noise complaints are actually editorial-quality complaints. The events were technically accurate but operationally useless: titles that did not identify the affected entity, severity inflated by default, no next-action guidance. The fix is a notable event template review that standardizes titles, severities, and action guidance across the surviving correlation searches.
After all six steps, a typical mid-size ES environment moves from 800 to 3,000 daily notable events to 80 to 300. The SOC team gets time back. The detection coverage stays intact. The Splunk ROI conversation changes.
bitsIO offers a free 2-hour Splunk ES tune-up for organizations evaluating where their environment sits. It runs the audit (Step 1) and surfaces the three highest-leverage tuning actions for the specific environment.
Frequently Asked Questions
