Imagine running a 911 call center where the switchboard is constantly lit up with incoming calls. The initial question, “What’s your emergency, please?” aims to funnel the event to the right responder for triage and assessment. Over the course of your shift, requests could range from soft-spoken “I’m having a heart attack” pleas to “Where’s my pizza?” freak-outs eating up important resources. Now add into the mix a volume of calls that burnout kicks in and important threats are missed.
That situation is what the average cybersecurity and response analyst deals with.
The concept of “alert fatigue” is real. A March 2023 study commissioned by IBM and completed by Morning Consult found security operation center (SOC) team members are “only getting half of the alerts that they’re supposed to review within a typical workday.” That’s a 50% blind spot.
Moreover, the same study found that most security analysts spend about a third of their typical workday investigating incidents that are not real and that the majority of threats are either low-priority or false positives.
Now, given the above, add in some desensitization to alerts, along with the disjointed, missing, or untested review, response and escalation processes, and the people, process and technology trifecta breaks down. The result? A successful attack.
The cost of delayed detection
A late spring 2023 supply chain attack on a communications provider shows how alert fatigue can contribute to the success of a cyberattack. Go back to the 911 call center: the stream of calls is not stopping, including the “Where’s my pizza?” hysteria. After a while, the call operator zones out. But what if one of those food delivery calls is really something else?
Let us go back to the cybersecurity world for a moment. Just to complicate matters, imagine more “call centers” (or detection tools, such as SIEMs and log aggregators and analyzers) are now triaging the same call but coming to different findings.
Some usual by-products caused by a thwack, or overuse, of similar tools and detection solutions inside the security stack are conflicting results, complacency and information paralysis. All of these issues result not only in delayed detection but also delayed action.
If you are a security operator, have you ever been:
- Overwhelmed by the level of data to analyze?
- In a situation that seems to have over-complicated itself?
- Fearful of making the wrong analysis or taking the wrong action?
- Faced with too many options and unable to choose?
- Stuck in the “I need to do more research” feedback loop?
If you have ever experienced these situations, it is quite likely information paralysis has fully kicked in, and response is likely to suffer. So what are the mechanisms to avoid this? Look into the people, process and technology trifecta, and some of the risk management answers lay there.
The right people and processes reduce fatigue
The cybersecurity skill gap, both in terms of available persons and experience, still exists. Therefore, finding the right people is a matter of hiring processes aligning with business needs and ensuring the necessary resources are available. An organization cannot use the “woe is me” line if investments in personnel are not there. Somebody has to run the machines but also give them a reality check.
Where it begins to get challenging are the processes. The processes are very much the bridge between people and technology, enabling sound decision-making. For example, reviewing an alert in isolation from other relevant factors could lead to the wrong action. Context matters. An analyst should not be pulling the fire alarm over a false positive.
Additionally, consider the “who” when it comes to analysis. Is there an escalation matrix available? A peer review process? Anything that does not put the weight of the world on a single person’s shoulders?
As threat investigators try to put the puzzle together, ask these questions. Are we:
- Getting the right alerts, and in a timely manner?
- Able to discern where discrepancies are coming from?
- Missing anything?
- Able to make a decision based on the information we have?
- Being kept honest by somebody else?
- Perhaps most importantly, looking at noise or signal?
Given the overwhelming amount of data and traffic, this is where the right tool stack for you can make all the difference.
The right technology streamlines and triages alerting into digestible pieces
Perhaps the best argument today for integrating more (appropriately tuned!) automation is burnout and fatigue. But remember: automation, orchestration and artificial intelligence, for their own sakes, are not the best investment. That is why “appropriately tuned” is an important caveat.
The most ideal situation is when your SIEM, SOAR and EDR solutions manage the low-hanging fruit. Put another way, the solutions manage the “noise” eating away at your staff’s time so that staff can focus more on the signal. Effectively, your technical solutions are doing the grunt work for you, triaging and funneling what looks to be serious to a human set of eyes. Those eyes — which should have greater context and understanding of the situation at hand — can now focus on the serious work but still maintain the ability to “look back” at what actions the automated tools performed.
To round out the technology piece of this conversation, keep these two thoughts in mind:
- Less can be more. More tools can create more alerts, including conflicting information. Streamline the technology stack where possible. Some products work well together; some do not. Do your homework for what is right for you.
- It’s a tool, not a crutch. If your technology solutions lack human oversight, expect a culture of complacency and knowledge reduction to creep in. Do not become dependent on automation, especially if it is only doing what it’s told.
In closing, look to technology not as your savior to the alert fatigue problem. Rather, treat technology as your partner that can do a good deal of the mundane heavy lifting. Do that, and you can focus your time on the threats that matter and ensure you do not fall behind answering those real emergency 911 calls.
