Test your correlation searches end-to-end with Morning Checks

I did a talk at Splunk .conf21 about how to maintain correlation searches: pdf/mp4. One of the topics is morning checks. Basically you need end-to-end tests for your correlations. Here are the 3 steps:

• Automate harmless ways to trigger your correlations. Don’t fake logs, just do something the same as the real deal. E.g. for malware use the Eicar string.

• Make sure your rules handle it: mark as informational, reduce risk score, suppress the notable, etc.

• Have a morning checks checks dashboard: something that checks for the appearance of the risk score and/or notable (even suppressed).

If a morning check passes, it’s a full end-to-end test of your correlation, from behaviour all the way to alert.

I go into a lot more details in my talk pdf/mp4.

If you want to get the source code of what you see in the talk, it’s all in the ES Choreographer app in splunkbase.