Make Alerts Great Again

Daniel Popescu - Make Alerts Great Again Why can’t this be easier? Writing good alerts and keeping them actionable is hard. Ask anyone on any security team, ever. Alerts are notoriously either too noisy or don’t have enough coverage, and finding the...

Daniel Popescu - Make Alerts Great Again

Why can’t this be easier? Writing good alerts and keeping them actionable is hard. Ask anyone on any security team, ever. Alerts are notoriously either too noisy or don’t have enough coverage, and finding the sweet spot is nearly impossible. Additionally, some alerts are idly sitting there functionally incorrect and don’t actually work as expected (when was the last time you tested some of yours?). To make matters worse, there is a general lack of industry standard for alert definitions, priorities, and incident response steps.

At Yelp, we have created tools and processes that enable the security team to keep a handle on our alerts, thus making the alerts actionable and maintainable. We do this by making sure we know which alerts are firing at what frequencies, having a run-book for writing new alerts, and utilizing self-service alerts whenever possible.

Certainly no alerting solution is perfect. However, by implementing some of these tools, we’ve effectively improved the signal-to-noise ratio for most of our important alerts. This in turn relieves the security team of tedious tasks and enables us to work on more important (and interesting!) things.