Why Insufficient Logging and Monitoring Can Help Attackers Hide in Plain Sight

I’ve been working in information security for nearly 45 years and started my long journey with punch cards and mainframes leading to today’s cloud and zero-trust. Our world of computing has certainly evolved, and I can’t even recall how many post-security breach investigation teams I’ve been part of or how many cyber incident management teams I’ve supported over the years. We’re talking security failures of all sizes and magnitudes – including incidents that slammed major companies and resulted in many millions of dollars in damage. There has been one common denominator across these incidents and it’s the difficulty in getting useful application logs to figure out what went wrong to help us keep these attacks from happening again.

Logs are the lifeblood of security practitioners. They tell us what actions took place, whether processes were successfully completed, or whether a series of events led to a dangerous situation that was simply not expected from the application or a user’s behavior. For a security professional, logs are not “nice to haveâ€; logs are our Rosetta Stone. I’m not referring to the popular language learning program, but the carving created thousands of years ago in Egypt during the Hellenistic period. Just like the Rosetta Stone allowed scientists to decipher Egyptian hieroglyphs, application logs help us translate what attacker behavior looked like during the pre-attack and attack phases. Logs also help us to track whether stealthy software has been injected and if the attacker is still sneaking around the network. 

The Open Web Application Security Project (OWASP) is a highly respected body that issues security guidance for developers related to web application security. OWASP has included insufficient logging and monitoring as one of their Top 10 Application Security Risks. OWASP defines this category as “Insufficient logging and monitoring, coupled with missing or ineffective integration with incident response, allows attackers to further attack systems, maintain persistence, pivot to more systems, and tamper, extract, or destroy data. Most breach studies show time to detect a breach is over 200 days, typically detected by external parties rather than internal processes or monitoring.†And I’ll just say Hallelujah, as this definition hits the nail on the head from the aspect of a cyber defender and breach responder. 

I often hear from developers “…what should be logged?†And my response is generally “…what can’t you log?â€. At a bare minimum I’d want to see an entry of every sensitive transaction that the application performs, as well as logins, and failed logins. These can be exported to a security information and event management (SIEM) platform or analytics tool and we can create what are known as use cases to provide security teams alerts of anomalous behavior. Going beyond the basics, I’d recommend logs regarding API traffic (a recent touch point for attackers), server-side validation issues, long user dwell time on an application, and logs of application throttling actions used to defeat a type of attack known as credential stuffing – using compromised user names and passwords from other applications to try to authenticate. In this case the adage less is more does not apply. As security practitioners, we’ll take everything a developer can offer and we are getting new and powerful tools every year that can corelate log data in our efforts shore up cyber defense and keep some types of attacks from happening. 

I don’t live in a perfect world. I know the intense pressures that developers face to get new applications and features out as fast as possible to enable business. I’ve heard many reasons why logging falls off the list and can only say that verbose logging adds value not just from a security perspective but also for troubleshooting and quality purposes. And storage is no longer as expensive as it once was. So, help your security brethren and give us all those golden nuggets of logs. 

Here are my specific recommendations:

  1. Take a few minutes to read the OWASP Top 10 at https://owasp.org/www-project-top-ten/
  2. Take a few more to read the section on Insufficient Logging and Monitoring. This section also has useful information on how logs should be protected and stored. 
  3. Share a favorite beverage with a member of your local cyber security team and see how they are using logs to support their cyber defense activities. They should be able to help you get additional context for why these logs are so important. 
  4. Log all you can – and we’ll take it from there.