New software vulnerabilities are being discovered every day, and what was secure yesterday might not be secure today. One obvious example of this is operating system patches: most organizations update each server with the latest patches from the vendor every month or so, but many will tell you that they struggle to get 100% coverage.

We do things a little differently at Bold Penguin, and by combining DevOps principles with some clever automation, we can address operating system vulnerabilities within minutes of receipt of notification. Here’s the path we took, and how you can incorporate the same path at your organization.‍

‍1. Understand the risks associated with patching.‍

Patching can fail for many reasons, including hardware failures, intermittent network errors, or snowflake configurations which inhibit updates. If (or when) the update fails, your resource will be left in a compromised state. What’s worse, it will be difficult to reverse because patching involves modifying an existing resource, not building a new one.

Another drawback to patching is that it usually requires a privileged user account and some ability to remotely administer the target machine. This is a security hole that many organizations accept to achieve the greater good of applying operating system patches. That said, 75% of breaches came from compromised stolen user IDs and passwords, so for each company that flies under the radar with untouched user accounts, there’s another one (or many) that risk serious damage.‍

‍2. Leverage immutable infrastructure

‍Rather than continue patching, we choose to rebuild a new resource so that we could easily apply updates, and roll back to an older version if we needed to. This effectively allows us to validate these new resources before decommissioning the older resources. For example, our new virtual machines are deployed to our Alpha environment, and assuming our testing succeeds, they are promoted to Beta and then to Production.

Thanks to this approach we don’t need administrative access to the underlying virtual machine, which is optimal from a security and compliance perspective. As a result, we don’t have to manage the logistical complexity of key management and our attack surface is greatly decreased.

3. Automate your response to security events‍

Even with all these fancy toys, we still needed a little bit of magic to keep everything on schedule. It turns out that AWS supplies an RSS feed for any security-related vulnerabilities that affect Amazon Linux. We use a custom-built AWS Lambda function to regularly check this RSS feed and invoke our automated-AMI creation pipeline. The end result? We are able to create new hardened AMIs within minutes of a new vulnerability being identified.

We ended up building an AWS Lambda function using Ruby.  It performs 3 tasks sequentially:

• It loads the RSS feed and detects the last security event that was either marked as important or critical
• If we’ve never handled the security alert before, we launch a CodePipeline automation to create brand new AMIs.
• We use DynamoDB to track which alerts we’ve already handled, as well as any CodePipeline IDs that were triggered by a specific security alert.
  

The code isn’t open source but the implementation is pretty simple:

‍

‍

This Ruby file is packaged up into an AWS Lambda function and is deployed via the Serverless framework. It runs once an hour like clockwork, just waiting for anything new vulnerabilities to pop up.

The end result is that we are able to address security vulnerabilities in minutes and hours, not days or weeks. The investment we’ve made into this initial automation pays dividends as we continue to scale.

We leverage our standard software development lifecycle to handle most operational processes. This allows us to confidently and rapidly deploy application or infrastructure changes with high confidence. Using this technique, operating system patches are trivial to build, test, and deploy. This is a great example of how operational processes can become easier, faster, and effective by employing DevOps principles.

‍

Written by Frank Lamantia, SVP of Engineering at Bold Penguin. He can be reached for further information or comment via email at frank@boldpenguin.com.

‍