by Shawn Bower
One of the challenges as you move your infrastructure to the cloud is keeping your ec2 machines patched. One highly effective strategy is to embrace ephemerality and keep rotating machines in and out of service. This strategy is especially easy to employ using Amazon Linux. check this out from the FAQ (https://aws.amazon.com/amazon-linux-ami/faqs/):
On first boot, the Amazon Linux AMI installs from the package repositories any user space security updates that are rated critical or important, and it does so before services, such as SSH, start.
Another strategy, the one we will focus on in this post, is to be able to patch your ec2 instances in place. This can be useful when your infrastructure has long running 24×7 machines. This can be accomplished using Amazon’s SSM service and the AWS CLI or SDK. First we have to install the SSM agent on all the machines we wish to patch. For example to install the agent (as root) on an Amazon Linux machine in us-east-1 you would:
- curl https://amazon-ssm-us-east-1.s3.amazonaws.com/latest/linux_amd64/amazon-ssm-agent.rpm -o /tmp/amazon-ssm-agent.rpm
- rpm -i /tmp/amazon-ssm-agent.rpm
- rm /tmp/amazon-ssm-agent.rpm
This will download the agent, install it and turn on the service. The agent will contact AWS over the public internet so makes sure your ec2 instance can send outbound traffic. Once your machine registers with SSM you should see it listed in the Run Command GUI (ec2>commands>run a command).
You can also verify that your instance has correctly registered using the AWS CLI, for example:
aws ssm describe-instance-information –instance-information-filter-list key=InstanceIds,valueSet=instance ID
If everything goes well you will get json like below:
If the instance can not be found you will receive an error, if the agent had registered but is no longer responding then you will see the PingStatus is Inactive. Once we have the SSM agent installed we are good to send commands to our ec2 machines including patching them. For our implementation I decided to use tags in ec2 to denote which machines should be auto patched. I also added a tag called “os” so we could support patching multiple OSes. Our ec2 console looks like:
With the tags in place and the SSM agent installed I used the Ruby SDK to create a script that would look for ec2 instances that had the auto_patch tag and then determine the patching strategy based on the OS. This script uses the cucloud module that is being collaboratively developed by folks all over campus. The main script is:
Finally the function that sends the command via SSM is:
This is just one example of how to keep your machines up to date and to improve your security posture. If you have any questions about patching in AWS or using the cloud here at Cornell please send a note to the Cloudification team.