June | 2016 | Cloudification Services

by Shawn Bower

The Cloud Services team in CIT maintains docker images for common pieces of software like apache, java, tomcat, etc. One of these images that we maintain is Cornellized Jenkins images. This image contains Jenkins with the oracle client and Cornell OID baked in. One of the easiest way to get up and running in AWS with this Jenkins instance is to use Elastic Beanstalk which will manage the infrastructure components. Using Elastic Beanstalk you don’t have to worry about patching as it will manage the underlying OS of your ec2 instances. The Cloud Services team releases patched version of Jenkins image on a weekly basis. If you want to stay current the you just need to kick off a new deploy in Elastic Beanstalk. Let’s walk through the process of getting this image running on Elastic Beanstalk!

A.) Save Docker Hub credentials to S3

INFO:

Read about using private Docker repos with Elastic Beanstalk.

We need to make our DTR credentials available to Elastic Beanstalk, so automated deployments can pull the image from the private repository.

Create an S3 bucket to hold Docker assets for your organization— we use cu-DEPT-docker
Login to Docker docker login dtr.cucloud.net
Upload the local credentials file ~/.docker/config.json to the S3 bucket cu-DEPT-docker/.dockercfg
Unfortunately, Elastic Beanstalk uses an older version of this file named .dockercfg.json The formats are slightly different. You can read about the differences here.

For now, you’ll need to manually create .dockercfg & upload it to the S3 bucket cu-DEPT-docker/.dockercfg

B.) Create IAM Policy to Read The S3 Bucket

Select Identity and Access Management for the AWS management console
Select Policies
Select “Create Policy”
Select “Create Your Own Policy”
Create a policy name “DockerCFGReadOnly,” see the example policy provided.

Below is an example Policy for reading from a S3 bucket.
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1466096728000",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::cu-DEPT-dockercfg"
            ]
        },
        {
            "Sid": "Stmt1466096728001",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:HeadObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::cu-DEPT-dockercfg/.dockercfg"
            ]
        }
    ]
}

C.) Setup the Elastic Beanstalk environment

Create a Dockerrun.aws.json fileHere’s an example.

{
  "AWSEBDockerrunVersion": "1",
  "Image": {
    "Name": "dtr.cucloud.net/cs/jenkins:latest"
 },
 "Ports": [
   {
     "ContainerPort": "8080"
   }
 ],
 "Authentication": {
   "Bucket": "cu-DEPT-dockercfg",
   "Key": ".dockercfg"
 },
 "Volumes": [
   {
     "HostDirectory": "/var/jenkins_home",
     "ContainerDirectory": "/var/jenkins_home"
   },
   {
     "HostDirectory": "/var/run/docker.sock",
     "ContainerDirectory": "/var/run/docker.sock"
   }
 ]
}

The Authentication section refers to the Docker Hub credentials that were saved to S3.

The Image section refers to the Docker image that was pushed to Docker Hub.

We will also need to do some setup to instance using .ebextenstions. Create a folder called “.ebextensions” and inside that folder create a file called “instance.config” Add the following to the file:

container_commands:
 01-jenkins-user:
 command: useradd -u 1000 jenkins || echo 'User already exist!'
 02-jenkins-user-groups:
 command: usermod -aG docker jenkins
 03-jenkins-home:
 command: mkdir /var/jenkins_home || echo 'Directory already exist!'
 04-changeperm:
 command: chown jenkins:jenkins /var/jenkins_home

Finally create a zip file with the Dockerrun.aws.json file and the .ebextenstions folder.
```
zip -r jenkins-stalk.zip Dockerrun.aws.json .ebextensions/ 
```

D.) Setup Web Server Environment

Choose Docker & Load balancing, autoscaling
Select your local zip file that we created earlier ( jenkins-stalk.zip ) as the “Source” for the application version section
Set the appropriate environment name, for example you could use jenkins-prod
Complete the configuration details

NOTE: There are several options beyond the scope of this article.

We typically configure the following:
Complete the Elastic Beanstalk wizard and launch. If you are working with a standard Cornell VPC configuration, make sure the ELB is in the two public subnets while the EC2 instances are in the private subnets.

NOTE: You will encounter additional AWS features like security groups etc… These topics are beyond the scope of this article. If presented with a check box for launching inside a VPC you should check this box.

The container will not start properly the first time. Don’t panic.

We need to attach the IAM Policy we built earlier to the instance role used by Elastic Beanstalk.

Select Identity & Access Management for the AWS management console

Select “Roles” then select “aws-elasticbeanstalk-ec2-role”

Attach the “DockerCFGReadOnly” Policy to the role

E.) Re-run the deployment in Elastic Beanstalk. You can just redeploy the current version.

Now find the URL to your Jenkins environment

And launch Jenkins

SUCCESS !

F.) (optional) Running docker command inside Jenkins

The Jenkins image comes with docker preinstalled so you can run docker build and deploys from Jenkins. In order to use it we need to make a small tweak to the Elastic Beanstalk Configuration. This is because we are keeping the docker version inside the image patched and on the latest commercially supported release however Elastic Beanstalk currently supports docker 1.9.1. To get things working we need to add an environment variable to use an older docker API. First go to configurations and select the cog icon under Software Configuration.

Now we need to add a new environment variable, DOCKER_API_VERSION and set its value to 1.21 .

That is it! Now you will be able to use the docker CLI in your Jenkins jobs

Conclusion

Within a few minutes you can have a managed Jenkins environment hosted in AWS.
There are a few changes you may want to consider for this environment.

Changing the autoscaling group to min 1 and max 1 makes sense since the Jenkins state data is stored on a local volume. Having more than one instance in the group would not be useful.
Also considering the state data, including job configuration, is stored on a local volume you will want to make sure to backup the EBS volume for this instances. You could also look into a NAS solution such as Elastic File Service to store state for Jenkins, this would require a modification to /var/jenkins_home path.
It is strongly encouraged that an HTTP SSL listener is used for the Elastic Load Balancer(ELB) and that the HTTP listener is turned off, to avoid sending credentials in plain text.

The code used in this blog is available at: https://github.com/CU-CloudCollab/jenkins-stalk, Please free to use and enhance it.

If you have any questions or issues please contact the Cloud Services team at cloud-support@cornell.edu

by Shawn Bower

The term DevOps is thrown around so much that it is hard to pin down it’s meaning. In my mind DevOps is about culture shift in the IT industry. It is about breaking down silos, enhancing collaboration, and challenging fundamental design principles. One principal that has been turned on its head because of the DevOps revolution is the no single point of failure design principle. This principle asserts simply that no single part of a system can stop the entire system from working. For example, in the Financial system the database server is a single point of failure. If it crashes we cannot continue to serve clients in any fashion. In DevOps we accept that failure is the norm and we build our automation with that in mind. In AWS we have many tools at our disposal like auto scaling groups, elastic load balances, multi-az RDS, dynamodb, s3, etc. When architecting for the cloud keeping these tools in mind is paramount to your success.

When architecting a software system there are a lot of factors to balance. We want to make sure our software is working and performant as well as cost effective. Let’s look at a simple example of building a self healing website that requires very little infrastructure and can be done for low cost.

The first piece of infrastructure we will need is something to run our site. If its a small site we could easy run it on a t2.nano in AWS which would cost less than 5 dollars a month. We will want to launch this instance with an IAM profile with the policy AmazonEC2RoleforSSM. This will allow us to send commands to the ec2 instance. We will also want to install the SSM agent, for full details please see: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-ssm-agent.html. Once we have our site up, we will want to monitor its health. At Cornell you can get free access to the Pingdom monitoring tool. Using Pingdom you can monitor your sites endpoint from multiple locations around the world and get alerted if your site is unreachable. If you don’t already have a Pingdom account please send an email to cloud-support@cornell.edu. So now that we have our site running and a Pingdom account lets set up an uptime monitor.

We are doing great! We have a site, we are monitoring, and we will be alerted to any downtime. We can now take this one step further and programmatically react to Pingdom alert using their custom webhook notifier. We will have to build an endpoint for Pingdom to send the alert to. We could use Lambda and API gateway, which is a good choice for many reasons. If we want we could start even simpler by creating a simple Sinatra app in Ruby.

This is a very simple bit of code that could be expanded on. It creates an endpoint called “/webhook” which first looks for an api-key query parameter. This application should be run using SSL/TLS as it sends the key in clear text. That key is compared against and environment variable that should be set before the application is launched. This shared key is a simple mechanism for security only in place to stop the random person from hitting the endpoint. For this example it is good enough but could be vastly improved upon. Next we look at the data that Pingdom has sent, for this example we will only react to DOWN alerts. If we have an alert in the DOWN state then we will query a table in DynamDB that will tell us how to react to this alert. The schema looks like:

check_id – This is the check id generated by Pingdom
type – is the plugin type to use to respond to the Pingom alert. The only implemented plugin is SSM which uses Amazon’s SSM to send a command to the ec2 host.
instance_id – This is the instance id of the ec2 machine running our website
command – This is the command we want to send to the machine

We will use the the type from our Dynamo table to respond to the down alert. The sample code I have provided only has one type which uses Amazon’s SSM service to send commands to the running ec2 instance. The plugin code looks like:

This function takes the data passed in and send the command from our Dynamo table to the instance. The full sample code can be found at https://github.com/CU-CloudCollab/pingdom-webhook. Please feel free to use and improve this code. Now that we have a simple webhook app we will need to deploy it to an instance in AWS. That instance will have to use an IAM profile that will allow it to read from our Dynamo table as well as send SSM commands. Again we can use a t2.nano so our cost at this point is approximately 10 dollars a month.

We need to make Pingdom aware of out new web hook endpoint. To do that navigate to “Integrations” and click “Add integration.”

The next form will ask for information about your endpoint. You will have to provide the DNS name for this service. While you could just use the IP of the machine its highly encourage to use a real host name with SSL.

Once you have added the integration it can be used by any of the uptime checks. Find the check you wish to use and click the edit button

Then scroll to the bottom of the settings page and you will see a custom hooks section. Select your hook and you are all done!

This is a simple and cost effective solution to provide self-healing to web applications. We should always expect failure will occur and look for opportunities to mitigate it’s effects. DevOps is about taking a wholistic approach to your application. Looking at the infrastructure side as we did in this blog post but also looking at the application it-self. For example move to application architectures which are stateless. Most importantly automate everything!

How to run Jenkins in ElasticBeanstalk