Here’s a little gotcha with AWS that I keep running into time and time again. By default, the aws command line interface, and AWS API libraries such as boto3, will always use the us-east-1 (Virginia) region by default, even when running on EC2 instances in other regions.

This is not what you expect, and it is almost never what you want.

There is an issue on the awscli GitHub issue tracker to fix this, but it is still open four years after first being raised, with no indication when (or even whether) it will ever be addressed.

User @BradErz suggests including these lines in your user_data to set the default region:

region=$(curl http://169.254.169.254/latest/dynamic/instance-identity/document|grep region|awk -F\" '{print $4}')
echo "[default]" > /root/.aws/config
echo "region = ${region}" >> /root/.aws/config

Note however that this will only set the default region for the root user; you will need to configure aws-cli separately for any other logins on your instance.

Annoying as this behaviour is, I would be surprised to see it fixed any time soon, as it would be a breaking change.

Posted at 08:00 on 28 November 2017.
Tagged: aws, ec2

I had to start some EC2 instances programmatically from inside an AWS lambda function. My code looked something like this:

import boto3

def handler(event, context):
    client = boto3.client('ec2')
    client.start_instances(InstanceIds=['i-0123456789abcdef0', ...])

This worked fine when I ran it from the command line, but when I ran it from inside the lambda, one particular instance stubbornly refused to start, even though the lambda ran without errors.

It turned out that the problem was a permissions issue. This particular instance had an additional encrypted EBS volume attached. The call to start_instances() was failing silently.

To fix this, make sure that the role under which your code runs is granted the kms:CreateGrant permission.

It took me a bit of trial and error to figure out which permission to add, but I wanted to make sure I got this one right. You should never give your code any more permissions than the bare minimum it needs in order to do what it needs to do. Unfortunately, figuring out exactly which permissions your code needs to run can sometimes be a bit of a challenge…

Posted at 08:00 on 24 October 2017.
Tagged: ec2, aws lambda, aws

For those of you not familiar with AWS, t2.nano is the smallest size of EC2 instance (virtual servers that you can spin up and tear down at will) that they sell. It gives you one vCPU and 512MB of memory, and it costs just $0.0063 per hour, which, at current exchange rates with VAT added on top, works out at about £4.20 a month.

Its main limitation is CPU time. While it allows you to utilise 100% of the CPU for short periods of time, it limits you to 5% on average over 24 hours by a system of CPU credits. You get enough CPU credits for a maximum of 72 minutes of CPU time a day, and if you run out of credits, your CPU time is throttled to 5%. This means that you can’t use it for CPU-intensive tasks such as trying to compare human and chimp DNA, but even so, there are still plenty of things that you can use it for nonetheless. Here are some examples:

1. Bastion servers. A bastion is a server that acts as a single point of access for certain services such as SSH or remote desktop, to reduce your network’s attack surface area. If you’re doing things right, with immutable servers, you should only occasionally need to ssh into your servers, if at all. Or, in other words, if a t2.nano instance is too small for your bastion requirements, you’re Doing It Wrong.

2. A cheap-and-cheerful alternative to a NAT gateway. For the cost-conscious, the price of a NAT gateway — needed to let you connect to the Internet from any of your servers that doesn’t have its own public IP address — can come as a shock. $0.048/hour works out at $420/year — a lot of money if all you’re doing with it is downloading software updates every once in a while. But it’s fairly easy to configure an Ubuntu instance as an alternative — and a t2.nano instance works out at a seventh of the price.

3. A source control, CI server or wiki for small teams. A t2.nano instance should easily be sufficient to act as a Gitea or Jenkins server for a Scrum team of about 5-10 people, possibly more. Note however that Go.CD and GitLab both require 1GB of memory, so those options will require larger instance sizes.

4. Low-traffic blogs, personal portfolio websites and the like. A t2.nano instance can handle hundreds of monthly visitors to your average personal website. Additionally, by putting it behind a CDN such as CloudFront or CloudFlare, you can get even more bang for your buck out of it and possibly scale into the thousands or beyond.

There may well be other cases where a t2.nano instance works fine. You should take a look at your metrics, and if your CPU usage on your larger instance is constantly low, you may well benefit from trying the smaller size. One point to note is that the t2.nano instance is not eligible for AWS’s free tier; only the t2.micro instance is available free of charge. However, if your free tier allowance has expired, or if you have maxed it out, do consider the t2.nano as a viable option. You can probably do a lot more with it than you think.

Posted at 08:00 on 11 May 2017.
Tagged: aws, ec2