james mckay dot net
because there are few things that are less logical than business logic

Posts tagged: aws

You have to tell AWS CLI that your EC2 instance is not in Virginia

Here’s a little gotcha with AWS that I keep running into time and time again. By default, the aws command line interface, and AWS API libraries such as boto3, will always use the us-east-1 (Virginia) region by default, even when running on EC2 instances in other regions.

This is not what you expect, and it is almost never what you want.

There is an issue on the awscli GitHub issue tracker to fix this, but it is still open four years after first being raised, with no indication when (or even whether) it will ever be addressed.

User @BradErz suggests including these lines in your user_data to set the default region:

region=$(curl|grep region|awk -F\" '{print $4}')
echo "[default]" > /root/.aws/config
echo "region = ${region}" >> /root/.aws/config

Note however that this will only set the default region for the root user; you will need to configure aws-cli separately for any other logins on your instance.

Annoying as this behaviour is, I would be surprised to see it fixed any time soon, as it would be a breaking change.

Introducing Lambda Tools: a new framework for deployment to AWS Lambda

Lambda Tools is a new project that I’ve been working on over the past few weeks or so. It is a build and deployment toolkit, written in Python, to make it easier to work with AWS Lambda functions. It started out as a user story in our backlog at work but since then it’s grown into a full blown open source project with a life of its own.

The “serverless” model offered by AWS Lambda is a useful and potentially cost-effective one, especially for scheduled tasks that only need to run every so often and don’t require a lot of resources. The “free tier” doesn’t expire at the end of your initial twelve month trial, which is an added bonus.

The downside is that it can be tricky to work with. If your function requires additional libraries, it starts to get a bit more complex, and if any of these are written partly in C and require special compilation, things can get pretty messy. On top of that, you will want to write unit tests for your functions and set up some sort of Continuous Delivery pipeline for them.

Lambda Tools includes several features to make these things easier. For example, it gives you an option to build your function in a Docker container to avoid these messy “invalid ELF header” errors. You configure it simply by creating a YAML file called aws-lambda.yml, which might look like this for example:

version: 1

    runtime: python3.6
      source: src/hello_world
        - file: requirements.txt

      handler: hello.handler
      role: service-role/NONTF-lambda

      description: A basic Hello World handler
      memory_size: 128
      package: build/hello_world.zip
      region: eu-west-2
      timeout: 60

        Account: Marketing
        Application: Newsletters

Here are some of the other ideas that I’m thinking of implementing for it:

  • A unit test runner
  • Support for Python 2.7
  • Support for languages other than Python (.NET, Node.js, and so on)
  • Integration with Terraform:
    • The ability to plug it into Terraform’s external data source provider
    • The ability to read configuration from stdin
    • A scaffolding engine to generate Terraform modules on the fly
  • Integration with triggers such as CloudWatch cron jobs, API Gateway, and so on
  • A full-blown sample application
  • The ability to include or exclude specific files when building your package

At the moment it only supports Python 3.6 but nevertheless it is in a usable state. You can install it using pip install lambda_tools and it includes full instructions in the readme file on the GitHub repository. Additionally, if you’re interested in getting involved with its development, feel free to fork the project and send me a pull request or three. If it’s anything more complex, raise a ticket on the GitHub issue tracker and we’ll chat about it there.

Programmatically starting an AWS instance with an encrypted EBS volume attached

I had to start some EC2 instances programmatically from inside an AWS lambda function. My code looked something like this:

import boto3

def handler(event, context):
    client = boto3.client('ec2')
    client.start_instances(InstanceIds=['i-0123456789abcdef0', ...])

This worked fine when I ran it from the command line, but when I ran it from inside the lambda, one particular instance stubbornly refused to start, even though the lambda ran without errors.

It turned out that the problem was a permissions issue. This particular instance had an additional encrypted EBS volume attached. The call to start_instances() was failing silently.

To fix this, make sure that the role under which your code runs is granted the kms:CreateGrant permission.

It took me a bit of trial and error to figure out which permission to add, but I wanted to make sure I got this one right. You should never give your code any more permissions than the bare minimum it needs in order to do what it needs to do. Unfortunately, figuring out exactly which permissions your code needs to run can sometimes be a bit of a challenge…

You probably only need a t2.nano instance for that

For those of you not familiar with AWS, t2.nano is the smallest size of EC2 instance (virtual servers that you can spin up and tear down at will) that they sell. It gives you one vCPU and 512MB of memory, and it costs just $0.0063 per hour, which, at current exchange rates with VAT added on top, works out at about £4.20 a month.

Its main limitation is CPU time. While it allows you to utilise 100% of the CPU for short periods of time, it limits you to 5% on average over 24 hours by a system of CPU credits. You get enough CPU credits for a maximum of 72 minutes of CPU time a day, and if you run out of credits, your CPU time is throttled to 5%. This means that you can’t use it for CPU-intensive tasks such as trying to compare human and chimp DNA, but even so, there are still plenty of things that you can use it for nonetheless. Here are some examples:

1. Bastion servers. A bastion is a server that acts as a single point of access for certain services such as SSH or remote desktop, to reduce your network’s attack surface area. If you’re doing things right, with immutable servers, you should only occasionally need to ssh into your servers, if at all. Or, in other words, if a t2.nano instance is too small for your bastion requirements, you’re Doing It Wrong.

2. A cheap-and-cheerful alternative to a NAT gateway. For the cost-conscious, the price of a NAT gateway — needed to let you connect to the Internet from any of your servers that doesn’t have its own public IP address — can come as a shock. $0.048/hour works out at $420/year — a lot of money if all you’re doing with it is downloading software updates every once in a while. But it’s fairly easy to configure an Ubuntu instance as an alternative — and a t2.nano instance works out at a seventh of the price.

3. A source control, CI server or wiki for small teams. A t2.nano instance should easily be sufficient to act as a Gitea or Jenkins server for a Scrum team of about 5-10 people, possibly more. Note however that Go.CD and GitLab both require 1GB of memory, so those options will require larger instance sizes.

4. Low-traffic blogs, personal portfolio websites and the like. A t2.nano instance can handle hundreds of monthly visitors to your average personal website. Additionally, by putting it behind a CDN such as CloudFront or CloudFlare, you can get even more bang for your buck out of it and possibly scale into the thousands or beyond.

There may well be other cases where a t2.nano instance works fine. You should take a look at your metrics, and if your CPU usage on your larger instance is constantly low, you may well benefit from trying the smaller size. One point to note is that the t2.nano instance is not eligible for AWS’s free tier; only the t2.micro instance is available free of charge. However, if your free tier allowance has expired, or if you have maxed it out, do consider the t2.nano as a viable option. You can probably do a lot more with it than you think.