Sharing data between Terraform configurations

Posted at 11:00 on 28 September 2016

I've been doing quite a lot of work with Terraform lately, Hashicorp's excellent infrastructure-as-code software, to provision some infrastructure on AWS.

When designing your infrastructure, it is often necessary to break it up into different configurations. For example, you may want to have one set of configuration for your data (databases, S3 buckets, EBS volumes, and so on), and another set of configuration for various compute resources (EC2 instances, ECS containers, AWS Lambda scripts, and so on) -- for example, to allow you to tear down your development environment when you are not using it in order to reduce costs, while preserving your data. Or, you may want to reference your DNS hosted zones in multiple different configurations.

When you do this, you will inevitably need to pass state information from one configuration through to the next -- for example, EBS volume identifiers, IP addresses, and so on. If you are using remote state, you can do this quite simply using a remote state data source.

In this example, I'll assume that you have configured Terraform to use remote state hosted in an S3 bucket, that you are using a separate configuration called "dns" for your DNS hosted zones, and that you wish to reference them in a different configuration.

First declare the values that you wish to reference as outputs:

output "zone_id" {
    value = "${module.dns.zone_id}"
}
 
output "zone_name" {
    value = "${module.dns.zone_name}"
}

In your target configuration, declare the remote state data source:

data "terraform_remote_state" "dns" {
    backend = "s3"
    config {
        bucket = "name-of-your-S3-bucket"
        key = "dns"
        region = "eu-west-1"
    }
}

Then, whenever you need to use any of the remote values, you can reference them as follows:

resource "aws_route53_record" "www" {
   zone_id = "${data.terraform_remote_state.dns.zone_id}"
   name = "www.${data.terraform_remote_state.dns.zone_name}"
   type = "A"
   ttl = "300"
   records = ["${aws_eip.lb.public_ip}"]
}

Note that you will need to run terraform apply on the source environment, even if you have not made any changes, before you attempt to do anything with the target environment. This is because you need to make sure that the outputs have been written into the Terraform state file, and are available to the other modules that depend on them.

Hat tip: Thanks to Paul Stack for pointing me in the right direction here.