Provision and manage your AWS resources with Terraform

Infrastructure as Code with Terraform and AWS

Avatar of Author
Jean-Christophe BaeyJuly 07, 2021
Puy Pariou, Orcines, France
Photo by Adrien Brun on Unsplash

TL;DR;

When you provision Cloud based resources there are 4 main options:

This article will focus on getting started with Terraform.

Local installation

Install latest version

$ brew install terraform

Install a specific version using Brew

When working with others on the same remote resources, you need to be aligned on the Terraform version you use.

$ brew install terraform@0.13.1

Install a specific version by hand

$ sudo mv ./terraform /usr/local/bin
  • Or using a 1 liner:
# curl -sSL https://releases.hashicorp.com/terraform/0.13.1/terraform_0.13.1_darwin_amd64.zip > /tmp/terraform.zip \
&& unzip /tmp/terraform.zip -d /usr/local/bin
  • You can also download specific Terraform plugins in case you don't want any internet access (for instance, in a Linux docker CI image)
$ mkdir -p /opt/terraform/plugins
$ curl -sSL https://releases.hashicorp.com/terraform-provider-aws/3.53.0/terraform-provider-aws_3.53.0_linux_amd64.zip > /tmp/google-plugin.zip && unzip /tmp/google-plugin.zip -d /opt/terraform/plugins
  • Then when you will run the terraform CLI, you can specify where to find the downloaded plugins:
terraform -plugin-dir=/opt/terraform/plugins

Multiple environment

Basically we would like to have 3 environments:

  • dev
  • staging
  • prod

Our project folder hierarchy

.
├── Makefile
├── README.md
├── main.tf
├── sns.tf
├── varfiles
│   ├── dev.tfvars
│   ├── staging.tfvars
│   └── prod.tfvars
└── variables.tf

variables.tf

It will hold the variable definitions including default values.

variable "company" {
  default     = "learnpulse"
  description = "Company name"
}

variable "region" {
  default = "eu-west-1"
}

variable "resource_owner" {
  default = "me@my-company.com"
}

variable "env" {
  description = "Environment name (dev/stg/prod)"
}

varfiles

It will contains only the override of variables per environment.

dev.tfvars

env="dev"

prod.tfvars

env="prod"

main.tf

terraform {
  # Define when the state will be stored:
  backend "s3" {
    bucket = "terraform.learnpulse"
    region = "eu-west-1"
  }

  # You can put some requirement on the Terraform version
  required_version = ">= v0.12.18"
}

# Configure the default AWS Provider
provider "aws" {
  region = var.region

  # Common tags to be assigned to all resources
  default_tags {
    tags = {
      "${var.company}:environment-type" = var.env
      "${var.company}:technical-owner"  = var.resource_owner
      "${var.company}:managed-by"       = "terraform"
    }
  }
}

# Alterative provider for ACM (ACM can only be created on us-east-1 if used by Cloudfront)
provider "aws" {
  alias  = "virginia"
  region = "us-east-1"

  # Common tags to be assigned to all resources
  default_tags {
    tags = {
      "${var.company}:environment-type" = var.env
      "${var.company}:technical-owner"  = var.resource_owner
      "${var.company}:managed-by"       = "terraform"
    }
  }  
}

data "aws_caller_identity" "current" {}

Notice that we assigned default tags on all resources that support tagging following AWS best practices.

sns.tf

This is an example of a SNS resource creation:

# SNS
resource "aws_sns_topic" "email_ses_notifications" {
  name = "${var.env}-ses-notifications"
}

State management

  • Terraform is stateful. It will store the list of the created resources it manages.

  • If you have multiple environments, you should avoid sharing the state between the environment otherwise your state will be mixed up.

  • There are 2 options on Terraform, you can use either the workspace option or specify where (called the backend) is the state folder for each environment.

Using workspace

$ terraform workspace new dev
$ terraform workspace select dev
$ terraform plan -var-file=varfiles/dev.tfvars

Using backend (recommended)

This option is recommended as it is compatible with a CI implementation. Basically, our main.tf file will contains the S3 based backend, meaning the location of the state files: they will be stored remotely on a S3 bucket (that needs to be created first manually).

terraform {
  backend "s3" {
    bucket = "terraform.learnpulse"
    region = "eu-west-1"
  }
}
  • The folder or key will be defined during the call of terraform init. This will be done in our makefile.

Makefile

ENV ?= dev
KEY := terraform/state/$(ENV)

tf-init:
	rm -rf .terraform
	terraform init -backend-config="key=$(KEY)"

tf-plan:
	terraform plan -var-file=varfiles/$(ENV).tfvars -out .terraform/plan.out

tf-output:
	terraform output

tf-plan-destroy:
	terraform plan -refresh=true -var-file=varfiles/$(ENV).tfvars -out .terraform/plan_destroy.out -destroy

tf-apply:
	terraform apply .terraform/plan.out

tf-destroy:
	terraform apply .terraform/plan_destroy.out

Usage

Terraform requires the AWS credentials, refer to the documentation.
By default, it will used what is used for your aws cli, you can check who you are using:

$ aws sts get-caller-identity --output json

Without anu environment variable, aws will used what is defined in the ~/.aws/profile file.

profile

[default]
output = json
region = eu-west-1
aws_access_key_id = <key id>
aws_secret_access_key = <secret>

(if you are using several profiles, you can select the one your want to use with an environment variable: export AWS_PROFILE=user1, refer to the documentation.)

Dev

  • To create/update resources:
$ cd terraform
$ export ENV=dev
$ make tf-init tf-plan
$ make tf-apply
  • To remove resources:
$ make tf-plan-destroy 
$ make tf-destroy

Staging

  • To create/update resources:
$ cd terraform
$ export ENV=staging
$ make tf-init tf-plan
$ make tf-apply
  • To remove resources:
$ make tf-plan-destroy 
$ make tf-destroy

Prod

  • To create/update resources:
$ cd terraform
$ export ENV=prod
$ make tf-init tf-plan
$ make tf-apply
  • To remove resources:
$ make tf-plan-destroy 
$ make tf-destroy

Start defining your stack

  • Follow the documentation to create your application resources on AWS.
  • Terraform is really powerful for linking resources together. It computes the dependency tree for you.

For instance, if you need to reference the arn of a resource created by terraform, just use <terraform_resource_type>.<name>.<output>, in the example below aws_sns_topic.email_ses_notifications.arn will be replaced by terraform at the execution time:

resource "aws_ses_event_destination" "email_campaigns_sns" {
  name                   = "${var.env}-campaign-email-notifications-to-sns"
  configuration_set_name = aws_ses_configuration_set.email_campaigns.name
  enabled                = true
  matching_types         = ["bounce", "complaint", "reject"]

  sns_destination {
    topic_arn = aws_sns_topic.email_ses_notifications.arn
  }
}

👉 Don't forget to follow me on Twitter to be notified when new posts are available!