Terraform and Multiple AWS Accounts

In the last 7 or so years I’ve been through a bunch of AWS Premier partners, and during this time I’ve saw so many environments being managed in so many ways… cloudfrustrat..formation, terraform, terragrunt, shells scripts, some obscure abstraction predating CDKs… you name it, I probably stumbled upon it or created something using it.

My most pleasant experiences, and the less headaches, came from using Terraform. Now even a good tool can be used in some really crappy ways, and over many iterations I arrived into something that doesn’t irk me out and keeps my life reasonably simple, as well as being easy enough that onboarding a new team member to it won’t take forever.

Now, disclaimer, this is all from my experience, and what works for me, you are welcome to disagree, I’m not looking for a discussion on the matter. I just had enough discussions around it that I figured I’d document my thoughts on it so I can point myself, and other people that ask me on the topic, to this post.

First let’s walk back from requirements. We want to:

• manage multiple AWS accounts from a single repository
• have consistent style
• automated style checks
• automated security checks for best practices
• use versioned modules

Let’s walk through through a complete strategy for using Terraform to manage multiple AWS accounts while keeping those 4 requirements in mind.

Managing Multiple Accounts From A Single Repository #

Why a single repository? Well, because I like it 😄 and it keeps most things in a single place. You can definitely do multiple repositories, and it might even make your life easier in some aspects.

Thinking of an AWS Landing Zone style deployment you might have at least a couple accounts like:

• management - your AWS organizations root account
• identity - AWS SSO or IAM users/groups/roles
• audit - for log and backup archiving
• shared services - for any kind of services that are shared by multiple accounts
• production
• staging (or qa, or uat, preprod, whatever you want to name where you test stuff before production)
• development

You might have more or less accounts, but for this post, we will work with this set.

I’ll start with a folder at the root of the repository for each of these environments/accounts.

.
├── env-audit
├── env-development
├── env-identity
├── env-management
├── env-production
├── env-shared-services
└── env-staging

Now let us add a subfolder for region and people will grab their pitchforks and scream “premature optimization”! I’ll remind you that some AWS services only run on us-east-1, and that we’re not always in that region, and even if we are, it is still worth it to be ready for it instead of reworking a lot of code to support it later.

So now our folder structure looks like this:

.
├── env-audit
│  └── ca-central-1
├── env-development
│  └── ca-central-1
├── env-identity
│  └── ca-central-1
├── env-management
│  └── ca-central-1
├── env-production
│  └── ca-central-1
├── env-shared-services
│  └── ca-central-1
└── env-staging
   └── ca-central-1

Cool! Now lets deploy something!

Where am I saving my Terraform state? #

Anywhere that is not on git or your laptop!

This can be really simple, or really complicated.

Nobody works in isolation, so you will need to make use of two features of Terraform, remote backend and state locking.

Almost everybody will say S3 bucket (for state) + DynamoDB table (for locking), to which I’ll say maaaaayyyyybe.

While S3+DDB is taken as a standard, you might not need it or even want it. Why? Because of access permissions and auditing.

It is common to do state lookups to grab outputs from some Terraform code to use in another, and for that code X needs access to state from code Y, and when that involves a S3 bucket it involves IAM permissions. Here is where things can get really simple or complicated.

Depending on how you structure your S3 state bucket(s), you might have access to all the state files, and cool, no problems, this is as simple as it gets. Now this is most likely not the case if you are in a larger company with multiple people working with Terraform, and thats where things can get really complicated.

I honestly want to avoid S3 and DDB like the plague if I can, there is a simpler way, and it is called Terraform Cloud.

OMG! I’ll need to pay!!! Worry not! If you are a small team, Terraform Cloud is free up to a couple users, and you can have as much state saved there as you need. If you are a larger team, you should have budget for some SaaS software, use it, it is worth it!

I am not going to go any deeper on this topic, we will be using Terraform Cloud for storing our state.

Setting-up Some Automation #

Ok! Time to roll up our sleeves.

For everything I do, I’ll create a folder under my region folder, each of these folders will have their own state (a workspace in Terraform Cloud, TFC going forward).

I like having docs, but I don’t like writing a lot of them, I also believe that consistency makes things easier for new people being onboarded, and for me when working.

Let’s set some standards, and some tooling that can help us stick to it:

• we want consistent formatting, terraform fmt will cover the Terraform code, an editorconfig compatible addon for our text editor will cover everything else
• we want some basic docs on the code we’re writing, terraform-docs will do that for us by running terraform-docs markdown . --output-file README.md
• we want to be secure by default, there are plenty of tools out there, since I had AWS SA’s asking me to use checkov multiple times on security reviews, let’s just use it. checkov -d . --framework terraform --quite will nag us about every little thing security wise (sometimes too much, and we might need to ignore a warning here and there)

If you’re working on multiple projects, you might need different versions of terraform for it, and there are several tools that can help you with it. I like using asdf-vm to manage my cli tooling versions.

These tools should take care of most of our needs for consistency and some basic automation. Most of them can be plugged in a commit hook, or using the pre-commit framework, or as settings in your editor (VSCode will apply editorconfig and terraform fmt on save).

Editorconfig can be configured by adding a .editorconfig file to the root of your repository, same thing for asdf-vm with .tool-versions.

Another tool that I want to talk about is direnv. In a nutshell you can automagically load environment variables based on definitions in each folder (a .envrc file, which is basically a shell script). Why is this useful here? You will see in the next section :D

Deploying Something #

Oh lord finally!

Ok lets do the classic thing, and deploy a S3 bucket! Checkov will cry a river, but we will do a really simple S3 bucket.

First, I’ll create a folder for this code. I will use my development environment, so under env-development/ca-central-1 I’ll create my bear-s3-bucket.

To deploy a bucket I need to configure an AWS provider, I need to tell it which region I want to deploy in, it would also be good to tell it which account id to use just in case someone messes up the credentials when applying (to ensure we don’t deploy to the wront account).

Our providers.tf file ideally would look something like:

provider "aws" {
  region = "ca-central-1"

  default_tags {
    tags = {
      environment = "development"
      owner       = "bearlabs"
      provisioner = "terraform"
      purpose     = "blog post example"
    }
  }

  allowed_account_ids = ["123456789012"]
}

We are using TFC to manage our state, so now I’ll go to TFC and create a workspace and name it… naming is hard right? I don’t want to have pet names, I’ll just use our environment name and the relative path to this folder as the workspace name, so development-ca-central-1-bear-s3-bucket.

With that, my backend.tf file will look like:

terraform {
  cloud {
    organization = "my-org-name"

    workspaces {
      name = "development-ca-central-1-bear-s3-bucket"
    }
  }

  required_version = ">= 1.3"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~>4.0"
    }
  }
}

Cool we are ready to roll, but… this is a lot of hardcoding!

Organization, workspace name, account id, environment name, all this could be automagically populated, minimizing the chance of someone adding a wrong value (and wasting a lot of time figuring out where).

First, lets make these all variables, and let’s have direnv populate all of them.

Remember that any variable you create in terraform, can have its value populated by an environment variable of the same name prefixed by TF_VAR_, and that other parts of terraform can also be configured with environment variables like the workspace (TF_WORKSPACE) and TFC organization (TF_CLOUD_ORGANIZATION).

To get all that done we will create a .envrc file at each level of our tree.

.
├── .envrc
...
├── env-development
│  ├── .envrc
│  └── ca-central-1
│     ├── .envrc
│     └── bear-s3-bucket
│        └── .envrc
...

At the root level we can define global configuration items:

# stop terraform from asking for input for missing variables
export TF_INPUT=0

# adjusts terraform output to avoid suggesting commands to run next
export TF_IN_AUTOMATION="true"

# the terraform organization name
export TF_CLOUD_ORGANIZATION="my-org-name"

Then at the environment level we can define things that are valid for the entire account:

source_up # tells direnv to walk up the tree looking for other .envrc files

# account id
export TF_VAR_account_id="123456789012"

# environment name
export ENVIRONMENT="development"
export TF_VAR_environment="${ENVIRONMENT}"

At the region level:

source_up

# aws region from the current folder name
export REGION=${PWD##*/}
export TF_VAR_region=${REGION}

And finally in our code folder:

source_up

export TF_WORKSPACE="${ENVIRONMENT}-${REGION}-${PWD##*/}"

Now our backend.tf file can be just:

terraform {
  cloud {}

  required_version = ">= 1.3"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~>4.0"
    }
  }
}

Our remote.tf file:

provider "aws" {
  region = var.region

  default_tags {
    tags = {
      environment = var.environment
      owner       = "bearlabs"
      provisioner = "terraform"
      purpose     = "blog post example"
    }
  }

  allowed_account_ids = [var.account_id]
}

I won’t bore you with the rest of the s3 bucket code, but our final structure should look like:

.
├── .envrc
...
├── env-development
│  ├── .envrc
│  └── ca-central-1
│     ├── .envrc
│     └── bear-s3-bucket
│        ├── .envrc
│        ├── datasources.tf
│        ├── main.tf
│        ├── outputs.tf
│        ├── README.md
│        └── variables.tf
...

The other files purposes:

• datasources.tf - all datasource lookups
• main.tf - all resources (if it gets too extensive, splitting into files by purpose is ok, but maybe you should be creating a module if it got too unwieldly)
• outputs.tf - all outputs
• README.md - :D
• variables.tf - all your variable definitions

Now when you terraform apply your code all those details will be filled automagically, and when you copy this boilerplate to another folder you don’t have to worry about setting up those again ;)

Versioned Modules #

I’ve mentioned versioned modules, and I like to keep them one module per repository, so we can release versions for the modules, and they’re well isolated. Of course you can argue the other way and have all modules in a single repo. Whatever floats your boat, like I said at the start, this is how I do it.

Having versioned modules in Terraform is the same as having versioned packages in whatever programming language you use, it allows updates to the code without breaking previous implementations. Simple as that.

So whenever I use a module, mine or someone else’s, it will always have a pinned version. We’re not using any in our simple example, but keep that in mind.

Pipelines #

Pipelines are that fun thing that there is no right answer to it, but whatever solution you end up with, ensure you use it to:

• keep your code style consistent, like we do with the editor tools we mentioned before, run your terraform init/validate/fmt, editorconfig, checkov and others here
• have an auditing trail of who proposed the change, who approved the code, etc…
• terraform plan. You can use atlantis, Terraform Cloud, your own scripts, but do expose the plan automatically for validation
• terraform apply. On a really small team, I’d be fine with applying manually after approval, but when you have a larger team, it will be easier to coordinate things if it all happens through a pipeline

Whichever platform you’re using you should be fine, this functionality should be supported in all well known CI/CD tooling.

Conclusion #

A little preparation goes a long way, starting with a good structure and supporting tools from the get go will allow you to focus on your work instead of fiddling with “how we should do these things” forever.

Create your repositories, your structure, your automated checks, have documentation explaining why things are the way they are, how they work and examples of “what good looks like”, and keep all this up to date!

It might feel like a lot of work to start with, but this will all pay off the time you need to onboard ANYBODY into your team.

Cheers!