DevOps Practices for Terraform

Introduction

When implementing Terraform in a production environment one can become overwhelmed with questions on DevOps practices for Terraform. When do you run a plan? How does CI/CD work? How can we build artifacts in such a way to quickly roll back? This post will attempt to answer these questions. The concepts discussed is agnostic of tooling; however, to keep grounded and provide examples I will reference Azure DevOps.

When to run a plan

This is a common first question. When and how often should we run a Terraform plan. The answer from my perspective is a minimum of three times.

• Submitted a Pull Request (CI)
• After the PR has been merged
• Prior to Terraform deployment

Submitted a Pull Request (CI)

Running the plan as part of a Continuous Integration (CI) check ensures the code being submitted will provide information to the developer and approval of what actions Terraform will take.

After the PR has been merged

Running the plan again after the code has been merged, ideally as part of your Continuous Deployment (CD) will help identify if the plan has drifted from the time of PR approval. In addition this plan will provide anyone approving a deployment what action Terraform will perform.

Prior to Terraform deployment

Here is the “last snapshot” of what Terraform will perform. Now this can be done as a final check prior to deployment; however, I prefer to run the plan at this time to save for historical/troubleshooting purposes. This helps when we may question what was actually performed as part of our deployment and thus we have a record. Additionally your organization may require an audit as part of their DevOps process accounting for what infrastructure changes have been performed.

How CI/CD Works

Building off our plan questions a recommended CI/CD workflow could look like:

1. Run Terraform validate and plan when submitting a pull request against ALL deployment environments
2. Approve the pull request based on the plan output
3. Run validate and plan again post merge
4. Publish Terraform files as a pipeline artifact
5. Run plan and apply to development environment
6. Approval needed to next environment
7. Run and save plan prior to Terraform apply
8. Repeat for x environments

The key thing to keep in mind here noting the repetitive tasks that are occurring. In this case the plan, validate and apply steps will all occur more then once. This is where we need to view these steps from a DRY (Don’t Repeat Yourself) lens and create templates for these actions. This process will have us define these actions one time and reuse them multiple times with different parameters. In our case passing in different environment information would be parameters. If wanting to see how to do this in DevOps I’d advise you to check out my post Terraform, CI/CD, Azure DevOps, and YAML Templates or check out my series on Microsoft’s Health and Life Sciences blog on YAML Pipelines.

Building off this the comment I have on running a plan against all environments is another important note. This may at first seem like overhead; however, this ensures we are failing faster in the case of forgetting to add environment specific variables or components as well as get ahead of any potential issues in our upstream environments.

To do this with a security best practice in mind we should ensure that our pipeline is running these plans scoped to different accounts who have the appropriate access. It is easier to have one account for all our Terraform actions across environments; however, this can expose you to cross environment contamination.

Creating Build Artifacts

When running Terraform as part of pipeline it is important to save the Terraform files we want to execute as part of our build process. In Azure DevOps this is a PublishPipelineArtifact task. In basic terms we will need to copy our terraform files and everything needed to deploy them to the agent. By doing this we can ensure we have a static version of the Terraform files we will be deploying. This concept is important to understand as now we can also roll back to this version since we have everything needed to redeploy our code. Additionally our organization may require x amount of versions of production code to be stored.

The alternative to this approach would be having to cherry pick and revert code in our git repositories. Which, if you haven’t done before, is not an ideal situation.

Conclusion

Hopefully this blog has provided a rough outline on some DevOps practices with Terraform. We discussed when to run a plan, how CI/CD works, and creating build artifacts. If you’d like to check out more material please refer to my post Terraform, CI/CD, Azure DevOps, and YAML Templates or check out my series on Microsoft’s Health and Life Sciences blog on YAML Pipelines.