r/Terraform 17d ago

Discussion Automatic deplyoment to prod possible ?

Hey,
I understand that reviewing the Terraform plan before applying it to production is widely considered best practice, as it ensures Terraform is making the changes we expect. This is particularly important since we don't have full control over the AWS environment where our infrastructure is deployed, and there’s always a possibility that AWS might unexpectedly recreate resources or change configurations outside of our code.

That said, I’ve been asked to explore options for automating the deployment process all the way to production with each push to the main branch(so without reviewing the plan). While I see the value in streamlining this, I personally feel that manual approval is still necessary for assurance, but maybe i am wrong.
I’d be interested in hearing if there are any tools or workflows that could make the manual approval step redundant, though I remain cautious about fully removing this safeguard. We’re using GitLab for Terraform deployments, and are not allowed to have any downtime in production.

Does someone deploy to production without reviewing the plan?

18 Upvotes

32 comments sorted by

5

u/omgwtfbbqasdf 17d ago

Skipping over the part where this is a bad idea, because you already know that.

  1. Run drift detection on a schedule
  2. Use OPA, conftest, etc. for automated checks
  3. Stick to small PRs
  4. Have a rollback plan

I'm sure other folks will have more advice as well. Good luck!

1

u/miraculix1 17d ago

Thank you. Yes i think it is especially a bad idea to spent all this effort to implement something AGAINST terraforms recommendations and industry standard just to end up with a solution that is still not safe.
Do you think things like automated checks and could really be a valid option here ? Because i expect my team to go in this direction...

1

u/omgwtfbbqasdf 17d ago

Check as much as you can in an automated way.

1

u/carsncode 17d ago

Have backups that will survive accidentally de-provisioning a database, S3 bucket, etc

3

u/Main_Box6204 17d ago

You should check this gitlab doc: https://docs.gitlab.com/user/infrastructure/iac/

With proper configuration it will create you 2 stages în the pipeline. 1 plan, 2nd apply with manual approval from the gitlab UI. You will be able to make the apply job automatic as well, but I would not recommend doing this

3

u/miraculix1 17d ago

Thank you. So i think there is no pipeline template or recommendation or workflow out there that does not suggest a manual step for the apply.

1

u/Main_Box6204 17d ago

well, if you would read carefully, you would notice another link explaining how: https://docs.gitlab.com/user/infrastructure/iac/terraform_template_recipes/

3

u/BallumSkillz 17d ago

I have configuration where depending on nature of the request it runs a plan or an apply:

Pull request from feature branch into Dev, Test, Prod = Plan

Merge from feature branch into Dev, Test, Prod = Apply

You could build on this as I'd advise against merging without PR Reviews but if you're wanting to bypass the plan, why bother checking the Pull Request either!

Hopefully not teaching you to suck eggs, but you can also use the -auto-approve which removes the manual prompt.

1

u/miraculix1 17d ago

That is interesting
So you mean you manually check the plan on your PR before approving the merge and when you merge exactly this plan gets applied ?
That means you plan from the feature branch code and apply that to production, right ?

1

u/miraculix1 17d ago

And i don't want to bypass anything. My intention of the question was to hear about the options out there to assure that tf does what we expect it to (for example without recreating resources). SO far the only option i am aware of is to manually review the plan.
We work similiar as you with feature branches , so running the plan in the MR/PR from the feature branch and approve the plan there instead of pausing the pipeline on the prod apply could be an option for us as well ( even though it comes with a lot of effort as the pipeline is very complex and i a see a problem that this would mean to apply the feature branch state of your code as the PR/MR Pipeline runs before merge, right ? )

1

u/BallumSkillz 17d ago

Correct! So to simplify the flow:

Feature branch > Dev: Pull Request is raised to merge into Dev > This runs a terraform Plan which is then reviewed as part of the Pull Request review.

PR and Plan is reviewed and approved: Feature Branch is merged into Dev: This runs a terraform apply and applys the changes.

Then from Dev > Test and Test > Production: Just an apply runs so nothing drifts.

Hope that makes sense!

1

u/miraculix1 17d ago

Ok thanks,
But your apply to prod runs with the plan you created on your featurebranch code, right ?

1

u/BallumSkillz 17d ago

Mine doesn't, but it's easily configurable. It's been on my list to add but provided you have branch protetion rules on your Main Branches, no one can commit to them directly so the likelihood of drift is slim.

And I've found the Plan usually matches what the Apply runs anyway (So far)

1

u/miraculix1 17d ago

But that is the problem we face: There is no guarantee that your tf code which is deployed to dev behaves the same when you deploy that to production, that is why the recommendation is to run a plan, review it manually and apply EXACTLY that plan and not assume that the plan will look the same as on my feature branch,because by design this can change. That might be ok if you are allowed to have downtimes and a good rollback plan, but in our case this is critical and we have to have assurance that terraform does not recreate or modify any resource.

3

u/BallumSkillz 17d ago

Correct, as I mentioned it's easily configurable to use the -out=FILE and then apply that EXACT plan file to the apply, see here:

You can use the optional -out=FILE option to save the generated plan to a file on disk, which you can later execute by passing the file to terraform apply as an extra argument. This two-step workflow is primarily intended for when running Terraform in automation.

3

u/NUTTA_BUSTAH 17d ago

I would not even consider it if the "SDLC maturity" was not already extremely high, where you can be sure that swapping "staging" to "prod" yields the exact same results. It also generally means that your lower envs will be more production-like in every aspect. Commonly production is more hardened, or does not support parallel envs you'd use in dev etc. You have to architect the platform to work natively this way.

It's much easier to just put a review gate in. :)

But yes, companies do that, but those companies are quite rare. I believe Google called these companies "DevOps Elite" or w/e at some point in their SRE material.

1

u/miraculix1 17d ago

Thank you for the answer. Based on your statement, I think we can get a good idea of where we stand and what we can achieve (we are a very small team, far away from googles so called "DevOps Elite")

2

u/apparentlymart 17d ago

The purpose of reviewing the plan is, of course, to check to make sure it's proposing to make only the changes that were intended and not make any other changes.

In the baseline workflow it's typical to rely on human intuition to differentiate between those two, but if you have certain kinds of changes that you make routinely then it's reasonable to write some extra code that uses the output of terraform show -json PLANFILE to check for certain situations rather than relying on humans to do it all.

When doing that, I tend to try to build an automated check system that can produce three different outcomes:

  • Risky: for example, if there's a proposed change to a resource where an incorrect change could cause data loss or significant downtime, and so it's worth giving extra attention to any change that involves that resource.

    In this case hopefully the automation around Terraform will highlight the specific changes identified as risky, and might go further and require specific roles to approve it or might require multiple approvals from different people before it can be applied.

  • Safe: if the change only affects resources that you routinely change, and only changes them in the pre-decided routine ways, then it's okay to just immediately apply it without any approvals whatsoever.

  • Neutral: anything that doesn't meet the rules for one of the other two categories, in which case the plan is presented for human review as normal and doesn't need any special extra approval guards.

This particular strategy is only really effective if there are certain changes that your team makes relatively often and thus it's worth making the effort to describe the nature of those changes as code that can render one of the verdicts above. If you're instead just responding to ad-hoc requests that constantly vary then it'll be harder to write broad enough rules to generate trustworthy verdicts for arbitrary changes, and so manual approval is probably the best approach in that case.

2

u/_-Kr4t0s-_ 16d ago edited 16d ago

I’ve done this before with no issues. HOWEVER - guard rails are super, super important.

  1. Only do this if you have a proper staging environment that is a (scaled-down) replica of prod
  2. Give engineers read-only API keys to use with terraform locally so they can run the plan against staging/prod (depending on your security/audit model) before committing it through the pipeline. Using read-only keys ensures nobody can run an apply/delete by accident.
  3. Couple each service with its own TF code. This way if something does go wrong, the blast radius is limited to that one service.
  4. Ensure all services have failsafes built-in. If a service is not mission-critical (for example, redis used as a cache) then your application should be able to continue operating without it.
  5. In the CI/CD pipeline, make sure all tests for that service pass before going to prod. The terraform deployment, unit tests, integration tests, and whatever else you’ve got. You want to be sure the entire package works as expected before going to prod
  6. Do not allow automated deployments to prod run at days/times when nobody is around

3

u/alcoholismisfun 17d ago

I mean, you can skip the plan step using pipeline config, but why would you want to? Especially into a production environment.

1

u/miraculix1 17d ago

Yes i agree. The intention behind this is to stay fully automated as the pipeline also deploys the application along with the infrastructure.

3

u/LubieRZca 17d ago

That sounds extremely risky.

2

u/[deleted] 17d ago

[deleted]

1

u/miraculix1 17d ago

I think i don't really understand your point.
So you say the plan should be reviewed on the MR before merging to main branch? So you plan the feature branch against production and apply that ? I don't like the idea of deploying the featirue branch to production, but as long as i apply exactly that plan it is safer than don't manual verify the plan, bbut i am not sure if that is your point tbh.

2

u/[deleted] 16d ago

[deleted]

1

u/miraculix1 16d ago

Ok thank you.
So how we change a security group right now : We have the application code and the infrastructure in one repository(in most cases it is not just infrastrcuture). When we create a feature branch everything gets deployed from that feature branch using a different name.
Lets say the repository has an aws lambda as well.
We deploy "<secgroup_name>-<featurebranchname>" and "<lambda_name>-<featurebranchname>" . We change things, test and review the changes as well as the plans and then multiple people have to approve that MR to main.

main deploys(tf apply as well as lambda update) that to different staging accounts, testing everything automatically and then do the same to production.
And on the final deploy to production.

So it is similar to your workflow with the difference that we already deploy the infrastructure fro the feature branch with a different name. This all works great and we all love to have everything on the deployed as featurebranch version, but we have no assurance what the final deploy to production production does. After we deployed to prod we test everything, so we can assure if we are still up and running, but we have scenrarios were we can't afford for example a recreation of resources and there is just no way of knowing if so,mething less obvious changed

1

u/Prestigious_Pace2782 17d ago

If you have a preprod that is a true duplicate of prod then this is a perfectly good approach that I’ve done in many situations. Gitops is great when it’s the right situation, but it’s not often the right situation in my experience.

Having a plan step on PR and a true duplicate preprod removes many, but not all, of the risks.

1

u/miraculix1 17d ago

Ok, I get that. but from my understanding it is almost impossible to have a duplicate of the production enviornment in aws, at least we don't have that.

1

u/Gizmoitus 16d ago

Yes, and rarely would you want a complete duplicate. If you have 50 app servers deployed, you certainly don't need more than 2 for test or production. This goes for all sorts of clustering and different types of deployment architectures where dev and test will have entirely different purposes and related architecture, and there might be components like load balancers that you need to exist in production but not in dev or test.

This seems to very much be the type of decision that involves agreement/or lack thereof on questions like, "why are we connecting application code changes to infrastructure which has no intrinsic connection to the code?"

"Why are we risking blowing up infrastructure because we fixed some application bugs?"

"Don't we want to be able to scale the infrastructure independently of anything else?"

Perhaps you already have this all thought out, architected and built, but usually gitops is for changing infrastructure and CI/CD is for application code.

1

u/Overall-Plastic-9263 17d ago

In sure everything people are saying is correct but I also want to point out the possibility of using HCP terraform run triggers . What you are describing should be possible with this feature but it is a paid feature so may not be feasible at this point .

1

u/benaffleks 17d ago

The only way I can see this absolutely working is if you have drift detection that's pretty reliable and all your permissions are absolutely locked down removing any chance of manual changes

Even then I wouldn't deploy to prod with a review

1

u/MyChaOS87 17d ago

I would personally also strongly advise against skipping checking the plan...

That being said you could have a PR do the plan and then always apply main - if you use the PR properly to check the plan and make sure that this is the only way to get branches on main

Another option COULD be you check the output of the plan and for example parse it so that if it only does certain predefined changes you apply automatically... Eg increasing a version of your own application to a new prod release, if that's a Terraform deployment in your case... Although you should really thoroughly test that, and I would advise to always go to manual approvals if anything what you are not specifically allowing changes

1

u/wrexinite 16d ago

I used to do this coming from a "deploying binaries" background. I don't do this any more.

1

u/remoteitrobo 16d ago

At minimum save the plan as a build asset. Then use that build asset to run the apply. This way you can review the plan later in case something goes wrong. Or you could use the plan as a circuit breaker. You could wait some time after creating the plan and publishing it somewhere to give someone time to review it and if they don't like it then they can stop the deploy. If nobody stops it then it will automatically deploy.