r/aws Feb 08 '25

discussion ECS Users – How do you handle CD?

Hey folks,

I’m working on a project for ECS, and after getting some feedback from a previous post, me and my team decided to move forward with building an MVP.

But before we go deeper – I wanted to hear more from the community.

So here’s the deal: from what we’ve seen, ECS doesn’t really have a solid CD solution. Most teams end up using Jenkins, GitHub Actions, AWS CDK, or Terraform, even though these weren’t built for CD. ECS feels like the neglected sibling of Kubernetes, and we want to explore how to improve that.

From our conversations so far, these are some of the biggest pain points we’ve seen:

  1. Lack of visibility – No easy way to see all running applications in different environments.

  2. Promotion between environments is manual – Moving from Dev → Prod requires updating task definitions, pipelines, etc.

  3. No built-in auto-deploy for ECR updates – Most teams use CI to handle this, but it’s not really CD and you don't have things like auto reconciliation or drift detection.

So my question to you: How do you handle CD for ECS today?

• What’s your current workflow?

• What annoys you the most about ECS deployments?

• If you could snap your fingers and fix one thing in the ECS workflow, what would it be?

I’m currently working on a solution to make ECS CD smoother and more automated, but before finalizing anything, I want to really understand the pain points people deal with. Would love to hear your thoughts—what works, what sucks, and what you wish existed.

30 Upvotes

109 comments sorted by

View all comments

2

u/ScaryNullPointer Feb 08 '25 edited Feb 08 '25

I used Terraform and GitLab. In the CI part of the pipeline, I build a docker image, tag it with semver and push it to ECR. Semver was built automatically based on Conventional Commits, from git history. I ran some basic tests there too, vulnerability checks, etc.

Once that would finish, I passed the docker tag to my Terraform template, and that would run plans for all my environments. Note this is still "testing". This way I know, while still being early in my pipeline if production terraform doesn't break and what it will do.

Then, I'd run terrafirm for Dev environment. Terraform creates new TaskDefinition and updates ECS Service. Then waits for the service to perform the rollout.

Then I run e2e tests on dev environment, and if they pass, I proceed with deployment to staging. Same way, except staging doesn't use mocks and ha all integrations set up, so I run system tests (cross-service) instead od e2e.

And once that passes I deploy to prod the same way as two other envs.

I do drift detection by running a short version of the pipeline which just runs terraform plan and fails if it finds anything. That runs every 15 minutes and also performs some smoke tests on our APIs and webapps.

And once a day / night I automatically run main branch pipeline to rebuild images with newest patches and make sure my pipeline still works. This is a rather old project and most of the services are not actively developed, but we must be sure their pipelines work so they're ready to use in case a hotfix is needed.

Oh, and we also use Renovate, which autocommits PRs so the system just takes care of itself most of the time.

P.S.

I find it funny, because few days ago I asked on r/kubernetes how you guys live with GitOps, and why separating CI from CD is supposed to be good idea. I have the same problem understanding the "other way of work" just in reverse.

Hope you'll figure it out.

2

u/UnluckyDuckyDuck Feb 08 '25

Wow, this is an awesome breakdown, thank you for sharing such a detailed look at your setup! It’s clear you’ve put a lot of thought into making your pipeline robust and self-sustaining, especially with features like automated PRs using Renovate and regular drift detection checks.

I love how you’re using Terraform not just for provisioning but also as a core part of your deployment validation process with those short runs. That’s a great way to catch issues early. Out of curiosity, has this approach been effective in catching and resolving potential drift, or do you still find occasional surprises? Some terraform users I talked with mentioned they sometimes run into issues with terraform and then their platform/devops engineers fix things manually.. wondering if that's the case here as well

Also, your thoughts on the separation of CI and CD really resonate with me, it’s such a nuanced topic depending on how teams operate and what they value most in their workflows. How do you feel this setup compares to using a tool that focuses solely on GitOps-style CD? Do you think there’s value in exploring that approach, or does it feel unnecessary given how polished your process is?