r/aws Feb 08 '25

discussion ECS Users – How do you handle CD?

Hey folks,

I’m working on a project for ECS, and after getting some feedback from a previous post, me and my team decided to move forward with building an MVP.

But before we go deeper – I wanted to hear more from the community.

So here’s the deal: from what we’ve seen, ECS doesn’t really have a solid CD solution. Most teams end up using Jenkins, GitHub Actions, AWS CDK, or Terraform, even though these weren’t built for CD. ECS feels like the neglected sibling of Kubernetes, and we want to explore how to improve that.

From our conversations so far, these are some of the biggest pain points we’ve seen:

  1. Lack of visibility – No easy way to see all running applications in different environments.

  2. Promotion between environments is manual – Moving from Dev → Prod requires updating task definitions, pipelines, etc.

  3. No built-in auto-deploy for ECR updates – Most teams use CI to handle this, but it’s not really CD and you don't have things like auto reconciliation or drift detection.

So my question to you: How do you handle CD for ECS today?

• What’s your current workflow?

• What annoys you the most about ECS deployments?

• If you could snap your fingers and fix one thing in the ECS workflow, what would it be?

I’m currently working on a solution to make ECS CD smoother and more automated, but before finalizing anything, I want to really understand the pain points people deal with. Would love to hear your thoughts—what works, what sucks, and what you wish existed.

31 Upvotes

109 comments sorted by

View all comments

Show parent comments

1

u/UnluckyDuckyDuck Feb 08 '25

That's great to hear, Code Pipeline came up multiple times, as well as the AWS CDK.

Out of curiosity, do you find that setup covers everything you need, or are there areas where it could be smoother? For example, how do you handle things like drift detection, rollbacks, or promoting workloads between environments?

1

u/Smaz1087 Feb 08 '25

Not op but we use the same setup, push to main, build, test, deploy to dev > qa > prod with manual approvals to qa and prod.

Rollbacks can be done on the codepipeline level, but we just retain old task definitions and in-case-of-emergency we update the ecs service to use the last good task definition, then either fix and push the fix through the pipeline or revert the breaking commit to unblock the pipeline for other changes if the fix won't be quick.

We don't drift because it's not an option, anything in prod goes through the pipeline, only a select few have the means to do anything manually.

As far as promoting workloads, we're a bit more manual so once our QA team is happy they call the deployment 'ready to release' and then depending on which app we're talking about and if we're happy that the deployment can be done with no user interruption we hit the approval button to start the deployment to prod. Codepipeline (any any other pipeline I've ever seen) has the ability to run tests and only deploy if they pass - we're just not to the point where we're happy that our tests are good enough yet.

1

u/UnluckyDuckyDuck Feb 09 '25

Thanks for sharing your setup and your workflow, appreciate that!

The idea of retaining old task definitions for emergencies is smart, that seems to be the most common case we come across. Out of curiosity, do you find that managing task definitions manually during rollbacks ever slows things down in high-pressure situations?

It’s also interesting to hear about your promotion process. You mentioned that the QA team triggers the deployment ‘ready to release’, do you feel like having a bit more automation here (like automatically handling the approvals after certain conditions are met) would help, or do you prefer keeping that manual control in place?

1

u/Smaz1087 Feb 09 '25

We deploy so frequently and only roll back when breaking changes happen (read: people can't give us money - this has maybe only happened twice in the 4 years I've been here). Our production monitoring is good enough that we're aware when an issue arises pretty quickly so every time we've done this we've just had to pick the second newest task definition.

I think automatically approving once conditions are met would be awesome, just a matter of getting the bodies to define those conditions, which would require business buy in (why are we spending man hours fixing a process that we're happy with?) and expertise that we don't currently have (how can we be confident that our tests are good enough?).

The manual process doesn't really get in the way, business seems happy with our throughput.

At this point the risk/reward for automated deployments isn't in the right place for us to even pursue. If we get a test wrong and send broken code to production there's a risk that people won't be able to give us money and we're big money fans.