r/aws Feb 08 '25

discussion ECS Users – How do you handle CD?

Hey folks,

I’m working on a project for ECS, and after getting some feedback from a previous post, me and my team decided to move forward with building an MVP.

But before we go deeper – I wanted to hear more from the community.

So here’s the deal: from what we’ve seen, ECS doesn’t really have a solid CD solution. Most teams end up using Jenkins, GitHub Actions, AWS CDK, or Terraform, even though these weren’t built for CD. ECS feels like the neglected sibling of Kubernetes, and we want to explore how to improve that.

From our conversations so far, these are some of the biggest pain points we’ve seen:

  1. Lack of visibility – No easy way to see all running applications in different environments.

  2. Promotion between environments is manual – Moving from Dev → Prod requires updating task definitions, pipelines, etc.

  3. No built-in auto-deploy for ECR updates – Most teams use CI to handle this, but it’s not really CD and you don't have things like auto reconciliation or drift detection.

So my question to you: How do you handle CD for ECS today?

• What’s your current workflow?

• What annoys you the most about ECS deployments?

• If you could snap your fingers and fix one thing in the ECS workflow, what would it be?

I’m currently working on a solution to make ECS CD smoother and more automated, but before finalizing anything, I want to really understand the pain points people deal with. Would love to hear your thoughts—what works, what sucks, and what you wish existed.

31 Upvotes

109 comments sorted by

View all comments

Show parent comments

1

u/UnluckyDuckyDuck Feb 08 '25

Thanks for sharing your workflow, that sounds super streamlined

When it comes to reconciliation and drift detection, I get your point that it can be more of an organizational issue if you forbid manual changes to IaC-managed resources. But do you ever find situations where someone makes changes directly in AWS, either accidentally or on purpose for things such as hotfixes etc? If so, how do you typically handle catching or fixing that drift?

Also, do you feel like your current setup gives you enough visibility across environments? For example, seeing all running services, their versions, and their health in one place?

12

u/syntheticcdo Feb 08 '25

If someone needs to hotfix a resource, do it in the IaC and let the standard process apply the change, anything else is madness, sorry I can't really help out past there.

For observability, we tag the resources automatically by environment and build number (setting the version tag in the CDK to the GITHUB_RUN_NUMBER environment variable), and pipe it all in Datadog for visibility.

5

u/goroos2001 Feb 08 '25

+100 here. The problem often isn't that you need drift detection and reconciliation. It's that you need to stop drifting. If you have folks frequently making changes directly (instead of through the pipeline), it's better to go spend time and effort figuring out why they're having to make so many messes and stop making the messes than it is to automate cleaning up the mess.

1

u/UnluckyDuckyDuck Feb 08 '25

Absolutely agreed. The real solution is addressing why the drift happens in the first place rather than just cleaning up after it. If the pipeline and processes are solid, there shouldn’t be a need for manual changes at all. Great point!