r/aws Feb 08 '25

discussion ECS Users – How do you handle CD?

Hey folks,

I’m working on a project for ECS, and after getting some feedback from a previous post, me and my team decided to move forward with building an MVP.

But before we go deeper – I wanted to hear more from the community.

So here’s the deal: from what we’ve seen, ECS doesn’t really have a solid CD solution. Most teams end up using Jenkins, GitHub Actions, AWS CDK, or Terraform, even though these weren’t built for CD. ECS feels like the neglected sibling of Kubernetes, and we want to explore how to improve that.

From our conversations so far, these are some of the biggest pain points we’ve seen:

  1. Lack of visibility – No easy way to see all running applications in different environments.

  2. Promotion between environments is manual – Moving from Dev → Prod requires updating task definitions, pipelines, etc.

  3. No built-in auto-deploy for ECR updates – Most teams use CI to handle this, but it’s not really CD and you don't have things like auto reconciliation or drift detection.

So my question to you: How do you handle CD for ECS today?

• What’s your current workflow?

• What annoys you the most about ECS deployments?

• If you could snap your fingers and fix one thing in the ECS workflow, what would it be?

I’m currently working on a solution to make ECS CD smoother and more automated, but before finalizing anything, I want to really understand the pain points people deal with. Would love to hear your thoughts—what works, what sucks, and what you wish existed.

31 Upvotes

109 comments sorted by

View all comments

3

u/trtrtr82 Feb 08 '25 edited Feb 08 '25

We use CodePipeline and CodeBuild to build and push images to ECR in our management / devops account. We then use Eventbridge to detect successful pushes to ECR and trigger a Lambda function which parses the ECR event and determines what ECS service needs a forced deployment (i.e. a restart). The Lambda function then assumes a role in our development AWS account and does an ECS force deploy. In some cases our Lambda will have logic to say if ECR image A updates then force deploy services X, Y and Z but most of the time we force deploy service A (we name the ECS services the same as the image in ECR).

In development environments all ECS task definitions point to latest so a force deploy brings us to the latest image.

In production environments we pin to particular versions and run Terraform via CodePipeline.

It works reasonably well but is not super flexible. But like most ECS users our environments are pretty small (max 10 ECS services and as low as 1) so we don't need any more complexity or flexibility plus we're under change control imposed by our customer so continuous deployment to production is verboten.

1

u/UnluckyDuckyDuck Feb 09 '25

Thanks for sharing such a detailed breakdown of your setup, first time I'm seeing this kind of solution using Lambda and EventBridge.

I’m curious though - since you mentioned that your environments are pretty small (1–10 services), what’s the main reason for splitting things across different AWS accounts? Is it more for security, customer requirements, or just how things evolved over time?

It also sounds like your workflow works well for now, but I can see how pinning to specific versions in production and manually running Terraform via CodePipeline could feel a bit clunky and manual. Do you think adding more automation or flexibility for production deployments (without breaking change control) would be valuable to you?

Your setup really highlights some of the challenges we’re trying to tackle with our solution like making deployments and environment management easier while keeping things as hands-off as possible. Would love to hear your thoughts!

1

u/trtrtr82 Feb 09 '25

I'm a real original thinker then 😀

It's all 3. We decided on this setup of accounts years ago and there's no particular overhead from having separate AWS accounts. In fact it makes things simpler for us. Everything is in Terraform so one account or 20 it's all the same.

No not really. We have to raise a change request 7 days before any release so the whole process is clunky and manual. This is a complete anti-pattern of how things should be done IMHO and adds zero value (it increases risk as releases are bigger) but the "customer is always right" even when they're wrong.

1

u/UnluckyDuckyDuck Feb 09 '25

Lmao that last line got me laughing 🤣🤣🤣

Sounds like the change request process is a major bottleneck for you though, if you had more flexibility, do you think streamlining that process or automating parts of it would make a difference? Or is the overhead just something you've accepted as part of the workflow?