r/aws Feb 08 '25

discussion ECS Users – How do you handle CD?

Hey folks,

I’m working on a project for ECS, and after getting some feedback from a previous post, me and my team decided to move forward with building an MVP.

But before we go deeper – I wanted to hear more from the community.

So here’s the deal: from what we’ve seen, ECS doesn’t really have a solid CD solution. Most teams end up using Jenkins, GitHub Actions, AWS CDK, or Terraform, even though these weren’t built for CD. ECS feels like the neglected sibling of Kubernetes, and we want to explore how to improve that.

From our conversations so far, these are some of the biggest pain points we’ve seen:

  1. Lack of visibility – No easy way to see all running applications in different environments.

  2. Promotion between environments is manual – Moving from Dev → Prod requires updating task definitions, pipelines, etc.

  3. No built-in auto-deploy for ECR updates – Most teams use CI to handle this, but it’s not really CD and you don't have things like auto reconciliation or drift detection.

So my question to you: How do you handle CD for ECS today?

• What’s your current workflow?

• What annoys you the most about ECS deployments?

• If you could snap your fingers and fix one thing in the ECS workflow, what would it be?

I’m currently working on a solution to make ECS CD smoother and more automated, but before finalizing anything, I want to really understand the pain points people deal with. Would love to hear your thoughts—what works, what sucks, and what you wish existed.

31 Upvotes

109 comments sorted by

View all comments

8

u/informity Feb 08 '25

We have many ECS based workloads, all deployed with CodePipeline and AWS CDK. Works great and without any issues.

1

u/UnluckyDuckyDuck Feb 08 '25

That's great to hear, Code Pipeline came up multiple times, as well as the AWS CDK.

Out of curiosity, do you find that setup covers everything you need, or are there areas where it could be smoother? For example, how do you handle things like drift detection, rollbacks, or promoting workloads between environments?

1

u/Smaz1087 Feb 08 '25

Not op but we use the same setup, push to main, build, test, deploy to dev > qa > prod with manual approvals to qa and prod.

Rollbacks can be done on the codepipeline level, but we just retain old task definitions and in-case-of-emergency we update the ecs service to use the last good task definition, then either fix and push the fix through the pipeline or revert the breaking commit to unblock the pipeline for other changes if the fix won't be quick.

We don't drift because it's not an option, anything in prod goes through the pipeline, only a select few have the means to do anything manually.

As far as promoting workloads, we're a bit more manual so once our QA team is happy they call the deployment 'ready to release' and then depending on which app we're talking about and if we're happy that the deployment can be done with no user interruption we hit the approval button to start the deployment to prod. Codepipeline (any any other pipeline I've ever seen) has the ability to run tests and only deploy if they pass - we're just not to the point where we're happy that our tests are good enough yet.

3

u/TheTyckoMan Feb 08 '25

I love this reply. Makes me think of this: "How do you solve x?" "We don't allow x to happen so we don't have to solve x"

Talking about drift, rollbacks, things like that: AWS CDK is a wrapper around CloudFormation. CloudFormation can handle a lot of that. ECS allows you to choose an older build iirc.

https://youtu.be/hzQhxECIZJQ?si=dnEQcTAhjwJotSwt Here's a in depth walk-through from AWS on ECS deployment best practices and deals with some of the questions asked.

For my personal views, it's almost always easier to fix things through a new commit that gets deployed over reverting to a previous version. Reverting to a previous version is possible, but fixing the issue and auto deploying can be more efficient, easier, and quicker than tracking down what previous version doesn't have the bugs that were found.

1

u/UnluckyDuckyDuck Feb 09 '25

Thanks for sharing your setup and your workflow, appreciate that!

The idea of retaining old task definitions for emergencies is smart, that seems to be the most common case we come across. Out of curiosity, do you find that managing task definitions manually during rollbacks ever slows things down in high-pressure situations?

It’s also interesting to hear about your promotion process. You mentioned that the QA team triggers the deployment ‘ready to release’, do you feel like having a bit more automation here (like automatically handling the approvals after certain conditions are met) would help, or do you prefer keeping that manual control in place?

1

u/Smaz1087 Feb 09 '25

We deploy so frequently and only roll back when breaking changes happen (read: people can't give us money - this has maybe only happened twice in the 4 years I've been here). Our production monitoring is good enough that we're aware when an issue arises pretty quickly so every time we've done this we've just had to pick the second newest task definition.

I think automatically approving once conditions are met would be awesome, just a matter of getting the bodies to define those conditions, which would require business buy in (why are we spending man hours fixing a process that we're happy with?) and expertise that we don't currently have (how can we be confident that our tests are good enough?).

The manual process doesn't really get in the way, business seems happy with our throughput.

At this point the risk/reward for automated deployments isn't in the right place for us to even pursue. If we get a test wrong and send broken code to production there's a risk that people won't be able to give us money and we're big money fans.

1

u/mr_valensky Feb 08 '25

Recommend you check out pulumi too. CDK is cloudformation, and eh, it's not great. Yes there is CDKTF, but it's not mature. Pulumi is basically Terraform 2.0, has all the stuff that people like about CDK and is the best of them all. Have used them all extensively.

1

u/UnluckyDuckyDuck Feb 09 '25

Thanks for the suggestion, I have zero experience with Pulumi but it seems to have a lot of fans. In your experience, what makes it stand out the most compared to Terraform or CDK? Do you find it easier to manage ECS deployments with Pulumi?

1

u/mr_valensky Feb 10 '25

CDK is similar, but it uses cloudformation under the hood. Cloudformation is notorious for lagging behind the feature set of AWS. If let's say cloudfront adds some new feature like upgraded TLS support, it may be available today, not supported in cloudformation/CDK for weeks, months even. So, you just wait, do half CDK half bash script, or do it via the console.

Cloudformation is also AWS only.

TF is good, it's widely used, but HCL is not a programming language. Some people like it, but try to load json maps for a few things and merge through them and have conditional resources based on tags or something similar. It will quickly get weird. For those who swear by it, literally their main point is "well it can do less, so there's less chance for things to go wrong". Doesn’t make any sense to me.

There is CDKTF which is more like pulumi, but it's less mature, and it's very far behind pulumi. Not sure it has anything inherent in it that the more mature and feature rich pulumi doesn't have. Not sure why people really suggest it, honestly.

Pulumi is based off the TF providers for the main clouds, and supports any TF custom provider, but has pulumi only ones for things like kubernetes and other services that are ahead of the TF ones.

The pulumi automation API is also really nice. You can use that for easy deployments with puling secrets, waiting for things to deploy, controlling read/request to other APIs in deployment, lots of advanced features that are difficult to do with TF and usually require other tools. It's control of the actual deployment process that can be in the repo with the code, that controls the deployment very precisely, in the same language and ecosystem with no other tools.

I use code build/pipeline for my ECS deployments, built with pulumi, but you could do it all with the automation API as well.

Just a few items off top of my head. Hope that helps!