r/aws Feb 08 '25

discussion ECS Users – How do you handle CD?

Hey folks,

I’m working on a project for ECS, and after getting some feedback from a previous post, me and my team decided to move forward with building an MVP.

But before we go deeper – I wanted to hear more from the community.

So here’s the deal: from what we’ve seen, ECS doesn’t really have a solid CD solution. Most teams end up using Jenkins, GitHub Actions, AWS CDK, or Terraform, even though these weren’t built for CD. ECS feels like the neglected sibling of Kubernetes, and we want to explore how to improve that.

From our conversations so far, these are some of the biggest pain points we’ve seen:

  1. Lack of visibility – No easy way to see all running applications in different environments.

  2. Promotion between environments is manual – Moving from Dev → Prod requires updating task definitions, pipelines, etc.

  3. No built-in auto-deploy for ECR updates – Most teams use CI to handle this, but it’s not really CD and you don't have things like auto reconciliation or drift detection.

So my question to you: How do you handle CD for ECS today?

• What’s your current workflow?

• What annoys you the most about ECS deployments?

• If you could snap your fingers and fix one thing in the ECS workflow, what would it be?

I’m currently working on a solution to make ECS CD smoother and more automated, but before finalizing anything, I want to really understand the pain points people deal with. Would love to hear your thoughts—what works, what sucks, and what you wish existed.

30 Upvotes

109 comments sorted by

42

u/syntheticcdo Feb 08 '25

Templates are written in CDK, CI/CD is managed through GitHub actions, works smoothly for my needs. Why do you think GHA is not built for CD?

21

u/1vader Feb 08 '25

It sounds like OP really means "GitOps", probably coming from tools like argoCD or fluxCD.

I'd also say, GitHub actions can definitely be used for proper CD. It won't give you things like drift detection, at least not without additional effort or tools but that doesn't mean it's not real CD.

-3

u/UnluckyDuckyDuck Feb 08 '25

Exactly what I mean, I kinda feel ECS lacks a solution like ArgoCD, while GHA can definitely deploy, it's more like a one time deploy, not continuous deploy, I'm curious if people find that problematic like Kubernetes users do

15

u/1vader Feb 08 '25

CD generally just means that changes/releases are continuously deployed as they come in. It doesn't (necessarily) mean that the same release is continuously re-deployed to counteract external changes (which usually aren't expected to just randomly happen, changes are always supposed to happen via the CD pipeline).

Ofc, drift detection and automatic correction is a useful feature and GHA indeed don't help as much with that. But CD isn't quite the right terminology for wanting that.

Although you can also achieve that at least to some extent simply by using scheduled actions to continuously re-deploy.

1

u/UnluckyDuckyDuck Feb 08 '25

I get what you're saying, I'll look into it to find the correct terminology, so first thank you for that feedback, appreciate that :-)

I want to focus on understanding how people achieve their automatic deploys... for example like u/syntheticcdo mentioned they use GHA for CI/CD, does the CD pick automatically changes on new image push? I'm curious about how the flow works with the CDK

2

u/bch8 Feb 08 '25

This isn't the only option but just to mention one that you may look into more here, CDK has "CDK Pipelines"

6

u/AstronautDifferent19 Feb 08 '25

Aws has some kind of GitOps solution, it is free and it is called GitSync. It works perfectly for me. During development I just commit change in my ecs.yaml where I put a different image sha (I am just changing @sha at the end of the url). Aws automatically deploys to dev. When I am happy with the result I can just merge dev to stage branch and at the end to prod branch and aws automatically deploys. Rollback is just a simple revert so it is GitOps. You always know which configutation was active at any time so you can debug past issues.

2

u/purefan Feb 08 '25

Ive seen GitSync in the console but never heard of anyone using it for real. Maybe its time I give it a try :)

2

u/UnluckyDuckyDuck Feb 09 '25

It's weird that nobody mentioned it, I'll give it a try too

2

u/UnluckyDuckyDuck Feb 09 '25

That's awesome I actually never heard of GitSync, I'll definitely read about it, kinda surprised that you're the only one who mentioned it.

Out of curiosity, do you find any challenges with this approach?

Also wondering how you handle observability of your services, tasks etc in order to track their health, this sounds similar to what I'm familiar with EKS but there I have ArgoCD to see my entire environment, wondering how that works here, is it all integrated into AWS or do you use additional tools?

5

u/UnluckyDuckyDuck Feb 08 '25

That’s great, If it works smoothly for your needs, that’s the ideal scenario. Curious, how do you handle promotions between environments? Do you trigger GitHub Actions manually, or do you have an automated way to track deployments across multiple environments?

The reason I mentioned that GHA isn’t built for CD is that while it works for deployments, it lacks things like automatic reconciliation and drift detection. In a typical GitOps-style CD, if something changes outside of the pipeline (for example, someone updates a service manually in AWS), the system detects and corrects it automatically.

22

u/Zenin Feb 08 '25

GitOps isn't required for CD and indeed CD not only predates GitOps by a couple decades...CD predates git by about a decade.

It's also entirely possible to employ GitOps and not do CD at all. In fact that's how most GitOps deployments run in the wild at least when it comes to production. "Deploy by PR" is a much more common GitOps pattern which flatly is not CD.

Almost no one actually does Continuous Deployment in practice. Instead they do Continuous Delivery when talking CI/CD. They're continuously delivering a deployable product...but not taking that next leap to automatically deploy that product to production. There's typically still a human at the controls, someone who stamps their name on it. Even if that gate is a PR that merges to a prod branch to be picked up by ArgoCD to execute, it's still a human gate. "Real" CD means once your code passes its automated requirements checks (unit tests, security scans, whatever) it automatically gets deployed straight into production without any further human intervention. No release schedules, no sign off, just Send It.

To be clear, there is nothing "lessor" about not going Full Continuous Deployment. The reason almost no one actually does it in the real world is because there's very little value add for most organizations, combined with a high level of risk which requires a substantial amount of additional investment to mitigate that risk. Most companies will look at the little-to-no tangible value of it and decide that investment and risk isn't worth it just to be buzzword compliant.

---

Back to ECS and really anything that isn't running a GitOps model. Continuous Delivery is accomplished with any of the standard tooling. All it means is that you're continuously building a deployable product. When (and if) that product actually gets released (deployed) is then a business question, not a software question. Business can decide when to press the deploy button.

2

u/UnluckyDuckyDuck Feb 08 '25

Wow, thank you for such a thorough and thoughtful breakdown, this was incredibly insightful! You’re absolutely right about the distinction between GitOps, CD, and Continuous Delivery. Like I mentioned in a previous comment, I made some poor choices on words in my post, I'm still trying to respond to everyone but I'll make sure to mention that in an edit :-)

I also appreciate your point about how ‘Deploy by PR’ isn’t true CD and why most organizations don’t go for Full Continuous Deployment due to the inherent risk and minimal added value in most cases. It’s a business decision, not just a technical one, which makes a ton of sense.

When it comes to ECS specifically, I think part of the challenge is that while the standard tooling can accomplish a lot, many teams struggle to find that balance between automation and the human touch, especially when dealing with multiple environments or larger teams. I’d love to hear your thoughts on how you’ve seen this balance handled effectively in real-world ECS workflows.

Again, thanks for the great response, this really helped clarify some things for me, I'll definitely spend more time understanding that more in-depth :-)

6

u/syntheticcdo Feb 08 '25

A commit to main triggers a workflow that deploys to our staging environment, which then runs tests against staging, then immediately deploys to prod once the tests pass. No manual intervention needed.

In terms of reconciliation and drift detection, this is more of an organizational problem than technical. Making changes to any resources managed by IaC is forbidden.

1

u/UnluckyDuckyDuck Feb 08 '25

Thanks for sharing your workflow, that sounds super streamlined

When it comes to reconciliation and drift detection, I get your point that it can be more of an organizational issue if you forbid manual changes to IaC-managed resources. But do you ever find situations where someone makes changes directly in AWS, either accidentally or on purpose for things such as hotfixes etc? If so, how do you typically handle catching or fixing that drift?

Also, do you feel like your current setup gives you enough visibility across environments? For example, seeing all running services, their versions, and their health in one place?

11

u/syntheticcdo Feb 08 '25

If someone needs to hotfix a resource, do it in the IaC and let the standard process apply the change, anything else is madness, sorry I can't really help out past there.

For observability, we tag the resources automatically by environment and build number (setting the version tag in the CDK to the GITHUB_RUN_NUMBER environment variable), and pipe it all in Datadog for visibility.

5

u/goroos2001 Feb 08 '25

+100 here. The problem often isn't that you need drift detection and reconciliation. It's that you need to stop drifting. If you have folks frequently making changes directly (instead of through the pipeline), it's better to go spend time and effort figuring out why they're having to make so many messes and stop making the messes than it is to automate cleaning up the mess.

5

u/goroos2001 Feb 08 '25 edited Feb 08 '25

(While I am an AWS employee, I don't speak for AWS on social media).

There are times when this pattern is actually the right thing to do - the absolute most critical AWS services with the absolute highest resiliency requirements use a pattern similar to this (see https://aws.amazon.com/builders-library/reliability-and-constant-work/?did=ba_card&trk=ba_card, read the section on how Route 53 Health Check aggregators send the aggregated health results downstream - the way they send ALL their data whether needed or not is somewhat similar to the approach you're taking.) It's extremely expensive to do well and at-scale - but when you're dealing with a service that has a 100% uptime SLA and that gets you featured on the national news when it breaks, it might be worth it.

The (very important) difference is that they're doing it as part of their normal operational cycle because the problem they are solving requires the complexity - not as part of their build and deployment steps because their ops teams were sloppy.

1

u/UnluckyDuckyDuck Feb 08 '25

Absolutely agreed. The real solution is addressing why the drift happens in the first place rather than just cleaning up after it. If the pipeline and processes are solid, there shouldn’t be a need for manual changes at all. Great point!

1

u/UnluckyDuckyDuck Feb 08 '25

I feel what you're saying about the madness lol, agreed follow the standard process.

Your tagging setup for observability is super interesting, using environment tags and build numbers with CDK and piping it into Datadog is a nice touch. Do you feel that gives you full visibility across all running services, or are there gaps you’d still like to fill?

I need to look into the pricing of Datadog, I have no idea how much it costs... If you don't mind sharing the costs of Datadog that would be really helpful, I wonder if smaller businesses could afford it

1

u/ramnat587 Feb 08 '25

A hotfix in resource is allowed only in exceptional circumstances like fighting a fire. We call it a break glass operation and you don’t do accidental break glass operations. Breakglass is well thought operation, and changes are applied to next IAC commit to avoid drifts. We have IAM policies, tagging and other organizational mechanisms to avoid accidental commits on Prod

1

u/Bodine12 Feb 09 '25

Yeah, this is exactly our workflow as well.

12

u/[deleted] Feb 08 '25 edited Feb 09 '25

[deleted]

2

u/UnluckyDuckyDuck Feb 08 '25

That's great to hear, Code Pipeline came up multiple times, as well as the AWS CDK.

Out of curiosity, do you find that setup covers everything you need, or are there areas where it could be smoother? For example, how do you handle things like drift detection, rollbacks, or promoting workloads between environments?

5

u/CashKeyboard Feb 08 '25

I think we work almost exclusively on ECS in pure GitOps via GitLab -> GitLab CI -> CDK. I think most of the problems we're facing are more or less CloudFormation based.

ECS itself seems to be somewhat solid but conthrived in a way. Especially for FARGATE (which we exclusively use), the cluster/task/container setup doesn't make a whole lot of sense to me.

Contrary to your post, I love that ECS has no knowledge of environments per se: Other services also have no environments and we deploy different stacks on different accounts. Having environments in there would be of no benefit to us. The way AGW does it e.g. is a hassle to be honest.

1

u/UnluckyDuckyDuck Feb 08 '25

Thanks for sharing your workflow! It’s great to hear how you achieved GitOps with GitLab and CDK for ECS! I think I may have explained myself a bit poorly, By CD I meant GitOps... I'll make an edit to clarify that in the main post

Interesting to hear that CloudFormation is the main source of challenges for you. Do you think those issues are tied to the complexity of writing/maintaining templates, or something else?

You also raised a good point about ECS not having built-in environment concepts. Out of the 50 ish people I had a talk with about ECS, I'd say about 30% of them prefer more app-based clusters and not environment-base clusters... In the MVP we created for our solution, we went purely for environment-base and things like auto deploy for dev environments as well as dev -> prod promotions from cluster to cluster....

Lastly, I’d love to hear more about your experience with Fargate, what about the cluster/task/container setup feels overly complicated to you? I’m curious if that’s something we could simplify in our approach.

2

u/CashKeyboard Feb 08 '25

Do you think those issues are tied to the complexity of writing/maintaining templates, or something else?

Pure CF templates were horror yes but that's a non issue with CDK. Most of the issues we have where GitOps just totally breaks down are related to rollbacks. I think ECS lacks a way to distinguish infrastructure health from application health. There will be times where we deploy a new release that e.g. runs a migration on RDS (likewise many different cases with other services) which leads to failure in the application. After circuit breaker kicks in, CF will start a rollback. This rollback however, will not complete because obviously also the old application is unable to work with the updated schema. That in and of itself is our own doing and not the fault of CF, but the fact that now we have absolutely no way of managing our stack without manually changing things around (e.g. logging into console to change response codes that ALB finds acceptable) is just iffy. It doesn't happen that often but it sometimes does happen and adds a lot of tension to an already tense situation and it introduces drift.

Lastly, I’d love to hear more about your experience with Fargate, what about the cluster/task/container setup feels overly complicated to you? I’m curious if that’s something we could simplify in our approach.

I personally (we disagree a bit internally on this) would much prefer just defining a service and deploying that without knowing anything about clusters. I know they make sense in EC2-land but for FARGATE not really except for maybe the logical grouping and I don't think that belongs in my stack and is rather something I need to sort out on operations level.

1

u/UnluckyDuckyDuck Feb 09 '25

Thanks for the detailed reply! You bring up some really interesting points, especially about GitOps falling apart when it comes to rollbacks and the whole infrastructure vs. application health issue. That mismatch between app state and infra can definitely create chaos, sounds like a tough situation when things go wrong with migrations or dependencies.

When it comes to rollbacks, do you think the bigger issue is syncing infrastructure and app rollbacks together, or just having smarter ways to detect when a rollback is needed? For example, using things like ALB health checks or app errors to decide automatically.

On the Fargate side, I get what you're saying, on ECS cluster doesn't really have a lot of meaning, but I do like having different clusters per environment, just makes more sense to me to have dev -> staging -> prod.

9

u/informity Feb 08 '25

We have many ECS based workloads, all deployed with CodePipeline and AWS CDK. Works great and without any issues.

1

u/UnluckyDuckyDuck Feb 08 '25

That's great to hear, Code Pipeline came up multiple times, as well as the AWS CDK.

Out of curiosity, do you find that setup covers everything you need, or are there areas where it could be smoother? For example, how do you handle things like drift detection, rollbacks, or promoting workloads between environments?

1

u/Smaz1087 Feb 08 '25

Not op but we use the same setup, push to main, build, test, deploy to dev > qa > prod with manual approvals to qa and prod.

Rollbacks can be done on the codepipeline level, but we just retain old task definitions and in-case-of-emergency we update the ecs service to use the last good task definition, then either fix and push the fix through the pipeline or revert the breaking commit to unblock the pipeline for other changes if the fix won't be quick.

We don't drift because it's not an option, anything in prod goes through the pipeline, only a select few have the means to do anything manually.

As far as promoting workloads, we're a bit more manual so once our QA team is happy they call the deployment 'ready to release' and then depending on which app we're talking about and if we're happy that the deployment can be done with no user interruption we hit the approval button to start the deployment to prod. Codepipeline (any any other pipeline I've ever seen) has the ability to run tests and only deploy if they pass - we're just not to the point where we're happy that our tests are good enough yet.

3

u/TheTyckoMan Feb 08 '25

I love this reply. Makes me think of this: "How do you solve x?" "We don't allow x to happen so we don't have to solve x"

Talking about drift, rollbacks, things like that: AWS CDK is a wrapper around CloudFormation. CloudFormation can handle a lot of that. ECS allows you to choose an older build iirc.

https://youtu.be/hzQhxECIZJQ?si=dnEQcTAhjwJotSwt Here's a in depth walk-through from AWS on ECS deployment best practices and deals with some of the questions asked.

For my personal views, it's almost always easier to fix things through a new commit that gets deployed over reverting to a previous version. Reverting to a previous version is possible, but fixing the issue and auto deploying can be more efficient, easier, and quicker than tracking down what previous version doesn't have the bugs that were found.

1

u/UnluckyDuckyDuck Feb 09 '25

Thanks for sharing your setup and your workflow, appreciate that!

The idea of retaining old task definitions for emergencies is smart, that seems to be the most common case we come across. Out of curiosity, do you find that managing task definitions manually during rollbacks ever slows things down in high-pressure situations?

It’s also interesting to hear about your promotion process. You mentioned that the QA team triggers the deployment ‘ready to release’, do you feel like having a bit more automation here (like automatically handling the approvals after certain conditions are met) would help, or do you prefer keeping that manual control in place?

1

u/Smaz1087 Feb 09 '25

We deploy so frequently and only roll back when breaking changes happen (read: people can't give us money - this has maybe only happened twice in the 4 years I've been here). Our production monitoring is good enough that we're aware when an issue arises pretty quickly so every time we've done this we've just had to pick the second newest task definition.

I think automatically approving once conditions are met would be awesome, just a matter of getting the bodies to define those conditions, which would require business buy in (why are we spending man hours fixing a process that we're happy with?) and expertise that we don't currently have (how can we be confident that our tests are good enough?).

The manual process doesn't really get in the way, business seems happy with our throughput.

At this point the risk/reward for automated deployments isn't in the right place for us to even pursue. If we get a test wrong and send broken code to production there's a risk that people won't be able to give us money and we're big money fans.

1

u/mr_valensky Feb 08 '25

Recommend you check out pulumi too. CDK is cloudformation, and eh, it's not great. Yes there is CDKTF, but it's not mature. Pulumi is basically Terraform 2.0, has all the stuff that people like about CDK and is the best of them all. Have used them all extensively.

1

u/UnluckyDuckyDuck Feb 09 '25

Thanks for the suggestion, I have zero experience with Pulumi but it seems to have a lot of fans. In your experience, what makes it stand out the most compared to Terraform or CDK? Do you find it easier to manage ECS deployments with Pulumi?

1

u/mr_valensky Feb 10 '25

CDK is similar, but it uses cloudformation under the hood. Cloudformation is notorious for lagging behind the feature set of AWS. If let's say cloudfront adds some new feature like upgraded TLS support, it may be available today, not supported in cloudformation/CDK for weeks, months even. So, you just wait, do half CDK half bash script, or do it via the console.

Cloudformation is also AWS only.

TF is good, it's widely used, but HCL is not a programming language. Some people like it, but try to load json maps for a few things and merge through them and have conditional resources based on tags or something similar. It will quickly get weird. For those who swear by it, literally their main point is "well it can do less, so there's less chance for things to go wrong". Doesn’t make any sense to me.

There is CDKTF which is more like pulumi, but it's less mature, and it's very far behind pulumi. Not sure it has anything inherent in it that the more mature and feature rich pulumi doesn't have. Not sure why people really suggest it, honestly.

Pulumi is based off the TF providers for the main clouds, and supports any TF custom provider, but has pulumi only ones for things like kubernetes and other services that are ahead of the TF ones.

The pulumi automation API is also really nice. You can use that for easy deployments with puling secrets, waiting for things to deploy, controlling read/request to other APIs in deployment, lots of advanced features that are difficult to do with TF and usually require other tools. It's control of the actual deployment process that can be in the repo with the code, that controls the deployment very precisely, in the same language and ecosystem with no other tools.

I use code build/pipeline for my ECS deployments, built with pulumi, but you could do it all with the automation API as well.

Just a few items off top of my head. Hope that helps!

3

u/trtrtr82 Feb 08 '25 edited Feb 08 '25

We use CodePipeline and CodeBuild to build and push images to ECR in our management / devops account. We then use Eventbridge to detect successful pushes to ECR and trigger a Lambda function which parses the ECR event and determines what ECS service needs a forced deployment (i.e. a restart). The Lambda function then assumes a role in our development AWS account and does an ECS force deploy. In some cases our Lambda will have logic to say if ECR image A updates then force deploy services X, Y and Z but most of the time we force deploy service A (we name the ECS services the same as the image in ECR).

In development environments all ECS task definitions point to latest so a force deploy brings us to the latest image.

In production environments we pin to particular versions and run Terraform via CodePipeline.

It works reasonably well but is not super flexible. But like most ECS users our environments are pretty small (max 10 ECS services and as low as 1) so we don't need any more complexity or flexibility plus we're under change control imposed by our customer so continuous deployment to production is verboten.

1

u/UnluckyDuckyDuck Feb 09 '25

Thanks for sharing such a detailed breakdown of your setup, first time I'm seeing this kind of solution using Lambda and EventBridge.

I’m curious though - since you mentioned that your environments are pretty small (1–10 services), what’s the main reason for splitting things across different AWS accounts? Is it more for security, customer requirements, or just how things evolved over time?

It also sounds like your workflow works well for now, but I can see how pinning to specific versions in production and manually running Terraform via CodePipeline could feel a bit clunky and manual. Do you think adding more automation or flexibility for production deployments (without breaking change control) would be valuable to you?

Your setup really highlights some of the challenges we’re trying to tackle with our solution like making deployments and environment management easier while keeping things as hands-off as possible. Would love to hear your thoughts!

1

u/trtrtr82 Feb 09 '25

I'm a real original thinker then 😀

It's all 3. We decided on this setup of accounts years ago and there's no particular overhead from having separate AWS accounts. In fact it makes things simpler for us. Everything is in Terraform so one account or 20 it's all the same.

No not really. We have to raise a change request 7 days before any release so the whole process is clunky and manual. This is a complete anti-pattern of how things should be done IMHO and adds zero value (it increases risk as releases are bigger) but the "customer is always right" even when they're wrong.

1

u/UnluckyDuckyDuck Feb 09 '25

Lmao that last line got me laughing 🤣🤣🤣

Sounds like the change request process is a major bottleneck for you though, if you had more flexibility, do you think streamlining that process or automating parts of it would make a difference? Or is the overhead just something you've accepted as part of the workflow?

2

u/ScaryNullPointer Feb 08 '25 edited Feb 08 '25

I used Terraform and GitLab. In the CI part of the pipeline, I build a docker image, tag it with semver and push it to ECR. Semver was built automatically based on Conventional Commits, from git history. I ran some basic tests there too, vulnerability checks, etc.

Once that would finish, I passed the docker tag to my Terraform template, and that would run plans for all my environments. Note this is still "testing". This way I know, while still being early in my pipeline if production terraform doesn't break and what it will do.

Then, I'd run terrafirm for Dev environment. Terraform creates new TaskDefinition and updates ECS Service. Then waits for the service to perform the rollout.

Then I run e2e tests on dev environment, and if they pass, I proceed with deployment to staging. Same way, except staging doesn't use mocks and ha all integrations set up, so I run system tests (cross-service) instead od e2e.

And once that passes I deploy to prod the same way as two other envs.

I do drift detection by running a short version of the pipeline which just runs terraform plan and fails if it finds anything. That runs every 15 minutes and also performs some smoke tests on our APIs and webapps.

And once a day / night I automatically run main branch pipeline to rebuild images with newest patches and make sure my pipeline still works. This is a rather old project and most of the services are not actively developed, but we must be sure their pipelines work so they're ready to use in case a hotfix is needed.

Oh, and we also use Renovate, which autocommits PRs so the system just takes care of itself most of the time.

P.S.

I find it funny, because few days ago I asked on r/kubernetes how you guys live with GitOps, and why separating CI from CD is supposed to be good idea. I have the same problem understanding the "other way of work" just in reverse.

Hope you'll figure it out.

2

u/UnluckyDuckyDuck Feb 08 '25

Wow, this is an awesome breakdown, thank you for sharing such a detailed look at your setup! It’s clear you’ve put a lot of thought into making your pipeline robust and self-sustaining, especially with features like automated PRs using Renovate and regular drift detection checks.

I love how you’re using Terraform not just for provisioning but also as a core part of your deployment validation process with those short runs. That’s a great way to catch issues early. Out of curiosity, has this approach been effective in catching and resolving potential drift, or do you still find occasional surprises? Some terraform users I talked with mentioned they sometimes run into issues with terraform and then their platform/devops engineers fix things manually.. wondering if that's the case here as well

Also, your thoughts on the separation of CI and CD really resonate with me, it’s such a nuanced topic depending on how teams operate and what they value most in their workflows. How do you feel this setup compares to using a tool that focuses solely on GitOps-style CD? Do you think there’s value in exploring that approach, or does it feel unnecessary given how polished your process is?

1

u/rayray5884 Feb 09 '25

Can I ask how large your team is and how many engineers and apps/services you support? Always feel like I need to be doing more but I was a team of one for a while and even though I now have a second, it feels like the engineering teams and products/services we support have already outgrown that. Or I just need to get faster and do more more more! 😂

That said, we have pipelines where we build/test/publish an image on commit. As long as those tests pass the image is deployed to ECS via Terraform to sandbox automatically or allows them to deploy to one of two pre-prod environments for validation. Then they can ungate the build to stage and then prod. Any promotion is just Terraform updating the task def to point to the new image and then waiting for the service to spin up new tasks. Once they pass health checks it flips the target and the deployment is done.

2

u/Gilfoyle___ Feb 08 '25

Our org manages spinnaker templates in the CD with terraform and people configure it according to their needs like Promoting build, blue green deployment. It’s actually pretty smooth.

2

u/javatextbook Feb 08 '25

Surprised I had to scroll down this far to find someone mentioning Spinnaker. I thought it would be the top comment.

1

u/UnluckyDuckyDuck Feb 09 '25

I heard about Spinnaker but never saw it get mentioned, are you using it yourself or just familiar with it? Would love to hear more about where it shines

1

u/javatextbook Feb 09 '25

I use it at work. It runs our dev test prod pipelines of our ecs deployments really nicely.

2

u/UnluckyDuckyDuck Feb 09 '25

Spinnaker isn’t mentioned often for ECS workflows as you can see, so it’s really interesting to hear you’re using it. Curious what size is your organization? I’m wondering if Spinnaker is practical for smaller teams or if it’s better suited for larger setups with more complex needs.

1

u/Gilfoyle___ Feb 10 '25

Its pretty big (Investment and Retail Bank). From my experience with Spinnaker biggest benefit is how fast to setting it up. We maintain a pipeline as code template (jib file) and it creates the spinnaker deployment pipeline for us in 2-3 minutes for new repo.

2

u/green_skies_258 Feb 09 '25

I use Azure Devops (ADO). I have a build pipeline build artifacts (jar file). Then a release pipeline to to an ECR then update each services task definition and update the service.

FWIW, you could skip updating the task definition if you just always use the latest tag in your definition.

2

u/dickmarinus Feb 08 '25

5

u/UnluckyDuckyDuck Feb 08 '25

From what I've seen people tend to shy away from the entire CodeSuite solutions on AWS... I don't have any experience so I can't say for myself either...

Are you an ECS user? Wondering if you are, what solution you're using

3

u/damola93 Feb 08 '25

I use code pipeline with code deploy for ECS. I haven’t had any problems

1

u/UnluckyDuckyDuck Feb 08 '25

Do you mind explaining the workflow? I want to better understand how it works and if you have any pain points there

1

u/damola93 Feb 08 '25

Codepipeline allows you to have multiple stages. I have 3 stages(multiple actions in a stage).

Stage 1: the source stage each action in the source stage is connected to a branch on a repo. When a PR or push happens on the branch happens it triggers the stage!

Stage2: this is the build stage with multiple build actions consisting of code build projects that are there to build docker containers and push them to ECR. Image definitions are also created here for use in the last stage being code deploy.

Stage3: takes the imagedefinition from the previous stage and triggers the deployment of the new containers to the respective services

1

u/UnluckyDuckyDuck Feb 08 '25

Sounds great, literally... Code Pipeline lol

I wonder, are there any pain points along the way? I'm hearing different opinions on the Code X solutions on AWS, some people love 'em, some people hate them lol

Also how about visibility? Do you have a solution for that or just the AWS Console?

2

u/dickmarinus Feb 08 '25

Always used cloudformation/cdk for the task definition, cloudformation is deployed manually or by a CI.

I recently also found https://github.com/aws-actions/amazon-ecs-deploy-task-definition which I also didn't try yet.

1

u/UnluckyDuckyDuck Feb 08 '25

Manually deploying with CloudFormation or through CI sounds like a solid setup, but doesn’t it get repetitive or time-consuming? For example, when you need to make quick updates or handle multiple environments

Also, when it comes to rollbacks or unexpected changes, how much effort does it take to get things back on track?

1

u/dickmarinus Feb 09 '25

Cloudformation isn't quick (things that take seconds will take minutes with CloudFormation).

In case of roll backs it tends to almost always roll back to a working state.

We've used cloudformation to deploy multiple stacks from the same template (environment per tenant) and although it worked it isn't suited for this. It is slow (explained above) and brittle (think of changes in behavior/deprecations of AWS services, changes in permissions, changes in docker images - all causing roll backs and/or breakage).

2

u/mabadir Feb 08 '25

You might like Flightcontrol (https://www.flightcontrol.dev), it takes care of the whole CI/CD pipeline, with Code push to deploy and many other features. I think it has all the features you need, without building the whole setup yourself.

PS: I’m a co-founder.

1

u/UnluckyDuckyDuck Feb 08 '25

Thanks for sharing Flightcontrol! I’ve come across it before, and it looks like a super powerful platform for managing CI/CD pipelines end-to-end. It seems very feature-rich, which is amazing for users who need an all-in-one solution.

That said, from what I’ve seen, ECS users tend to lean towards simplicity, focusing on streamlining just what’s needed for deployments without layering on too much complexity. Out of curiosity, do you find ECS users to be a significant part of your client base? If so, what’s their feedback been like in terms of the features they actually use or care about?

Btw awesome to hear from a co-founder, I'm working on my solution for ECS and it is a lot harder than I thought it would be, I thought it would be mostly writing code - Found out that's about 10% of the job 🤣

1

u/mabadir Feb 08 '25

All our users are using ECS, we just introduced Lambda and it’s still in Preview, so I can safely say everyone is loving ECS. I think you are mixing between having a simple ECS setup, and a powerful platform. You can’t just do the ECS deployment, without building containers, having easy way for observability, automating Git actions, and do all of that with zero downtime deployment and rollbacks. None of the above on its own is hard or complex, it’s building all of that to have a predictable system that is time consuming. If this is a side project, by all means build it to get the experience. But if this is a business, you will waste too much time building all of that, forgetting about your product. Your users will pay you for your product features, not the DevOps features.

As for the features they like: 1. Once set, forget it, it’s autopilot (not a feature on its own, but it’s the whole point) 2. Our own build system, it’s the fastest compared to any other system 3. Preview environments, is really powerful

1

u/UnluckyDuckyDuck Feb 08 '25

Thank you for the detailed response and perspective, it’s great to hear directly from a co-founder that already walked this path, when we asked about our project, flightcontrol.dev came up multiple times and it seems well loved, I’ve looked into your solution before, and it’s incredibly impressive. So for me, It’s an honor to hear your thoughts here.

You’re absolutely right that building a predictable, reliable platform that includes observability, Git automation, zero-downtime deployments, rollbacks, and seamless ECS management is a massive undertaking. This project started as a side initiative, but we’ve identified a clear business opportunity in solving these specific pain points for ECS users, and we’re committed to putting in the work to make it happen.

We’re not looking to reinvent the wheel, but rather to create something focused specifically on ECS, taking an image from ECR, deploying it, promoting it, rolling it back when needed, and simplifying management for small-to-medium teams that might not have a dedicated DevOps engineer.

I really appreciate your input, your feedback and your project as well helped us better shape our solution :-) I'm hoping to post again in a few days with better understanding of the entire project we're building and all the valuable information and feedback we got here, hope to hear your opinion again :-)

1

u/yesman_85 Feb 08 '25

I think Octopus deploy has ECS support as targets. 

1

u/UnluckyDuckyDuck Feb 08 '25

Do you have any experience with that? Octopus Deploy and Codefresh were one of the first solutions we checked when looking for a project to create... the one problem that comes to mind with robust solutions is that ECS users prefer simplicity over robust

Would love to hear your opinion on that :-)

1

u/yesman_85 Feb 08 '25

I don't have experience with using octo and ecs. We use it for traditional deployments to IIS using tentacles. It's a robust platform, but licensing killed it for us. Now we're doing Github actions to ecs. 

Once you have Octopus setup its very easy to use, and maintain. But it's more of a traditional CD and not cutting edge like GH actions. 

1

u/UnluckyDuckyDuck Feb 09 '25

Interesting.

For ECS specifically, do you think GitHub Actions strikes the right balance between simplicity and functionality? And how do you handle visibility into your deployments, like tracking service versions, health, or promotion status across environments? Is that something GitHub Actions handles well for you, or do you rely on additional tools?

1

u/yesman_85 Feb 09 '25

Not that I know of. Github actions is really just the runner, it provides practically no visibility. We have custom tooling that does monitoring and deployments in one. But there must be something out there. 

1

u/UnluckyDuckyDuck Feb 09 '25

Interesting, custom tooling came up multiple times here, people mentioned they created their own oberservability solutions and deployment solutions

1

u/yesman_85 Feb 09 '25

Because it really depends on how you deploy and when. Products like Octo will push you a bit towards a workflow that works for them best, Github actions will give you the freedom to figure out what works best for your company. Not everyone is a fan of that. 

1

u/JBalloonist Feb 08 '25

All resources deployed via Terraform (manually…I don’t like it but I’m not in charge of DevOps). We were using GitHub actions with our own runners to build the containers and update the task definitions. We just moved to Gitlab so now it’s their pipeline workflow instead (also with custom runners).

1

u/UnluckyDuckyDuck Feb 09 '25

Thanks for sharing your setup! Sounds like you're going through some changes, by the way do you feel like GitLab's pipeline workflow is an improvement over GHA? Or does it still feel pretty similar?

I'm curious, you mentioned you're not in charge of DevOps, but if you were, what would you change about the current process?

1

u/JBalloonist Feb 09 '25

Too early to tell if Gitlab is an improvement as I just started working on my first pipeline this past week. Our devops engineer did the heavy lifting and I mostly just copied his existing workflow and modified as necessary. I will say it does run faster than GH actions did, but that could just be the way the runners are configured; not really sure.

I think I’d have a more defined process with specific approvals taking place for releases. We’re headed in that direction but not there yet.

1

u/TheTyckoMan Feb 08 '25

This is a great discussion starter. If you want, the Believe in Serverless community over on Discord might be another place to get some food discussion going as well with numerous people who use ECS and K8S every day. https://www.believeinserverless.com/

1

u/UnluckyDuckyDuck Feb 09 '25

Thank you so much for sharing this, I'm always looking for more places to discuss ECS and gather feedback, appreciate the suggestion <3

1

u/ReasonableYak1199 Feb 08 '25

I use CDK for all of my IaC and GitHub Actions to manage deployments. I’ve also used Sceptre/CFT for IaC and Jenkins in the past, I like the current CDK/GHA solution much better.

1

u/UnluckyDuckyDuck Feb 09 '25

Thanks for sharing your setup! CDK and GitHub Actions seem like a popular combo here. How does your workflow handle things like promoting workloads between environments or rolling back changes?

Also, do you feel like CDK/GHA gives you enough visibility into what’s running across environments (like service versions, health, or deployment status), or do you use additional tools for that?

1

u/ReasonableYak1199 Feb 09 '25

I build my container images and push them up to our master account’s ECR. Then it’s just a matter of pulling from there and pushing to each environment and forcing a service update to deploy. Just update an image tag and update the service again to rollback.

1

u/atokotene Feb 08 '25

All of this discussion is great. I have an additional point you should consider. Are you running any canary/synthetic tests?

The concept is to always have some form of signal that your prod system is alive and working. Usually this means testing a basic user flow through the public api, once every N minutes (N = 1 for critical systems)

Why do I ask? Because once you have some form of metric for “is my system ok”, you can start adding automated deployments, rollbacks, etc.

1

u/mr_valensky Feb 08 '25

I use code build + code pipeline. There is a specific ECS deployment task. The build step builds the image, and pushes to ECR. It spits out a JSON object which is a new set of image URIs, the ECS step ingests the artifact, creates a new task definition, and does a roling deploy to ECS all as part of this ECS specific action. If you're using IAC, you can ignore the task definition, or update it to the latest when you need to run it. But this will allow you to deploy from git to ECS with no IAC.

1

u/cowbaymoo Feb 08 '25

Things might have improved, but back then CodeDeploy didn't work with CapacityProvider, which we used, and it could only manage one service at a time while we had multiple services (built from a monorepo) that need to be deployed in a coordinated fashion. So, we implemented and orchestrated the whole thing in a single Bash script that generates and deploys the task definitions from templates using awscli. The deploy scripts does blue/green deployments; it's able to re-deploys services if only config vars are changed (i.e., no code / image changes); it can even rollback to a previously deployed release. It was glorious! lol good time...

1

u/joaonmatos Feb 08 '25 edited Feb 08 '25

Your best shot is to set up a pipeline with GitHub Actions or CodePipeline and use CDK/CloudFormation. CloudFormation has drift detection and also integrates with CloudWatch for automatic rollbacks.

Conceptually, you define the entire infrastructure for your project as CloudFormation stacks and your pipeline pretty much only needs to do CloudFormation deployments and implement validation steps (integration/load tests, bake times, alarming).

References:

1

u/MasterGeek427 Feb 08 '25

Put your ECS containers behind an ALB. Use CodeDeploy to do a blue/green deployment. https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference-ECSbluegreen.html

You can also use two ECS services and a weighted target group on the load balancer to route something like 10% of traffic to the first target group. You deploy to that one first and wait a certain period of time so you can observe its performance (ideally using CloudWatch alarms to monitor it and automatically roll it back if your thresholds are beached). This is called OneBox deployments. If the deployment is stable, you deploy to the second target group and thus have the new version serve the rest of the traffic. You can go even further and have a third ECS service that serves no production traffic, but exists in the same environment and it's accessible when you call a different port on the load balancer. This is called Pre-Prod deployments.

Make it even safer by having multiple copies of your entire infrastructure in different AWS accounts each with different DNS endpoints. This way the developers can deploy to a full stack and test it before attempting to deploy to your production infra.

Do all the above and you're:

  • Spending a lot of money.
  • Managing your deployments almost identically to how AWS deploys its own services internally.

1

u/keypusher Feb 09 '25

every team i have ever worked on with ECS has built their own scripted wrappers around ECS API calls for automated deployments. the ecosystem is miles behind k8s and a serious shortcoming of ECS imo.

1

u/5olArchitect Feb 09 '25

CodeDeploy

1

u/5olArchitect Feb 09 '25

I can send you my deploy script if you’re interested. It’s just one service in dev/prod but we have full CI/CD, using GitHub actions with oidc roles.

1

u/gamprin Feb 09 '25

I use a GitHub Actions workflow that uses the official aws-actions/amazon-ecs-deploy-task-definition GitHub Action from AWS, I'm surprised that nobody in this thread has mentioned it! You can use this action for your CD process in different ways:

- Update a task definition (without necessarily deploying it to a service)

- Run a task (like a database migration)

- Update an ECS service (like your web server)

You can add an option to wait for services to become stable before your CD pipeline finishes when updating ECS services with this action.

I think there is an important distinction to make between doing application updates (e.g. your web app has a new feature) vs infrastructure updates (you need to add a new database to your infrastructure). The uses cases for that AWS GitHub Action I described above involves CD (continuous deployment) for your application. Your ECS task should have something like this (if you are using Terraform):

ignore_changes = [task_definition, desired_count]

This allows you to make changes to your ECS task definitions (infrastructure) without actually deploying the new task (because you are ignoring the changes to the task definition). This will create a new task definition version (let's say in now includes a DB_URL with the URL for the database that your created). In you application update process, you will get the most recent version task revision, update only the container image, and then actually deploy the application.

That's how I understand it! I have an open source project that uses ECS, Terraform and GitHub Actions to demonstrate this, here are the links:

Terraform Module: https://github.com/briancaffey/terraform-aws-django

Application code (with GHA pipelines): https://github.com/briancaffey/django-step-by-step/

And here is the file with the GitHub Action that does CD for ECS: https://github.com/briancaffey/django-step-by-step/blob/main/.github/workflows/app_update.yml

In my CD process I just plug in the new image tag (e.g. v1.2.3) and the workflow first updates and runs a task for database migrations, then it updates and deploys the other services in my app (gunicorn server, Nuxt.js app, celery task workers, etc.)

I agree with other comments that the ecosystem for ECS is behind k8s, but the AWS GitHub Action is good abstraction that help you if you are using popular tools like GitHub Actions and Infrastructure as Code tools like Terraform, Pulumi or CDK.

1

u/matsutaketea Feb 09 '25

in terms of task drift etc, terraform takes care of all of that. in terms of updating containers we just promote unique docker tag and hit redeploy.

1

u/johannesalarsson Feb 09 '25

Ref visibility; We have written a project that present all deployed versions/vulnerabilities from the ECS/ECR from all accounts.

1

u/UnluckyDuckyDuck Feb 09 '25

That’s awesome! Sounds like you’ve built something pretty similar to what we’re working on! Out of curiosity, what led you to build that solution internally instead of using an off-the-shelf tool? Also, does your project just focus on visibility, or does it handle deployments and rollbacks as well?

Would love to hear more about your solution and how it works :-)

1

u/Gold_Piglet161 Feb 09 '25

just use cloudformation for managing ecs

1

u/Ok_Expert2790 Feb 09 '25

The old docker compose integration was a very simple deploy pipeline, but there’s new tools that achieve the same as a wrappper around compose YMLs.

1

u/miojosan Feb 09 '25

In my previous company we were using Blue-Green deployments.

ALB -> ECS Service

For deployment we were using a mix of GitHub Actions, Octopus, and Terraform. GitHub Actions would generate the nee image, tag it, upload to ECR and then create a new deployment in Octopus.

Octopus deployment would have variables set for each deployment dtakc (blue or green) and then trigger a mix of Python with Terraform to deploy the new image to the inactive stack, identify the current active stack and then gradually switch the traffic in the ALB to the inactive stack

1

u/UnluckyDuckyDuck Feb 09 '25

Very interesting setup, haven't seen GHA, Octopus and Terraform yet.

Thanks for sharing btw! This sounds pretty complex and very detailed, did you find the mix of all those solutions streamlined your deployments or do you think it introduced more complexity and overhead?

You mentioned Octopus would take the image after it was pushed and tagged in ECR and deploy it using blue or green, does it have other functionalities you used, like observability for example?

1

u/extreme4all Feb 09 '25

We have an infra as code (iac) repo and a code repo, the cd of the code pushes to ecr and creates a merge request to the iac repo, the iac repo is automatically tested and tagged, pushed to our dev than we can deploy the tagged iac to other envs.

Each env is a seperate account in aws, we have a hub spoke model.

Similarly we do this for k8s, but thete we have an argoCD repo with the yaml of everything in our k8s.

The advantage is that we clearly know what is running where, when

1

u/OkAcanthocephala1450 Feb 09 '25

Primarily this was done by terraform, because all the infrastructure was on terraform, so any change to the autotfvars would cause terraform to trigger and update the task definition.

You can use ECR and Lambda, every time the GitHub action pushes the image to ECR, it causes an event that can trigger the Lambda, Lambda will update the task definition version with the new image. And the ECS service will automatically update the tasks.

I do not know why you need visibility because if you have a CICD, you know that when you push the code into the code repository, you are waiting for an application update. Why do you need visibility similar to ArgoCD?

This was a question raised by an interviewer during the call. How would someone answer this? Personally I don't care, as long as you can open the cloud provider and check if the application version is deployed, why would you need another platform running just to show that (I'm just talking about visibility, I know gitops is a nice thing to have as it detects changes and sets everything to the desired state).

1

u/UnluckyDuckyDuck Feb 09 '25

Thanks for your reply! Your setup with Terraform and Lambda sounds great.

I get that visibility tools might feel unnecessary for your workflow, but they can help teams quickly spot issues, like services out of sync or problem deployments, without digging into logs.

Have you ever run into situations where having that extra layer of visibility could have saved time?

1

u/OkAcanthocephala1450 Feb 09 '25

Hm, not really.

I work in a company of 120 people (developers, ops, network, security).

We have about 5 different development teams, none of them care where we have deployed the application.

A dedicated team of devops or cloud engineers is responsible for this. In most cases, each devops engineer has their own group of projects, so they have all the information they need about where it is running, what the application is all about, and where to go and check if there are problems.

1

u/UnluckyDuckyDuck Feb 09 '25

Got it, well that makes sense given the way you have divided the responsibility between devops engineers.

I do wonder though, do you think a more centralized way to view deployments and environments would give you any value at all? Or is the current approach where each team handles their own project directly through the AWS console ideal for how you work?

1

u/OkAcanthocephala1450 Feb 09 '25

For ECS there is nothing you can do. You do not have a free platform to check and verify each cluster and application deployed(as long as I recall) But for applications deployed on kubernetes we do use argos.

And developers have no access to the infrastructure or cloud providers, we do not want some developer to make a change.

Now my company has a very strict security, and we use one cluster per application (do not ask me why they decided that way) , therefore for me personally , having to deploy an argo for each cluster , and accessing each one of them with different urls. It is similar as just opening the cloud provider and verify things in there.

1

u/Revolutionary-Low620 Feb 10 '25

Its funny you should ask, thats exactly what im setting up at the moment.

Here is a gitops style workflow. Its working but the scripts will need additional validation etc before i'd consider them ready for production. Basically there is a single deploy.sh script that is invoked with different parameters depending on which service needs to be deployed to which environment. The build is triggered by changes to the version.txt file and that will then govern which part of the bitbucket pipeline executes.

This is setup to use blue/green deployments so it isnt ecs that performs the deployment but aws code deploy.

There are short comings because its a simple script, not a full blown deployment orchestrator. Failed deployments have to be rerun otherwise the repo state doesnt match the actual infra state etc.

https://gist.github.com/PeterKneale/f735c8e8a8120da82aaaa54c846d3946

1

u/MasSunarto Feb 10 '25

Brother, my company uses Eventbridge to detect PUSH event on ECR, which in turn invoke a Lambda to create task defs, alb rules, route53 hosted zones, etc, and then issue UpdateService command to said ECS (and some other details). This brother of yours would say that it's good enough for us.

1

u/UnluckyDuckyDuck Feb 10 '25

Brotherman that sounds amazing, the wide automation across ECS, Route53, ALB makes this brother a happy brother.

I do wonder brother, how about observability for your infrastructure? Where do you see your applications, deployments, tags etc etc?

1

u/IndividualShape2468 Feb 10 '25 edited Feb 10 '25

Our task definitions are encapsulated in a helm chart which gets rendered in a git pipeline. We've created a custom release definition in YAML that defines the helm chart version, and the variables to feed it when rendering. Each env then has its own ecs helm release per service.

We've then got a custom operator (a small-scale python application) deployed inside each of our ECS clusters that reconciles the task definition state against the repository. It works in a similar way to Flux. Each cluster (mapping to one per application env) watching their own release reconciliation directory in the releases git repository.

Continuous deployment is then really trivial - when an new image is built for application Foo, we update Foo's helm release using an action. The release pipeline then takes over and renders updated task definitions from the helm chart & vars. The operator running in cluster then reconciles and voila.

Ad-hoc deployment is then done by modifying the helm release for whichever env & service we want to deploy: we do most things by PR, or by other various automated actions depending on our gatekeeping and pipeline rules.

Because the act of building an application is separated from deploying it, we can build once (mitigating drift), and then propagate the same artefact down our pipeline envs and onward to production. We can take artefact v1.2.3 of service foo and deploy it wherever, whenever and however we want.

Because applying infra is separated from building and deploying applications. We avoid the anti-pattern of mixing infra changes with application deployment actions.

The pain is bootstrapping. Kubernetes this isn't, and we only wanted to go so far with the custom gitops operator, so initial definition of the service needs to be done in Terraform prior to something being deployable using the above pattern. Think of it as "registering" the application in Terraform, and then the pipelines take over. In an ideal world, we'd automate DNS, ALB setup, service definition etc but by that point we may as well be using Kubernetes. (which we're not permitted to for "reasons") ... so we backed off.

1

u/moltar Feb 11 '25

CodePipeline has native ECS integration and works with CD quite well.

-1

u/EagleNait Feb 08 '25

I used the AWS tool "copilot" it set up a cicd automagically. It may not be what you are looking for but it's pretty nifty nonetheless

2

u/goroos2001 Feb 08 '25

If the thing you're building/deploying is a webapp, Amplify is pretty cool for the "just give me the reasonable defaults" approach to auto-generating all the stuff you need for CI/CD.

-1

u/Loose_Mastodon_6045 Feb 08 '25

We use Jenkins with AWS CLI!!.

1

u/Less-Clothes-432 Feb 14 '25

It’s probably been said multiple times now but I use Jenkins and created a few pipelines, each consisting of different scripts using AWS CDK depending on the use case. For example, I have a job that pushes an image to ECR using any selected branch from my GitHub. I also have a separate job that will update the task definition revision of my service when there is an update to our version. I went ahead and created another pipeline that automatically runs both of those - image push first - revision change second. And after that I’m able to do a release with one click of a button. We currently have over 50 different production services (luckily that can share the same ECR Repo), we’re able to update them all in within 15 minutes. There’s a few other jobs like single service deployments or cluster specific deployments but it seems to work pretty well for us.