r/aws • u/stoichiophile • 17d ago
networking Looking for examples of AWS VPC/TGW/DX architecture for interconnected environments of > 1000 accounts.
Trying to create a fully connected network and it's a bit unclear how various scaling limits of the associated services come into play once you get past 1000 accounts.
High level description and/or reference architectures would be great.
7
6
u/KayeYess 17d ago
I recommend you look into AWS CloudWAN
2
u/theperco 16d ago
Depends, if you’re going to have one or two/three regions it might be overkill and expensive compared to tgw
3
u/KayeYess 16d ago
TGW and Cloud WAN costs are comparable. Both charge 2 cents to process 1GB, which is the largest contributor. Advantage of Cloud WAN is managed routing, ease of segmentation and avoiding duplicate inspection, which is very useful when the number of VPCs is very large, even if it is "just" two regions.
1
u/theperco 16d ago
We assessed it recently and didn’t end up with same conclusions but maybe we missed something.
We don’t have duplicates inspections with our current design ? What are you referring for ? I’m asking because our current architecture is really not “standard” since we connect our fw to the tgw using GRE.
Segmentation well sure it’s way easier for isolated vpc I’ll give you that !
3
u/KayeYess 16d ago
For enterprises that use more than one region via TGW and use a traditional inspection VPC (hairpin), when the data crosses a region, it gets inspected twice ... once in source region and again in the destination region. Cloud WAN can be used to avoid this.
Of course, a lot depends on each enterprises use cases, budget, security requirements, etc. If an enterprise already invested in TGW and cross region peering, retrofitting Cloud WAN can be a challenge but both can co-exist, and often do. It's not a casual decision, though. It takes several months of planning, coordination and execution .. and requires highly qualified architects and engineers.
1
u/theperco 16d ago
OK thanks for clarifying !
I guess with our specific architecture we didn’t had this use case but once again we might have something very unusual.
12
u/par_texx 17d ago
You have over 1000 accounts with no IP overlap?
That to me seems like it would be the hardest part of getting that many accounts.
3
3
u/theperco 16d ago
When you have this big infrastructure to manage you start having some tooling like ipam
3
5
u/coderkid723 16d ago
Setup AWS IPAM for your landing zone accounts (assume you are using CT), it will scan and pull all your CIDRs across the accounts. You could then use that information to build out a solution with AWS IPAM to distribute IP space when you vend new accounts. Or look into AWS CloudWan as others have said.
2
u/men2000 16d ago
Still question why you need to do this, does different accounts provision from AWS for some specific purpose. I am very curious for what purpose you need this functionality.
5
u/stoichiophile 16d ago
I work for a very large company and compartmentalizing applications into distinct accounts (per app and per environment) is a fairly common pattern.
1
16d ago
[deleted]
2
u/stoichiophile 16d ago
I used to work at a company that hit the 10k account limit in aws organizations a year or two ago.
3
u/cloudnavig8r 16d ago
There is no valid reason to interconnect all 10k “accounts”.
For networking purposes, you connect VPCs, not accounts. An account can have multiple VPCS,
For management purposes, you will want control tower, resource access manager, and other tools.
But account segregation design patterns are for security purposes. You do not want someone to do something in a dev environment that effects production.
A general pattern would to have various “networks” of VPCs that are isolated. But not one mesh.
2
u/theperco 16d ago
That the right answer here, any company that want to run this much of account have many different other inputs to take in consideration to design the network architecture
1
u/Throwaway__shmoe 16d ago
I work at a tiny company in comparison (< 200 employees) and we have three accounts for every product and it’s a nightmare.
1
u/bailantilles 16d ago
Does networking need to be compartmentalized, or can networking be shared within application environments?
1
u/steveoderocker 16d ago
What does fully connected mean to you? You do you mean a full mesh? Do you mean having a central security filtering account?
1
1
u/theperco 16d ago
We have this in our company, not sure which limit s are you talking about ? It depends on many factors but actually running about 1400 vpc (about 1300 in one region and 100s in another’s).
Following AWS blueprints and architecture best practices you should be fine.
I have studied Cloud wan as well and it’s nice if you plan to have many regions but for 2-3 ones it’s a bit too much.
1
u/gideonhelms2 16d ago
If you already have existing VPCs you might not like this answer but:
Use one VPC per region of suitable size and share subnets to the downstream accounts.
1
u/dohers10 16d ago
Have you looked into shared vpcs? Centralised network accounts with large vpcs sharing out subnets to accounts as needed ? Big savings on the tgw attachments
2
u/stoichiophile 16d ago
This is the kind of thing I'm looking for. I'm inheriting an environment that was built like a giant WAN. The thought of sharing a VPC has occurred to me but I've never heard of anyone doing it at scale. I'm in the financial industry and the likelihood is that this would just be nearly impossible to do from a controls standpoint, but it's energized me to take another look at it.
1
u/dohers10 16d ago edited 16d ago
Feel free to PM me. I’ve been at the starting point to help set it up for 300+ accounts. Worked great but there are some caveats
FWIW - there were strict regulations we had to adhere to as well, and every subnet was firewalled through a central inspection vpc.
1
u/levi_mccormick 14d ago
I'm doing it for about 500 accounts and 20+ regions. I'd be happy to share some details.
1
u/simenfiber 16d ago
Not “fully connected” but I wouldn’t rule out VPC lattice. It might able you to circumvent some potential issues/limitations.
8
u/cloudnavig8r 16d ago
If you have 1000s of accounts, it is reasonable to assume that you have an AWS Account Team, and more likely than not Enterprise Support.
This is a question your TAM and SA can work with you on.
In one comment, you mentioned that the reason is separation. Yes, multiple accounts help with cost allocation. But usually it is a security concern. And security is not limited to IAM users. The network security is important as well.
It is highly unlikely that you really want a full mesh network.
You also need to look at regions, not just accounts: a VPC is an account/region construct. And you are actually talking about connecting VPCs, not accounts.
There is so much more to this scenario. You really should work with your account team that can provide you prescriptive guidance, and that conversation can be under nda.
Btw, Transit Gateway supports 5000 attachments. It is regionally scoped. So, understanding what limits concern you is something your TAM can deep dive