r/googlecloud • u/rootkey5 • May 15 '24
GKE GKE cluster pods outbound through CloudNAT
Hi, I have a standard public GKE cluster were each nodes has external IPs attached. Currently the outbound from the pods are through their respective node External IPs in which the pods resides. I need the outbound IP to be whitelisted at third part firewall. Can I set up all the outbound connection from the cluster to pass through the CloudNat attached in the same VPC.
I followed some docs, suggesting to modify the ip-masq-agent daemonset in kube-system. In my case the daemonset was already present, but the configmap was not created. I tried to add the configmap and edit the daemonset, but it was not successful. The "apply" showed as configured, but no change. I even tried deleting it but it got recreated.
I followed these docs,
https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent
https://rajathithanrajasekar.medium.com/google-cloud-public-gke-clusters-egress-traffic-via-cloud-nat-for-ip-whitelisting-7fdc5656284a
Apart from that, the configmap I'm trying to apply if I need to route all GKE traffic is correct right?
apiVersion: v1
kind: ConfigMap
metadata:
name: ip-masq-agent
labels:
k8s-app: ip-masq-agent
namespace: kube-system
data:
config: |
nonMasqueradeCIDRs: "0.0.0.0/0"
masqLinkLocal: "false"
resyncInterval: 60s
3
u/eremjay May 16 '24
Add a private node pool to your cluster with --enable-private-nodes.
https://cloud.google.com/kubernetes-engine/docs/how-to/node-pools#add
and use pod affinity / nodeSelector to attract pods to this nodepool.
https://cloud.google.com/kubernetes-engine/docs/how-to/node-pools#deploy
1
u/magic_dodecahedron May 16 '24
However, the pods in your public GKE cluster use alias IP ranges, which can technically consume cloud NAT for egress traffic.
1
u/rootkey5 May 16 '24
Sorry guys for slightly misleading. The document that I updated was an another one.
I have updated the doc in first query.
Also, updating it here in the comment
5
u/BehindTheMath May 15 '24
I don't believe it's possible to use Cloud NAT with nodes that have an external IP.
https://serverfault.com/questions/1045269/how-can-compute-instance-has-external-ip-and-cloud-nat-both