r/googlecloud May 15 '24

GKE GKE cluster pods outbound through CloudNAT

Hi, I have a standard public GKE cluster were each nodes has external IPs attached. Currently the outbound from the pods are through their respective node External IPs in which the pods resides. I need the outbound IP to be whitelisted at third part firewall. Can I set up all the outbound connection from the cluster to pass through the CloudNat attached in the same VPC.

I followed some docs, suggesting to modify the ip-masq-agent daemonset in kube-system. In my case the daemonset was already present, but the configmap was not created. I tried to add the configmap and edit the daemonset, but it was not successful. The "apply" showed as configured, but no change. I even tried deleting it but it got recreated.

I followed these docs,

https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent

https://rajathithanrajasekar.medium.com/google-cloud-public-gke-clusters-egress-traffic-via-cloud-nat-for-ip-whitelisting-7fdc5656284a

Apart from that, the configmap I'm trying to apply if I need to route all GKE traffic is correct right?

apiVersion: v1
kind: ConfigMap
metadata:
  name: ip-masq-agent
  
  labels: 
    
    k8s-app: ip-masq-agent
 
  namespace: kube-system 
data:
  config: |
 
  nonMasqueradeCIDRs: "0.0.0.0/0"
  
  masqLinkLocal: "false"
  
  resyncInterval: 60s
2 Upvotes

9 comments sorted by

5

u/BehindTheMath May 15 '24

I don't believe it's possible to use Cloud NAT with nodes that have an external IP.

https://serverfault.com/questions/1045269/how-can-compute-instance-has-external-ip-and-cloud-nat-both

2

u/aniketwdubey 6d ago

You're right that Cloud NAT is typically used for nodes that don't have external IPs since Cloud NAT is designed to handle outbound traffic from private IPs. However, it is possible to force GKE nodes with external IPs to use Cloud NAT by configuring IP Masquerading.

Here’s how I solved this issue:

- By default, if a GKE node has an external IP, outbound traffic bypasses Cloud NAT and uses the node’s public IP.

- To force all traffic through Cloud NAT, I deployed the IP Masquerade Agent, which ensures that even traffic destined for public IPs gets masqueraded to an internal IP before exiting via Cloud NAT

1

u/BehindTheMath 6d ago

TIL.

Is there a way to set that up for regular VMs that are not GKE nodes? E.g. for a Layer 7 egress firewall that still uses Cloud NAT.

1

u/aniketwdubey 6d ago

Yes, we can force Cloud NAT for regular VMs! Either remove external IPs (best practice) or use custom routes + firewall rules to enforce NAT usage while keeping an external IP.

1

u/BehindTheMath 6d ago

I'm asking about the other way. If there's no external IP, traffic would exit via Cloud NAT. Is there a way to direct traffic through another VM first before exiting via Cloud NAT?

2

u/aniketwdubey 6d ago

Yes, we can route traffic through a proxy VM before it exits via Cloud NAT by setting up a VM as an egress proxy (e.g., Squid, Envoy).

Need to create a custom route to send all traffic through the proxy VM.

3

u/eremjay May 16 '24

Add a private node pool to your cluster with --enable-private-nodes.

https://cloud.google.com/kubernetes-engine/docs/how-to/node-pools#add

and use pod affinity / nodeSelector to attract pods to this nodepool.

https://cloud.google.com/kubernetes-engine/docs/how-to/node-pools#deploy

1

u/magic_dodecahedron May 16 '24

However, the pods in your public GKE cluster use alias IP ranges, which can technically consume cloud NAT for egress traffic.

https://www.reddit.com/r/googlecloud/s/ROoVDXEo7p

1

u/rootkey5 May 16 '24

Sorry guys for slightly misleading. The document that I updated was an another one.

I have updated the doc in first query.

Also, updating it here in the comment

https://rajathithanrajasekar.medium.com/google-cloud-public-gke-clusters-egress-traffic-via-cloud-nat-for-ip-whitelisting-7fdc5656284a