Networking in AWS and GCP, with a Stress on Secondary IPs in AWS EKS and GCP GKE: A Deep Dive

Kubernetes networking is a fascinating topic that underpins the scalability and connectivity of modern cloud-native applications. A crucial aspect of this is how IP addresses are assigned to pods. Both AWS Elastic Kubernetes Service (EKS) and Google Kubernetes Engine (GKE) have their own unique ways of handling this. In this post, we’ll break it down in a conversational and easy-to-follow manner, focusing on how these platforms assign secondary IPs to pods.

Setting the Stage: Pod Networking in Kubernetes

Imagine Kubernetes as a big digital city, and each pod is like a house. Just like houses need unique addresses for mail delivery, each pod in Kubernetes needs its own IP address to communicate. Both AWS and GCP handle this, but their methods are tailored to their respective ecosystems.

How AWS EKS Handles Secondary IPs

AWS uses something called the Amazon VPC CNI Plugin to manage pod networking. Let’s break it down step by step:

1. Elastic Network Interfaces (ENIs): The Backbone

Think of ENIs as virtual network cards attached to your EKS worker nodes. These ENIs have:

• A primary private IP (used by the node itself).
• A pool of secondary private IPs, which are handed out to pods as needed.

2. Where Do the IPs Come From?

Pods get their IPs directly from the VPC subnet where the worker node resides. So, pods in AWS are like tenants sharing the same apartment complex (subnet) as other AWS resources.

3. Scaling with ENIs

Here’s the catch: The number of IPs available for pods depends on the instance type of your worker node. For example, an m5.large instance supports 3 ENIs, each with 10 secondary IPs. That’s 30 IPs for pods.

To manage scaling efficiently, AWS pre-allocates a pool of secondary IPs using something called an ENI warm pool. If the pool runs out, new ENIs are attached to the node.

Why This Matters

• Direct Integration: Pods can communicate natively with AWS services like RDS and S3.
• High Performance: Native VPC networking avoids extra layers, keeping latency low.
• Challenges: You need to manage ENI limits carefully, as scaling involves adding more ENIs.

How GCP GKE Does It Differently

Google takes a slightly different approach, relying on alias IP ranges for pod networking. Here’s how it works:

1. Primary vs. Secondary IP Ranges

When setting up a VPC for GKE, you define:

• A primary IP range for the nodes.
• A secondary IP range specifically for pods.

This separation ensures pods get their own exclusive range of IPs, avoiding conflicts with other resources.

2. Simplified IP Allocation

Pods are automatically assigned IPs from the secondary range. There’s no need to worry about attaching additional network interfaces or running out of IPs tied to instance types.

Why This Matters

• Isolation: Pods have their own dedicated IP range, making management simpler.
• Scalability: You can reserve a large secondary range upfront to accommodate future growth.
• Challenges: You need to plan ahead when creating the network, as the secondary range must be defined at setup.

Which One Should You Choose?

Here’s the thing: both AWS EKS and GCP GKE have their strengths. Your choice depends on what you prioritize:

• Go with AWS EKS if you need tight integration with AWS services like RDS or S3 and value low-latency networking.
• Opt for GCP GKE if you want a clean separation of pod and node networking with straightforward scaling.

Wrapping It Up

Networking in Kubernetes might seem daunting, but understanding how cloud providers handle pod IPs can help you make better architectural decisions. AWS EKS’s ENI-based model offers deep VPC integration, while GCP GKE’s secondary range system simplifies scalability and isolation. Whichever path you choose, both are powerful tools in the Kubernetes ecosystem. Happy building!