Troubleshooting Pod Scheduling Issues in Amazon EKS

When a Kubernetes Pod does not start in Amazon EKS, sometimes, the first thing i check is CPU and memory.

In Amazon EKS, network capacity can also prevent Pods from starting.

A node may have plenty of available CPU and memory, but Kubernetes may still be unable to schedule another Pod because the node cannot allocate another network IP address.

In this article, i will share a notes to troubleshoot this problem

---

1. Start with the Pod Status

First, check the Pod:

kubectl get pods -n <namespace>

If the Pod is stuck in Pending.

inspect the Pod events with:

kubectl describe pod <pod-name> -n <namespace>

The Events section is usually the first place to look.

We may find messages related to:

• Insufficient CPU
• Insufficient memory
• Node affinity
• Taints and tolerations
• Maximum Pod capacity
• Network or IP allocation problems

I think it’s better to always start troubleshoot with the Pod events before making assumptions.

---

2. Check CPU and Memory

The most common reason for a Pod remaining Pending is insufficient resources.

Check the node:

kubectl top nodes

WE can also inspect the node:

kubectl describe node <node-name>

Look at:

Capacity
Allocatable
Allocated resources

Remember that Kubernetes schedules Pods based mainly on their resource requests.

For example:

resources:
  requests:
    cpu: 500m
    memory: 512Mi

The scheduler needs to find a node with enough allocatable CPU and memory for these requests.

However, even if these resources are available, the Pod may still fail to start.

---

3. Check Pod IP Capacity

This is especially important in Amazon EKS when using the Amazon VPC CNI.

With the Amazon VPC CNI, Pods receive IP addresses from our VPC network.

The networking model looks roughly like this:

VPC Subnet
    ↓
Node
    ↓
ENI
    ↓
Secondary IP Addresses
    ↓
Pods

Each Pod needs an IP address.

If Kubernetes cannot allocate another IP address for the Pod, the Pod may not start even if the node still has available CPU and memory.

This creates an important troubleshooting situation:

CPU Available      ✅
Memory Available   ✅
Pod IP Available   ❌

Result: Pod cannot start

---

4. Check Available Subnet IP Addresses

The Pod IP addresses ultimately come from VPC subnet.

Check whether the subnet has enough available IP addresses.

If the subnet is running out of IP addresses:

Subnet IP Capacity
        ↓
No available IP addresses
        ↓
VPC CNI cannot allocate IP
        ↓
Pod cannot start

This can become a problem when:

• Pod density increases (More application “pods” run on a single server)
• Multiple nodes use the same subnet
• The subnet CIDR range is too small

For example, a cluster might have enough EC2 compute capacity, but the subnet itself may not have enough IP addresses for additional Pods.

---

5. Check the Amazon VPC CNI

The Amazon VPC CNI is responsible for configuring networking for Pods.

One important component is:

aws-node

We can check it with:

kubectl get pods -n kube-system

The aws-node Pods run on the worker nodes and manage Pod networking.

If Pods cannot receive IP addresses, inspect the CNI logs and status.

The troubleshooting question becomes:

Can the VPC CNI allocate another IP address for this node?

---

6. Check ENI Limits

EC2 instances have limits on how many Elastic Network Interfaces (ENIs) they can attach.

They also have limits on how many IP addresses can be assigned to those ENIs.

We can refer to AWS Documentation IP Address per Network Interface.

This means that every instance type has a networking capacity.

For example:

EC2 Instance
      ↓
Maximum ENIs
      ↓
Maximum IPs per ENI
      ↓
Maximum Pod Networking Capacity

Even if:

CPU = Available
Memory = Available

the node might already have reached its networking capacity.

At that point:

Cannot attach another ENI
        ↓
Cannot allocate another Pod IP
        ↓
Cannot run additional Pods

---

7. Check Pod Density

Every node has a practical maximum number of Pods.

The number is influenced by several factors:

Maximum Pods per Node
        ↓
CPU Requests
Memory Requests
Pod IP Capacity
ENI Capacity
Kubernetes Configuration

This means the number of Pods is not determined only by CPU and memory.

For example:

Node Capacity:

CPU allows:        40 Pods
Memory allows:     50 Pods
Network allows:    29 Pods

The actual maximum becomes:

29 Pods

The smallest limiting resource determines how many Pods can actually run.

---

8. Check Prefix Delegation

One way to increase IP capacity with the Amazon VPC CNI is Prefix Delegation.

Instead of allocating individual IP addresses one by one, the CNI can allocate an IP prefix to an ENI.

Conceptually:

Without Prefix Delegation

ENI
├── IP
├── IP
├── IP
└── IP

With Prefix Delegation:

ENI
└── IP Prefix
      ├── Pod IP
      ├── Pod IP
      ├── Pod IP
      └── Pod IP

This can improve Pod IP availability and support higher Pod density.

However, enabling Prefix Delegation should be planned carefully because it affects VPC networking and IP address usage.

---

9. Consider Custom Networking

Sometimes the main VPC subnet does not have enough available IP addresses.

One solution is Custom Networking.

Custom Networking allows Pods to use different subnets from the worker nodes.

Conceptually:

Worker Node
     ↓
Primary ENI
     ↓
Node Subnet


Pods
     ↓
Secondary ENI
     ↓
Pod Subnet

This allows you to separate:

• Node IP address space
• Pod IP address space

It can also help when you need to scale the number of Pods beyond the available capacity of the original subnet.

---

A Practical Troubleshooting Flow

When Pods remain Pending in EKS, I recommend following this order:

Pod Pending
     ↓
kubectl describe pod
     ↓
Check Events
     ↓
Enough CPU and Memory?
     │
     ├── No → Check resource requests and node capacity
     │
     └── Yes
           ↓
      Check Pod Capacity
           ↓
      Check Available Subnet IPs
           ↓
      Check VPC CNI
           ↓
      Check ENI Limits
           ↓
      Check Prefix Delegation
           ↓
      Consider Custom Networking

---

Key Takeaway

When troubleshooting Pod scheduling problems in Amazon EKS, do not only think about CPU and memory.

A Kubernetes node has multiple resource limits.

For EKS, think about the cluster like this:

Can the Pod Start?

├── CPU available?
├── Memory available?
├── Scheduler constraints satisfied?
├── Node Pod capacity available?
├── Subnet IP available?
├── ENI capacity available?
└── VPC CNI able to allocate networking?

The first resource that reaches its limit becomes the bottleneck.

In production EKS environments, capacity planning should consider:

• Compute capacity
• CPU requests
• Memory requests
• Pod density
• VPC subnet IP addresses
• ENI limits
• Amazon VPC CNI behavior

CPU and memory may be available, but network capacity can become the bottleneck first.

Understanding all of these components together makes troubleshooting EKS much easier.