Automating EKS Auto Mode Pod Subnet Range Customization with EKS Node Classes, Karpenter Node Pools, and Terraform

#terraform #aws #networking #kubernetes

Introduction

On June 13, 2025, the EKS Auto Mode Node Class feature was updated to include podSubnetSelectorTerms and podSecurityGroupSelectorTerms parameters to specify subnets for pod IP addressing. You can use an EKS Node Class with a Karpenter Node Pool to specify additional configurations such as which types of instances to use for your workloads. You can then specify that Node Pool in any pod manifests that you deploy to your EKS Auto Mode cluster.

EKS clusters have supported pod-specific subnets since around 2019/2020, and it is great to see this feature come to EKS Auto Mode clusters. Now, how do we simplify using this feature? Let's automate it!

EKS Node Class and Karpenter Node Pool Creation...powered by Terraform!

Prerequisites:

Deploy an EKS Auto Mode cluster with an authentication mode that includes the API option.
Create the subnets you want to use with your pods (I created a /21 in each of my availability zones).
(Recommended) Create a security group specifically for your pods rather than reuse your node's security group for the podSecurityGroupSelectorTerms parameter in the Node Class spec.
(Optional) Set a distinct tag in each of the pod subnets from step 2 to reference in the Node Class spec.

NOTE

The Node Class podSubnetSelectorTerms parameter requires you to either specify the subnet IDs you want to use with your pods or the tag you assigned to those subnets (see the documented example). I opted to use tags and assigned the tag of kubernetes.io/role/cni with a value of 1. The tag key and value do not matter; what matters is that the tag you specify in your Node Class spec matches the actual subnet tag.

For this blog, I will focus on the Terraform automation for the EKS Node Class and Karpenter Node Pool.

Feature Deployment Checklist

Before we write our Terraform, let's review the steps required to configure dedicated subnets for our pods.

Create a new IAM role and Instance Profile for the nodes that will run the pods with the custom IP addresses.
Create an EKS Access Entry for that IAM role that has the AmazonEKSAutoNodePolicy access policy.
Create your EKS Node Class with your podSubnetSelectorTerms and podSecurityGroupSelectorTerms parameters.
Create your Karpenter Node Pool to assign any other desired characteristics like instance types, spot vs on-demand, etc, and reference your Node Class in the Node Pool spec.
Deploy a sample workload that uses the Karpenter Node Pool.

Fortunately, the AWS and Kubernetes Terraform providers include all the necessary functionality to complete these tasks. All of my Terraform code for this blog can be found on GitHub, but I will focus on interesting code snippets here:

First, the Kubernetes nodes need IAM permissions to manage AWS resources including assigning the pod subnets:

resource "aws_iam_role_policy_attachment" "custom_nodeclass_worker_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
  role       = aws_iam_role.custom_nodeclass_role.name
}

resource "aws_iam_role_policy_attachment" "custom_nodeclass_cni_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  role       = aws_iam_role.custom_nodeclass_role.name
}

resource "aws_iam_role_policy_attachment" "custom_nodeclass_registry_policy" {
  policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
  role       = aws_iam_role.custom_nodeclass_role.name
}

# VPC networking policy as well for the pod subnet functionality
resource "aws_iam_role_policy" "custom_nodeclass_vpc_policy" {
  name = "eks-t4g-nodeclass-vpc-policy"
  role = aws_iam_role.custom_nodeclass_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "ec2:DescribeSubnets",
          "ec2:DescribeNetworkInterfaces",
          "ec2:CreateNetworkInterface",
          "ec2:AttachNetworkInterface",
          "ec2:DeleteNetworkInterface",
          "ec2:DetachNetworkInterface",
          "ec2:ModifyNetworkInterfaceAttribute",
          "ec2:AssignPrivateIpAddresses",
          "ec2:UnassignPrivateIpAddresses"
        ]
        Resource = "*"
      }
    ]
  })
}

The Kubernetes nodes also need an EKS access entry with the AmazonEKSAutoNodePolicy access policy to join the EKS Auto Mode cluster:

resource "aws_eks_access_entry" "nodeclass_access_entry" {
  cluster_name      = local.cluster_name
  principal_arn     = aws_iam_role.custom_nodeclass_role.arn
  kubernetes_groups = []
  type             = "EC2"
}

resource "aws_eks_access_policy_association" "nodeclass_policy" {
  cluster_name  = local.cluster_name
  principal_arn = aws_iam_role.custom_nodeclass_role.arn
  policy_arn    = "arn:aws:eks::aws:cluster-access-policy/AmazonEKSAutoNodePolicy"

  access_scope {
    type = "cluster"
  }

  depends_on = [aws_eks_access_entry.nodeclass_access_entry]
}

Next comes the EKS Node Class and Karpenter Node Pool deployment. It is important to note that these are Kubernetes resources. As of this blog's publication date, it is not possible to use AWS APIs to create your Node Class nor your Node Pool. Hence we use the kubernetes_manifest resource type from the Terraform Kubernetes provider.

Notice the podSubnetSelectorTerm and the podSecurityGroupSelectorTerms parameters in the Node Class:

resource "kubernetes_manifest" "custom_nodeclass" {
  manifest = {
    apiVersion = "eks.amazonaws.com/v1"
    kind       = "NodeClass"
    metadata = {
      name = "t4g-nodeclass"
    }
    spec = {
      # Use the role NAME (not ARN)
      role = aws_iam_role.custom_nodeclass_role.name

      # Subnets for EC2 instances (nodes) - using tags for flexibility
      subnetSelectorTerms = [
        {
          tags = {
            "kubernetes.io/role/internal-elb" = "1"
          }
        }
      ]

      # Security groups for nodes
      securityGroupSelectorTerms = [
        {
          id = local.node_security_group_id
        }
      ]

      # Dedicated pod subnets - EKS Auto Mode will handle VPC CNI automatically
      podSubnetSelectorTerms = [
        {
          tags = {
            "kubernetes.io/role/cni" = "1"
          }
        }
      ]

      # Pod security groups (dedicated security group for pods)
      podSecurityGroupSelectorTerms = [
        {
          id = local.pod_security_group_id
        }
      ]

      # Let EKS Auto Mode manage networking automatically
      snatPolicy             = "Random"
      networkPolicy          = "DefaultAllow"
      networkPolicyEventLogs = "Disabled"

      # Ephemeral storage configuration optimized for t4g instances
      ephemeralStorage = {
        size       = "40Gi"
        iops       = 3000
        throughput = 125
      }

      # Tags for cost allocation and management
      tags = merge(
        local.common_tags,
        {
          NodeClass    = "t4g-nodeclass"
          InstanceType = "t4g-arm64"
          Purpose      = "general-workload"
        }
      )
    }
  }

  depends_on = [
    data.terraform_remote_state.eks_infrastructure,
    aws_eks_access_entry.nodeclass_access_entry,
    aws_iam_role.custom_nodeclass_role
  ]
}

# NodePool using the custom NodeClass
resource "kubernetes_manifest" "t4g_nodepool" {
  manifest = {
    apiVersion = "karpenter.sh/v1"
    kind       = "NodePool"
    metadata = {
      name = "t4g-nodepool"
    }
    spec = {
      template = {
        metadata = {
          labels = {
            nodepool-type = "t4g-arm64"
            billing-team  = local.project_name
            environment   = local.environment
          }
        }
        spec = {
          # Reference our custom NodeClass
          nodeClassRef = {
            group = "eks.amazonaws.com"
            kind  = "NodeClass"
            name  = "t4g-nodeclass"
          }

          # Requirements for t4g.small and t4g.medium only
          requirements = [
            {
              key      = "node.kubernetes.io/instance-type"
              operator = "In"
              values   = ["t4g.small", "t4g.medium"]
            },
            {
              key      = "kubernetes.io/arch"
              operator = "In"
              values   = ["arm64"]
            },
            {
              key      = "kubernetes.io/os"
              operator = "In"
              values   = ["linux"]
            },
            {
              key      = "karpenter.sh/capacity-type"
              operator = "In"
              values   = ["spot", "on-demand"]
            },
            # Distribute across all available zones
            {
              key      = "topology.kubernetes.io/zone"
              operator = "In"
              values   = local.availability_zones
            }
          ]

          # Node termination settings
          terminationGracePeriod = "30s"
        }
      }

      # Resource limits for the NodePool
      limits = {
        cpu    = "100"   # 100 vCPUs total limit
        memory = "400Gi" # 400 GiB memory total limit
      }

      # Disruption settings for cost optimization
      disruption = {
        consolidationPolicy = "WhenEmptyOrUnderutilized"
        consolidateAfter    = "30s"
        # expireAfter = "72h"  # Uncomment to auto-expire nodes after 72 hours
      }
    }
  }

  depends_on = [
    kubernetes_manifest.custom_nodeclass
  ]
}

Finally, I deploy a sample workload to verify that my pods get ip addresses from a different subnet than my nodes. Notice the nodeSelector parameter to specify our Node Pool:

resource "kubernetes_manifest" "test_deployment" {
  count = var.create_test_deployment ? 1 : 0

  manifest = {
    apiVersion = "apps/v1"
    kind       = "Deployment"
    metadata = {
      name      = "t4g-test-app"
      namespace = "default"
    }
    spec = {
      replicas = 2
      selector = {
        matchLabels = {
          app = "t4g-test-app"
        }
      }
      template = {
        metadata = {
          labels = {
            app = "t4g-test-app"
          }
        }
        spec = {
          # Node selector to target our t4g NodePool
          nodeSelector = {
            "nodepool-type" = "t4g-arm64"
          }

          containers = [
            {
              name  = "nginx"
              image = "nginx:alpine"
              ports = [
                {
                  containerPort = 80
                }
              ]
              resources = {
                requests = {
                  cpu    = "100m"
                  memory = "128Mi"
                }
                limits = {
                  cpu    = "200m"
                  memory = "256Mi"
                }
              }
            }
          ]
        }
      }
    }
  }

  depends_on = [
    kubernetes_manifest.t4g_nodepool
  ]
}

Verifying our work

If the configuration was successful, the terraform apply succeeds AND my pods and nodes are in different subnets. Let's use kubectl to verify.

Checking the Node

❯ kubectl describe node i-04ddfb1f83f3288fe
Name:               i-04ddfb1f83f3288fe
.
.
.
Addresses:
  InternalIP:   10.0.33.121
  InternalDNS:  ip-10-0-33-121.ec2.internal
  Hostname:     ip-10-0-33-121.ec2.internal

Checking the Pods

❯ kubectl get pods -o wide
NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE                  NOMINATED NODE   READINESS GATES
t4g-test-app-5ccc4dcd59-6p5hp   1/1     Running   0          57s   10.0.41.144   i-04ddfb1f83f3288fe   <none>           <none>
t4g-test-app-5ccc4dcd59-8tfvx   1/1     Running   0          57s   10.0.41.145   i-04ddfb1f83f3288fe   <none>           <none>

Success! My node's subnet CIDR is 10.0.33.0/24, and my pods' subnet CIDR is 10.0.40.0/21; my node IP (10.0.33.121) and pod IPs (10.0.41.144 and 10.0.41.145) fit into their respective subnet IP ranges.

Wrapping Things Up...

In this blog post, we discussed EKS Auto Mode's support for pod-specific subnets by leveraging EKS Node Class and Karpenter Node Pool resources. Further, we demonstrated how to automate this process with Terraform.

If you found this article useful, let me know on BlueSky or on LinkedIn!