On Cloud Nine with Ratnopam

Reduce cross-AZ traffic costs on EKS using topology aware hints

Ratnopam Chakrabarti — Wed, 04 Jan 2023 02:56:37 GMT

From a high availability standpoint, it's considered a best practice to spread workloads across multiple nodes in an EKS cluster. In addition to having multiple replicas of the application, one should also consider spreading the workload across multiple Availability Zones to attain high availability and improve reliability. This ensures fault-tolerance and avoids application downtime in the event of a worker node failure. One way to achieve this kind of deployment in EKS is to use podAnitiAffinity. For example, the below manifest tells EKS scheduler to deploy each replica of the Pod on a node that's in a separate Availability Zone(AZ).

apiVersion: apps/v1kind: Deploymentmetadata:  name: spread-az  labels:    app: web-serverspec:  replicas: 3  selector:    matchLabels:      app: web-server  template:    metadata:      labels:        app: web-server    spec:      affinity:        podAntiAffinity:          requiredDuringSchedulingIgnoredDuringExecution:          - labelSelector:              matchExpressions:              - key: app                operator: In                values:                - web-server            topologyKey: kubernetes.io/zone      containers:      - name: web-app        image: nginx:1.16-alpine

The above manifest makes use of topologyKey kubernetes.io/zone . It tells the Kubernetes scheduler not to schedule two Pods in same AZ.

One of the other approaches that can be used to spread Pods across AZs is to use Pod Topology Spread Constraints which was GA-ed in Kubernetes 1.19. This mechanism aims to spread pods evenly onto multiple node topologies.

While both of these approaches provide high-availability and resiliency for application workloads, customers incur costs for data transfers in inter-AZ traffic routing within an EKS cluster. For large EKS clusters running hundreds of nodes and thousands of pods, the data transfer costs for cross-AZ traffic can be significant.

Enter Topology Aware Hints

To address cross-AZ data transfer costs (which comes up during many EKS conversations on cost optimzation), pods running in a cluster must be able to perform topology-aware routing based on Availability Zone. And this is precisely what Topology Aware Hints helps achieve. Topology Aware Hints provides a mechanism to help keep traffic within the zone it originated from. Prior to topology-aware-hints, Service topology-keys could be used for similar functionality. This was deprecated in kubernetes 1.21 in favor of topology-aware-hints which was introduced in kubernetes 1.21 and became "beta" in Kubernetes 1.23. With EKS 1.24 however, this is enabled by default and EKS users and customers can leverage this feature to keep kubernetes service traffic within the same AZ.

Let's dive in further and see this in action!

For the purposes of this blogpost, let's create a three-node EKS cluster.

Type the following command in your cloud9 terminal.

cat <<EOF>> eks-config.yamlapiVersion: eksctl.io/v1alpha5kind: ClusterConfigmetadata:  name: topology-demo-cluster  region: us-west-2  version: "1.24"managedNodeGroups:  - name: appservers    instanceType: t3.xlarge    desiredCapacity: 3    minSize: 1    maxSize: 4    labels: { role: appservers }    volumeSize: 8    iam:      withAddonPolicies:        imageBuilder: true        autoScaler: true        xRay: true        cloudWatch: true        albIngress: true    ssh:       enableSsm: trueEOFeksctl create cluster -f eks-config.yaml

Once the cluster is created, check the status of the worker nodes and their distribution across AZs.

kubectl get nodes -L topology.kubernetes.io/zoneNAME                                           STATUS   ROLES    AGE   VERSION               ZONEip-192-168-4-149.us-west-2.compute.internal    Ready    <none>   36h   v1.24.7-eks-fb459a0   us-west-2bip-192-168-48-125.us-west-2.compute.internal   Ready    <none>   36h   v1.24.7-eks-fb459a0   us-west-2cip-192-168-75-68.us-west-2.compute.internal    Ready    <none>   36h   v1.24.7-eks-fb459a0   us-west-2d

We have each worker node in a separate AZ deployed in our EKS cluster. Let's now try to run a sample application in this cluster.

Use the below application manifest to deploy three replicas of our sample application to deploy in the newly created EKS cluster as below

cat <<EOF>> app-manifest.yamlapiVersion: v1kind: Namespacemetadata:   name: topology-demo-ns---apiVersion: apps/v1kind: Deploymentmetadata:  name: getaz  namespace: topology-demo-ns  labels:    app: getazspec:  replicas: 3  selector:    matchLabels:      app: getaz  template:    metadata:      labels:        app: getaz      namespace: topology-demo-ns    spec:      topologySpreadConstraints:      - maxSkew: 1        topologyKey: topology.kubernetes.io/zone        whenUnsatisfiable: DoNotSchedule        labelSelector:          matchLabels:            app: getaz      containers:      - name: getaz-container        image: getazcontainer:latest        imagePullPolicy: Always        ports:        - containerPort: 3000          name: web-port        resources:          requests:            cpu: "256m"---apiVersion: v1kind: Servicemetadata:  name: getazservice  namespace: topology-demo-nsspec:  selector:    app: getaz  ports:    - port: 80      targetPort: web-port      protocol: TCPEOFkubectl apply -f app-manifest.yaml

The application manifest creates -

a Namespace named "topology-demo-ns"
a Deployment named "getaz" with three Pods. Each Pod runs a container named "getaz-container".
a Service named "getazservice".

The "getaz" Pods and Service "getazservice" all run in the "topology-demo-ns" namespace.

In the above example, we're using the Pod topologySpreadConstraints with maxSkew set to 1 and whenUnsatisfiable set to "DoNotSchedule" to deploy each replica of our sample application in a separate AZ. This example leverages a well-known node label called topology.kubernetes.io/zone that worker nodes in an EKS cluster is assigned to by default, as the topologyKey in the pod topology spread. To get the labels on a worker node in the EKS cluster that we spun up, use the below command

$ kubectl describe node ip-192-168-48-125.us-west-2.compute.internalName:               ip-192-168-48-125.us-west-2.compute.internalRoles:              <none>Labels:             alpha.eksctl.io/cluster-name=topology-demo-cluster                    alpha.eksctl.io/nodegroup-name=appservers                    beta.kubernetes.io/arch=amd64                    beta.kubernetes.io/instance-type=t3.xlarge                    beta.kubernetes.io/os=linux                    eks.amazonaws.com/capacityType=ON_DEMAND                    eks.amazonaws.com/nodegroup=appservers                    eks.amazonaws.com/nodegroup-image=ami-0b149b4c68ab69dce                    eks.amazonaws.com/sourceLaunchTemplateId=lt-0a47ee5069d44e8d4                    eks.amazonaws.com/sourceLaunchTemplateVersion=1                    failure-domain.beta.kubernetes.io/region=us-west-2                    failure-domain.beta.kubernetes.io/zone=us-west-2c                    k8s.io/cloud-provider-aws=8d60a23f89f8b00a31bfef5d05edc662                    kubernetes.io/arch=amd64                    kubernetes.io/hostname=ip-192-168-48-125.us-west-2.compute.internal                    kubernetes.io/os=linux                    node.kubernetes.io/instance-type=t3.xlarge                    role=appservers                    topology.kubernetes.io/region=us-west-2                    topology.kubernetes.io/zone=us-west-2c

In the topologySpreadConstraints section of the example manifest,

maxSkew defines the degree to which Pods may be distributed unevenly. This field must be filled out, and the value must be greater than zero. Its semantics vary depending on the value of whenUnsatisfiable field.

whenUnsatisfiable specifies how to handle a Pod placement that does not satisfy the spread constraint:

- DoNotSchedule (the default value) instructs the scheduler not to schedule it.

- ScheduleAnyway instructs the scheduler to continue scheduling it while prioritizing Nodes with the lowest skew.

Let's check the status and spread of our application pods.

kubectl get po -n topology-demo-ns -o wideNAME                    READY   STATUS    RESTARTS   AGE   IP               NODE                                           NOMINATED NODE   READINESS GATESgetaz-9685bbd44-65wcn   1/1     Running   0          2m    192.168.63.154   ip-192-168-48-125.us-west-2.compute.internal   <none>           <none>getaz-9685bbd44-kf7gs   1/1     Running   0          2m    192.168.69.57    ip-192-168-75-68.us-west-2.compute.internal    <none>           <none>getaz-9685bbd44-tjqkd   1/1     Running   0          2m    192.168.24.149   ip-192-168-4-149.us-west-2.compute.internal    <none>           <none>

We see from above output that each replica is running on a separate node and since each node is running in a separate AZ, effectively we have three pods each running in a different AZ in the EKS cluster.

For getting detail information about topologySpreadConstraints, you can use kubectl explain Pod.spec.topologySpreadConstraints command. You can mix and match these attributes to achieve different spread topologies.

Let us now check the Service that got created by deploying the app-manifest.yaml file.

kubectl -n topology-demo-ns get svcNAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGEgetazservice   ClusterIP   10.100.9.165   <none>        80/TCP    53mkubectl -n topology-demo-ns describe svc getazservice Name:              getazserviceNamespace:         topology-demo-nsLabels:            <none>Annotations:       <none>Selector:          app=getazType:              ClusterIPIP Family Policy:  SingleStackIP Families:       IPv4IP:                10.100.9.165IPs:               10.100.9.165Port:              <unset>  80/TCPTargetPort:        web-port/TCPEndpoints:         192.168.24.149:3000,192.168.63.154:3000,192.168.69.57:3000Session Affinity:  NoneEvents:            <none>

We have a Service named "getazservice" of ClusterIP type deployed. The service doesn't have any Annotations set on it.

As next step, let's deploy a test container that we're going to use to call "getazservice" and check if there are any inter-AZ calls we can spot.

Use the below command to deploy a curl container and ensure curl is installed.

kubectl run curl-debug --image=radial/busyboxplus:curl -l "type=debug" -n topology-demo-ns -it --tty  sh# check if curl is installedcurl --version#exit the containerexit

Once the debug container is running, create a bash script to call "getazservice" in a loop and print the Availability Zone of the pod that responded to the call.

kubectl exec -it --tty -n topology-demo-ns $(kubectl get pod -l  "type=debug" -n topology-demo-ns -o  jsonpath='{.items[0].metadata.name}') sh #create a test script and call service cat <<EOF>> test.sh n=1 while [ \$n -le 5 ] do     curl -s getazservice.topology-demo-ns     sleep 1     echo "---"     n=\$(( n+1 )) doneEOF chmod +x test.sh clear ./test.sh #exit the test container exit

The above execution of the test script in the debug container should produce an output like below which shows that the calls to the "getazservice" Service and its backing Pods are distributed across AZs.

us-west-2d---us-west-2b---us-west-2d---us-west-2c---us-west-2d---

The load-balancing and forwarding logic of the service call in this case is based on kube-proxy mode. EKS by default implements "iptables" mode of kube-proxy. When the curl-debug container sends the curl request to the "getazservice" virtual IP, the packet is then processed by the iptables rules on that worker node which are configured by the kube-proxy. Then a Pod backing the "getazservice" Service gets chosen at random by default. For detailed documentation on different kube-proxy modes(iptables, ipvs) please refer to Kubernetes Documentation.

To avoid this "randomness" of routing and reduce the cost of inter-AZ traffic routing and network latency, topology-aware-hints can be activated for the Service to ensure that the service call is routed to a Pod that resides in the same AZ as that of the Pod which the request originated from.

To enable topology-aware routing, simply add the service.kubernetes.io/topology-aware-hints annotation to "auto" for the "getazservice" as below and re-deploy the manifest.

apiVersion: v1kind: Servicemetadata:  name: getazservice  namespace: topology-demo-ns  annotations:    service.kubernetes.io/topology-aware-hints: autospec:  selector:    app: getaz  ports:    - port: 80      targetPort: web-port      protocol: TCP

When we describe the Service, we see the Annotation associated with it.

kubectl -n topology-demo-ns describe svc getazservice Name:              getazserviceNamespace:         topology-demo-nsLabels:            <none>Annotations:       service.kubernetes.io/topology-aware-hints: autoSelector:          app=getazType:              ClusterIPIP Family Policy:  SingleStackIP Families:       IPv4IP:                10.100.9.165IPs:               10.100.9.165Port:              <unset>  80/TCPTargetPort:        web-port/TCPEndpoints:         192.168.24.149:3000,192.168.63.154:3000,192.168.69.57:3000Session Affinity:  NoneEvents:            <none>

If we run the same test as before with the debug container, this time we should see an output similar to below

us-west-2b---us-west-2b---us-west-2b---us-west-2b---us-west-2b---

This shows that the calls to "getazservice" is getting consistently picked up by the backing Pod that resides in the same AZ as the requester Pod. Topology aware routing in this case is enabled by EndPointSlice Controller and the kube-proxy components. EndPointSlice API in Kubernetes provides a way to track network endpoints within a cluster. EndpointSlices offer a more scalable and extensible alternative to Endpoints and is available since Kubernetes 1.21. When calculating the endpoints for a Service that's annotated with service.kubernetes.io/topology-aware-hints: auto , the EndpointSlice controller considers the topology (region and zone) of each Service endpoint and populates the hints field to allocate it to a zone. Once the "hints" are populated, kube-proxy can then consume these hints, and use them to influence how the traffic is routed (favoring topologically closer endpoints).

This solution reduces inter-AZ traffic routing and in turn lowers the cross-AZ data transfer costs in an EKS cluster. By enabling "intelligent" routing, it also helps reduce the network latency. While this approach works well in most cases, sometimes the EndpointSlice controller allocates endpoints from a different zone to ensure more even distribution of endpoints between zones. This results in some traffic being routed to other zones. Thus, when using Topology-Aware-hints, its important to have application pods balanced across AZs using Topology Spread Constraints to avoid imbalances in the amount of traffic handled by each pod. Additionally, there are some other safeguards and constraints that one should be aware of before using this approach. As alternative solutions, one can use Service Mesh technologies like Istio or Linkerd to achieve topology-aware routing; however service mesh based solutions present additional complexities for the cluster operators to manage. In comparison, using topology-aware-hints is much simpler to implement, is supported out-of-the-box in EKS 1.24 and works great in reducing cross-AZ traffic costs within an EKS cluster.

]]>

Taking Amazon EKS Anywhere for a spin

Ratnopam Chakrabarti — Sun, 26 Jun 2022 10:14:49 GMT

Prelude

Before we discuss about EKS Anywhere, it's useful to have a basic idea about Amazon EKS and more importantly Amazon EKS Distro.

Amazon Elastic Kubernetes Service a.k.a. Amazon EKS is a managed kubernetes service from AWS to run and scale Kubernetes applications on AWS cloud platform. EKS uses Amazon EKS Distro otherwise known as EKS-D to create reliable and secure Kubernetes clusters.

EKS Distro includes binaries and containers of open-source Kubernetes, etcd (cluster configuration database), networking, and storage plugins, tested for compatibility.

Maintaining and running your own Kubernetes clusters takes a lot of effort for teams in tracking updates, figuring out compatibility between different kubernetes versions and simply keeping up-to-date with upstream kubernetes release cadence. This is where EKS-D comes to the rescue. EKS Distro reduces the need to track updates, determine compatibility, and standardize on a common Kubernetes version across teams.

WHAT is EKS Anywhere

With that basic understanding of EKS and EKS-D, let's now take a closer look at EKS-Anywhere.

Amazon EKS Anywhere is a new deployment option for Amazon EKS that enables you to easily create and operate Kubernetes clusters on-premises with your own virtual machines. EKS Anywhere is an open-source deployment option for Amazon EKS that builds on the strengths of EKS-D and allows teams to create and operate Kubernetes clusters on-premises. EKS Anywhere is based on the overarching design principle that supports BYOI (Bring Your Own Infrastructure) model when it comes to deploying kubernetes clusters. It supports deploying production grade kubernetes clusters on VMWare's vSphere and plans to add support for bare metal in 2022.

WHAT problem does it solve

The salient use cases that EKS Anywhere solves are

Hybrid cloud consistency
Disconnected environment
Application modernization
Data sovereignty

The core benefit of using EKS Anywhere is that it offers customers a consistent and reliable mechanism of running Amazon's kubernetes distribution within their own on-premises infrastructure. Some enterprises have a mix of deployment architecture with some kubernetes workload running in cloud on AWS EKS while some other applications still running in on-premise kubernetes clusters. EKS Anywhere offers strong operational consistency with Amazon EKS so teams can standardize their Kubernetes operations across a hybrid cloud environment based on a unified toolset.

Businesses that have a large on-premises footprint and want to modernize their applications can leverage EKS Anywhere to simplify the creation and operation of on-premises Kubernetes clusters and focus more on developing and modernizing applications. In addition, customers who want to keep their data within private data centers due to legal reasons can benefit by using trusted Amazon EKS Kubernetes distribution and tools to where their data needs to be.

WHY should one use it

Kubernetes adoption is growing every day and the tooling around it also keeps on piling up. While this provides customers with multiple options, it is quite challenging to ensure that teams pick the right tools for the job and doesn't add complexity in their operations workflow. Keeping pace with upstream kubernetes release cadence without breaking existing applications is also a non-trivial task.

Teams that are operating kubernetes clusters on-premises typically need to take on a lot of operational challenges such as creating and upgrading clusters in a timely manner with upstream releases, maintaining and resolving version mismatches between kubernetes releases and integrating a variety of third party tools to perform cluster operations. The same applies to hybrid cluster set ups and leads to unnecessary complexity, fragmented tooling and support options, and inconsistencies between the cloud and on-premises clusters that make it hard to manage applications across environments.

With Amazon EKS Anywhere, teams have Kubernetes operational tooling that is consistent with Amazon EKS and is optimized to simplify cluster installation with default configurations for the operating system and networking needed to operate Kubernetes on-premises. If you're someone who wants to reduce operational complexity, adopt a consistent and reliable workflow of managing kubernetes clusters across cloud and on-premises, leverage latest tooling and security hardened updated kubernetes distribution to operate on then EKS Anywhere might be a worthy option for you.

Kicking The Tires

EKS Anywhere allows you to create and manage production kubernetes cluster on VMWare vSpehere. However, if you don't have a vSphere environment at your disposal, EKS Anywhere also supports creating development clusters locally with Docker provider.

Here, I will walk you through the cluster creation process on a virtual machine using the Docker provider. This set up is for local use and not recommended for Production purposes. However the concepts around the cluster creation and management workflow across providers are the same.

Clusters and more Clusters

This is the path we're going to take for the purposes of this post.

Cluster Management Workflow

The EKS Anywhere cluster creation process makes it easy not only to bring up a cluster initially, but also to update configuration settings and to upgrade Kubernetes versions going forward. The cluster creation process involves stepping through different types of clusters.

Bootstrap cluster - A temporary kubernetes clusters that's ephemeral in nature and is solely used for creating a management cluster.
Management cluster - A Kubernetes cluster that manages the lifecycle of Workload Clusters.
Workload cluster - A Kubernetes cluster whose lifecycle is managed by a Management Cluster.

To manage the lifecycle of a Workload kubernetes cluster we need to have a Management kubernetes cluster in place first. And to have a Management cluster, we need to spin up a bootstrap kubernetes cluster to start the cluster creation workflow.

Think of the Bootstrap cluster as a launchpad for EKS Anywhere to start the Workload cluster creation process. Once it's able to create the Management cluster, from that point on, Management cluster continues the process and takes over the lifecycle management of the Workload cluster. The essence of this design paradigm is to make kubernetes clusters "self-aware" and be able to manage its lifecycle themselves without the need of a launchpad (i.e. Bootstrap cluster). A common practice is to delete the bootstrap cluster once its job is done and repurpose the infrastructure to save resource.

An obvious question about the above scenario is how to spin up the bootstrap cluster and how to avoid a chicken and egg situation where we attempt to create bootstrap cluster using EKS Anywhere even before there's a launchpad cluster in place.

Enter [KinD] (https://kind.sigs.k8s.io/) or Kubernetes in Docker. KinD can create kubernetes clusters that run kubernetes as docker containers. EKS Anywhere runs a KinD cluster on an administrative workstation or virtual machine to act as a bootstrap cluster.

Let's roll up our sleeves now and see EKS Anywhere in action!

Prerequisites

To start with, prepare your Administrative Workstation with following pre-requisites.

Docker 20.x.x
Ubuntu (20.04.2 LTS). If you're on a Mac use 10.15
4 CPU cores
16GB memory
30GB free disk space

Make sure that your local workstation or virtual machine in cloud meets all of the above requirements.

I used a virtual machine in cloud running Ubuntu 20.04 LTS with above configurations.

Docker

Install Docker on Ubuntu.Check the docker version to make sure it's 20.x.x.

$ docker versionClient: Docker Engine - Community Version:           20.10.17 API version:       1.41 Go version:        go1.17.11 Git commit:        100c701 Built:             Mon Jun  6 23:02:57 2022 OS/Arch:           linux/amd64 Context:           default Experimental:      trueServer: Docker Engine - Community Engine:  Version:          20.10.17  API version:      1.41 (minimum version 1.12)  Go version:       go1.17.11  Git commit:       a89b842  Built:            Mon Jun  6 23:01:03 2022  OS/Arch:          linux/amd64  Experimental:     false containerd:  Version:          1.6.6  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 runc:  Version:          1.1.2  GitCommit:        v1.1.2-0-ga916309 docker-init:  Version:          0.19.0  GitCommit:        de40ad0

kubectl

You need kubectl installed to connect to kubernetes clusters from your workstation. If you don't have it installed, use the snap commands to install it.

sudo snap install kubectl --classic

eksctl

Install the latest release of eksctl. The EKS Anywhere plugin requires eksctl version 0.66.0 or newer.

curl "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \    --silent --location \    | tar xz -C /tmpsudo mv /tmp/eksctl /usr/local/bin/

eksctl anywhere plugin

Install the eksctl-anywhere plugin.

export EKSA_RELEASE="0.9.1" OS="$(uname -s | tr A-Z a-z)" RELEASE_NUMBER=12curl "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/${RELEASE_NUMBER}/artifacts/eks-a/v${EKSA_RELEASE}/${OS}/amd64/eksctl-anywhere-v${EKSA_RELEASE}-${OS}-amd64.tar.gz" \    --silent --location \    | tar xz ./eksctl-anywheresudo mv ./eksctl-anywhere /usr/local/bin/

Verify your installed version.

$ eksctl anywhere versionv0.9.1

Create your local EKS Anywhere cluster

Now that you have all the tools installed, let's proceed with creating the local EKS Anywhere cluster using the Docker provider.

First we create a cluster configuration and save it in a file.

$ CLUSTER_NAME=rc-devubuntu@ip-10-0-0-12:~$ eksctl anywhere generate clusterconfig $CLUSTER_NAME \>    --provider docker > $CLUSTER_NAME.yaml

Check the configuration file.

$ cat rc-dev.yaml apiVersion: anywhere.eks.amazonaws.com/v1alpha1kind: Clustermetadata:  name: rc-devspec:  clusterNetwork:    cniConfig:      cilium: {}    pods:      cidrBlocks:      - 192.168.0.0/16    services:      cidrBlocks:      - 10.96.0.0/12  controlPlaneConfiguration:    count: 1  datacenterRef:    kind: DockerDatacenterConfig    name: rc-dev  externalEtcdConfiguration:    count: 1  kubernetesVersion: "1.22"  managementCluster:    name: rc-dev  workerNodeGroupConfigurations:  - count: 1    name: md-0---apiVersion: anywhere.eks.amazonaws.com/v1alpha1kind: DockerDatacenterConfigmetadata:  name: rc-devspec: {}---

By default, EKS Anywhere creates a kubernetes cluster with one control plane and one worker node. Currently it installs the kubernetes 1.22 version.

Another configuration worth noticing is the default cni provider which is cilium.

You can customize and alter these settings to suit your workload cluster requirement.

Start Small

We're going to start with installing our first EKS Anywhere cluster with bare minimum set up with 1 control plane and 1 worker node. Later on, we'd increase the worker node count.

Once we have the configuration file, all you need to do is

$ time eksctl anywhere create cluster -f $CLUSTER_NAME.yamlPerforming setup and validationsWarning: The docker infrastructure provider is meant for local development and testing only Docker Provider setup is valid Validate certificate for registry mirror Create preflight validations passCreating new bootstrap clusterProvider specific pre-capi-install-setup on bootstrap clusterInstalling cluster-api providers on bootstrap clusterProvider specific post-setupCreating new workload clusterInstalling networking on workload clusterInstalling cluster-api providers on workload clusterInstalling EKS-A secrets on workload clusterMoving cluster management from bootstrap to workload clusterInstalling EKS-A custom components (CRD and controller) on workload clusterInstalling EKS-D components on workload clusterCreating EKS-A CRDs instances on workload clusterInstalling AddonManager and GitOps Toolkit on workload clusterGitOps field not specified, bootstrap flux skippedWriting cluster config fileDeleting bootstrap cluster🎉 Cluster created!real    5m1.553suser    0m2.941ssys    0m2.084s

Within approx. 5 mins you have the workload cluster ready for use without much hassle; that's pretty neat!

By default, the console log above highlights the different stages of the workload cluster creation lifecycle; however if you want to see more logs, then add the -v parameter in the eksctl anywhere cluster create command to tun on the verbose mode.

You can check the bootstrap cluster by issuing the below command. Install KinD if you don't have it on your workstation by following these instructions.

$ kind get clustersrc-dev

Now it's time to verify the cluster and check kubernetes version installed.

# export the kubeconfig to point to the cluster$ export KUBECONFIG=${PWD}/${CLUSTER_NAME}/${CLUSTER_NAME}-eks-a-cluster.kubeconfig# check nodes of the k8s cluster$ kubectl get nodesNAME                           STATUS   ROLES                  AGE   VERSIONrc-dev-gzktk                   Ready    control-plane,master   15m   v1.22.6-eks-bb942e6rc-dev-md-0-7c4c7f595d-5lcq8   Ready    <none>                 14m   v1.22.6-eks-bb942e

Spin Up a multi node cluster

While the basic cluster with a control plane and a worker node is nice, having the ability to spin up a multi node kubernetes cluster is fantastic. Let's see if EKS Anywhere is up to the task.

In order to increase the number of nodes in the cluster, you need to modify the cluster configuration file. There's no option to pass this as a parameter to the eksctl anywhere command; at least for now.

$ cp rc-dev.yaml rc-dev-multinode.yaml#change metadata and name of the cluster configuration$ sed -i 's/rc-dev/rc-dev-multinode/g' rc-dev-multinode.yaml

And finally, change the workerNodeGroupConfigurations.count value to 3. Now let's deploy the cluster.

$ eksctl anywhere create cluster -f rc-dev-multinode.yaml

Verify cluster

Export the kubeconfig file as before and check nodes status

$ kubectl get nodesNAME                                  STATUS   ROLES                  AGE     VERSION$ kubectl get nodesNAME                                     STATUS   ROLES                  AGE     VERSIONrc-dev-multinode-7dwtd                   Ready    control-plane,master   2m40s   v1.22.6-eks-bb942e6rc-dev-multinode-md-0-86786b8dfb-dsfrw   Ready    <none>                 2m9s    v1.22.6-eks-bb942e6rc-dev-multinode-md-0-86786b8dfb-gpx2x   Ready    <none>                 2m10s   v1.22.6-eks-bb942e6rc-dev-multinode-md-0-86786b8dfb-tx6b4   Ready    <none>                 2m10s   v1.22.6-eks-bb942e6

Also, check for all system pods if they're in running state.

$ kubectl get po -ANAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGEcapd-system                         capd-controller-manager-6466d54c9d-j9klq                         1/1     Running   0          112scapi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-7c6bcf98b7-tb2zx       1/1     Running   0          2m1scapi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-68b4c848dc-zz6gs   1/1     Running   0          115scapi-system                         capi-controller-manager-64647f455d-x5zx7                         1/1     Running   0          2m3scert-manager                        cert-manager-8674857d7b-5hmgq                                    1/1     Running   0          2m41scert-manager                        cert-manager-cainjector-f5b94ccdf-9bglv                          1/1     Running   0          2m41scert-manager                        cert-manager-webhook-84fbd6fb68-6xkkj                            1/1     Running   0          2m40seksa-system                         eksa-controller-manager-79b4b76bb8-lbc79                         2/2     Running   0          92setcdadm-bootstrap-provider-system   etcdadm-bootstrap-provider-controller-manager-664f699b7c-q44db   1/1     Running   0          2metcdadm-controller-system           etcdadm-controller-controller-manager-59dc96c7b9-hpnnp           1/1     Running   0          118skube-system                         cilium-858bk                                                     1/1     Running   0          2m51skube-system                         cilium-89r45                                                     1/1     Running   0          2m51skube-system                         cilium-bxfv6                                                     1/1     Running   0          2m51skube-system                         cilium-operator-7698596ff4-2k264                                 1/1     Running   0          2m51skube-system                         cilium-operator-7698596ff4-cgjqn                                 1/1     Running   0          2m51skube-system                         cilium-z4zgt                                                     1/1     Running   0          2m51skube-system                         coredns-55467bc785-n4mv2                                         1/1     Running   0          3m15skube-system                         coredns-55467bc785-qgdhx                                         1/1     Running   0          3m15skube-system                         kube-apiserver-rc-dev-multinode-7dwtd                            1/1     Running   0          3m18skube-system                         kube-controller-manager-rc-dev-multinode-7dwtd                   1/1     Running   0          3m18skube-system                         kube-proxy-2npkx                                                 1/1     Running   0          3m16skube-system                         kube-proxy-7j7vm                                                 1/1     Running   0          2m56skube-system                         kube-proxy-8ntxj                                                 1/1     Running   0          2m56skube-system                         kube-proxy-cwgtn                                                 1/1     Running   0          2m55skube-system                         kube-scheduler-rc-dev-multinode-7dwtd                            1/1     Running   0          3m18s

To verify that a cluster control plane is up and running, use the kubectl command to show that the control plane pods are all running.

$ kubectl get po -A -l control-plane=controller-managerNAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGEcapd-system                         capd-controller-manager-6466d54c9d-j9klq                         1/1     Running   0          20mcapi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-7c6bcf98b7-tb2zx       1/1     Running   0          20mcapi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-68b4c848dc-zz6gs   1/1     Running   0          20mcapi-system                         capi-controller-manager-64647f455d-x5zx7                         1/1     Running   0          20metcdadm-bootstrap-provider-system   etcdadm-bootstrap-provider-controller-manager-664f699b7c-q44db   1/1     Running   0          20metcdadm-controller-system           etcdadm-controller-controller-manager-59dc96c7b9-hpnnp           1/1     Running   0          20m

Once you've verified that all nodes in the workload cluster are in ready state and the pods are in Running status, you can go ahead and deploy some test workloads onto the cluster.

Deploy sample workload

To deploy a sample test workload in your shiny new multinode workload cluster, do

$ kubectl apply -f "https://anywhere.eks.amazonaws.com/manifests/hello-eks-a.yaml"deployment.apps/hello-eks-a createdservice/hello-eks-a created

Check all kubernetes resources in the default namespace.

$ kubectl get allNAME                              READY   STATUS    RESTARTS   AGEpod/hello-eks-a-9644dd8dc-znwzr   1/1     Running   0          37sNAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGEservice/hello-eks-a   NodePort    10.106.132.175   <none>        80:30687/TCP   37sservice/kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP        6m32sNAME                          READY   UP-TO-DATE   AVAILABLE   AGEdeployment.apps/hello-eks-a   1/1     1            1           37sNAME                                    DESIRED   CURRENT   READY   AGEreplicaset.apps/hello-eks-a-9644dd8dc   1         1         1       37s

To access the default web-page of the sample workload, forward the deployment port to your localhost

$ kubectl port-forward deploy/hello-eks-a 8000:80Forwarding from 127.0.0.1:8000 -> 80Forwarding from [::1]:8000 -> 80

From a second terminal, try accessing the webpage by doing curl localhost:8000

Scale your cluster

Currently, the only way you can scale the workload cluster is by manually incrementing the number of control plane and worker nodes in the cluster configuration file and upgrading the cluster.

Increment the worker node count to 5 as shown below

$ cat rc-dev-multinode-more.yamlapiVersion: anywhere.eks.amazonaws.com/v1alpha1kind: Clustermetadata:  name: rc-dev-multinodespec:  clusterNetwork:    cniConfig:      cilium: {}    pods:      cidrBlocks:      - 192.168.0.0/16    services:      cidrBlocks:      - 10.96.0.0/12  controlPlaneConfiguration:    count: 1  datacenterRef:    kind: DockerDatacenterConfig    name: rc-dev-multinode  externalEtcdConfiguration:    count: 1  kubernetesVersion: "1.21"  managementCluster:    name: rc-dev-multinode  workerNodeGroupConfigurations:  - count: 5    name: md-0---apiVersion: anywhere.eks.amazonaws.com/v1alpha1kind: DockerDatacenterConfigmetadata:  name: rc-dev-multinodespec: {}---

Let's give it a try to see if we can upgrade the workload cluster to add more worker nodes.

$ eksctl anywhere upgrade cluster -f rc-dev-multinode-more.yaml Performing setup and validations Docker Provider setup is valid Validate certificate for registry mirror Control plane ready Worker nodes ready Nodes ready Cluster CRDs ready Cluster object present on workload cluster Upgrade cluster kubernetes version increment Validate immutable fields Upgrade preflight validations passEnsuring etcd CAPI providers exist on management cluster before upgradeUpgrading core componentsPausing EKS-A cluster controller reconcilePausing Flux kustomizationGitOps field not specified, pause flux kustomization skippedCreating bootstrap clusterInstalling cluster-api providers on bootstrap clusterMoving cluster management from workload to bootstrap clusterUpgrading workload clusterMoving cluster management from bootstrap to workload clusterApplying new EKS-A cluster resource; resuming reconcileResuming EKS-A controller reconciliationUpdating Git Repo with new EKS-A cluster specGitOps field not specified, update git repo skippedForcing reconcile Git repo with latest commitGitOps not configured, force reconcile flux git repo skippedResuming Flux kustomizationGitOps field not specified, resume flux kustomization skippedWriting cluster config file🎉 Cluster upgraded!

Let's check the nodes to verify the state of the cluster.

$ kubectl get nodesNAME                                     STATUS   ROLES                  AGE   VERSIONrc-dev-multinode-7dwtd                   Ready    control-plane,master   43m   v1.22.6-eks-bb942e6rc-dev-multinode-md-0-86786b8dfb-dsfrw   Ready    <none>                 43m   v1.22.6-eks-bb942e6rc-dev-multinode-md-0-86786b8dfb-gpx2x   Ready    <none>                 43m   v1.22.6-eks-bb942e6rc-dev-multinode-md-0-86786b8dfb-stkx8   Ready    <none>                 10m   v1.22.6-eks-bb942e6rc-dev-multinode-md-0-86786b8dfb-tx6b4   Ready    <none>                 43m   v1.22.6-eks-bb942e6rc-dev-multinode-md-0-86786b8dfb-vqpf9   Ready    <none>                 10m   v1.22.6-eks-bb942e6

And sure enough, we have all 5 worker nodes in ready status.

You can scale workload clusters in a semi automatic way by storing your cluster config manifest in git and then having a CI/CD system deploy your changes. Or you can use a GitOps controller to apply the changes. EKS Anywhere allows cluster management by incorporating GitOps. It uses Flux to manage clusters with GitOps. More on this in a next post.

Adding Integration to your EKS Anywhere Cluster

Standing up a workload cluster is awesome; however as mentioned earlier, Kubernetes can quickly become operation extensive and needs the flexibility of integrating with a wide array of tools. And EKS Anywhere doesn't disappoint.

EKS Anywhere offers custom integration for certain third-party vendor components, namely Ubuntu TLS, Cilium, and Flux. It also provides flexibility for you to integrate with your choice of tools in other areas. This is really great and frees up the teams from being locked in with a certain vendor and enables them to swap out default components with tools of their choice; for instance, swapping out cilium with a different CNI such as calico.

Some of the key integration components compatible with EKS Anywhere clusters are

Load Balancer - KubeVip, MetalLB
Local container repository - Harbor
Monitoring - Prometheus , Grafana , Datadog , or NewRelic
Logging - Splunk or Fluentbit
Secret Management - Hashicorp Vault
Policy agent - Open Policy Agent (OPA)
Service mesh - Istio, Linkerd
Cost management - KubeCost
Etcd backup and restore - Valero

Cluster API Kubernetes (CAPI)

Any introduction to EKS Anywhere will not be complete without mentioning Cluster API kubernetes project.

Kubernetes Cluster API or CAPI is a Kubernetes SIG (Special Interest Group) project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. Two of the main goals of CAPI project are

To manage the lifecycle (create, scale, upgrade, destroy) of Kubernetes-conformant clusters using a declarative API.
To work in different environments, both on-premises and in the cloud.

To add some context, the notion of Bootstrapping cluster, Management cluster and Workload cluster in the context of managing kubernetes cluster lifecycle was first championed by CAPI.

EKS Anywhere works under the same principles and uses CAPI underneath to implement some of these features. It uses an infrastructure provider model for creating, upgrading, and managing Kubernetes clusters that leverages the Kubernetes Cluster API project. The first supported EKS Anywhere provider, VMware vSphere, is implemented based on the Kubernetes Cluster API Provider vSphere (CAPV) specifications. Similarly, EKS Anywhere supports Cluster API for Docker Provider (CAPD) for creating development and test workload clusters.

The EKS Anywhere project wraps Cluster API, various other CLIs and plugins (eksctl cli, anywhere plugin, kubectl, aws-iam-authenticator) and bundles them in a single package to simplify the creation of workload clusters.

Epilogue

Amazon EKS Anywhere aims to solve the pain points of managing the lifecycle of kubernetes clusters in on-premise set up and provides a consistent and reliable workflow for creating and managing kubernetes clusters across deployment models and providers. With its currently supported VMWare's vSphere provider and upcoming support for bare metal, I am keen to explore its potential and study its adoption across customers and teams.

Let me know your thoughts in the comments.

]]>