Autoscale AKS cluster with Cluster Autoscaler (CA) using VMSS

Set environment variables.

$ export AKS_REGION=southeastasia
$ export AKS_CLUSTER_RG=vmssResourceGroup
$ export AKS_CLUSTER_NAME=vmssAKSCluster

# First create a resource group
$ az group create --name $AKS_CLUSTER_RG --location $AKS_REGION

Create an AKS cluster + VMSSPreview and enable the cluster autoscaler [1]

# Now create the AKS cluster and enable the cluster autoscaler
$ az aks create \
  --resource-group $AKS_CLUSTER_RG \
  --name $AKS_CLUSTER_NAME \
  --node-count 1 \
  --enable-vmss \
  --enable-cluster-autoscaler \
  --min-count 1 \
  --max-count 3 \
  --kubernetes-version 1.13.5

$ az aks get-credentials --resource-group $AKS_CLUSTER_RG --name $AKS_CLUSTER_NAME

## check actual state of the CA and the AKS cluster.
#
## CA creates a Kubernetes configMap object to report the actual state of the CA and the AKS cluster.
$ kubectl -n kube-system describe configmap cluster-autoscaler-status

Enable the cluster autoscaler on an existing AKS cluster

$ az aks update \
  --resource-group $AKS_CLUSTER_RG \
  --name $AKS_CLUSTER_NAME \
  --enable-cluster-autoscaler \
  --min-count 1 \
  --max-count 3

Change the cluster autoscaler settings

$ az aks update \
  --resource-group $AKS_CLUSTER_RG \
  --name $AKS_CLUSTER_NAME \
  --update-cluster-autoscaler \
  --min-count 1 \
  --max-count 5

Disable the cluster autoscaler

$ az aks update \
  --resource-group $AKS_CLUSTER_RG \
  --name $AKS_CLUSTER_NAME \
  --disable-cluster-autoscaler

# You can manually scale your cluster using the `az aks scale` command.
$ az aks scale \
  --resource-group $AKS_CLUSTER_RG \
  --name $AKS_CLUSTER_NAME \
  --nodepool-name nodepool1 \
  --node-count 2

Configure cluster autoscaler startup parameters [2][3]

For AKS clusters using the standard configuration with availability sets, you are responsible for deploying, configuring and operating the cluster autoscaler. Run following script to create ca-secret.yaml. [2]

#! /bin/bash
# This is also the way to obtain information of specific service principal.
ID=$(az account show --query id)
SUBSCRIPTION_ID=$(echo -n $ID | tr -d '"')
TENANT=$(az account show --query tenantId)
TENANT_ID=$(echo -n $TENANT | tr -d '"' | base64)
read -p "What is your AKS cluster name? " AKS_CLUSTER_NAME
read -p "What is the AKS cluster resource group name? " AKS_RESOURCE_GROUP
CLUSTER_NAME=$(echo -n $AKS_CLUSTER_NAME | base64)
RESOURCE_GROUP=$(echo -n $AKS_RESOURCE_GROUP | base64)
PERMISSIONS=$(az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/$SUBSCRIPTION_ID")
CLIENT_ID=$(echo $PERMISSIONS | jq .appId | tr -d '"','\n' | base64)
CLIENT_SECRET=$(echo $PERMISSIONS | jq .password | tr -d '"','\n' | base64)
SUBSCRIPTION_ID=$(echo -n $ID | tr -d '"' | base64)
NODE_RESOURCE_GROUP=$(az aks show --name $AKS_CLUSTER_NAME  --resource-group $AKS_RESOURCE_GROUP -o tsv --query 'nodeResourceGroup' | tr -d '\n'  | base64)
echo "---
apiVersion: v1
kind: Secret
metadata:
    name: cluster-autoscaler-azure
    namespace: kube-system
data:
    ClientID: $CLIENT_ID
    ClientSecret: $CLIENT_SECRET
    ResourceGroup: $RESOURCE_GROUP
    SubscriptionID: $SUBSCRIPTION_ID
    TenantID: $TENANT_ID
    VMType: YWtz
    ClusterName: $CLUSTER_NAME
    NodeResourceGroup: $NODE_RESOURCE_GROUP
---" > ca-secret.yaml

## create CA secret. (CA is Cluster Autoscaler)
$ kubectl apply -f ca-secret.yaml

Before we continue with the CA deployment, we need some adjustments in the ca-deployment.yaml file.

1. Adjust the cluster autoscaler container image tag to match the required version of the cluster autoscaler for our Kubernetes version we are using in our AKS cluster.

   Change from

   - image: k8s.gcr.io/cluster-autoscaler:v1.12.3

   to

   - image: k8s.gcr.io/cluster-autoscaler:v1.13.5

2. Adjust the agent pool name to match the one of our AKS cluster and the CA range for min and max nodes.

   get pool name using following command

   $ az aks show --name $AKS_CLUSTER_NAME --resource-group $AKS_CLUSTER_RG -o tsv --query 'agentPoolProfiles[].name'

   change from

   - --nodes=3:6:agentpool

   to

   - --nodes=1:3:nodepool1

## create deployment 'cluster-autoscaler'.
$ kubectl apply -f ca-deployment.yaml

## check actual state of the CA and the AKS cluster.
$ kubectl -n kube-system describe configmap cluster-autoscaler-status

Testing the AKS Cluster Autoscaler [4]

AKS have two types of scaling.

1. Cluster Auto Scaler – This will auto scale AKS cluster worker nodes based on the resource utilization
2. Horizontal Pod Auto Scaler – This will autoscale pods inside the Kubernetes cluster using ReplicaS

# create ReplicaSet
$ cat <<EOF|kubectl apply -f -
apiVersion: apps/v1beta2
kind: ReplicaSet
metadata:
  name: kubia-rs
spec:
  replicas: 3
  selector:
    matchLabels:
      app: kubia
  template:
    metadata:
      labels:
        app: kubia
    spec:
      containers:
        - image:  kasunsjc/kubia
          name:  kubia
          ports:
          - containerPort: 8080
            protocol: TCP
EOF

# scale ReplicaSet
$ kubectl scale --replicas=200 rs/kubia-rs

# monitor status
$ watch -d -n 2 "kubectl get rs,rc,deploy"
$ watch -d -n 2 "kubectl get nodes"

Reference

[1] Automatically scale a cluster to meet application demands on Azure Kubernetes Service (AKS), https://docs.microsoft.com/en-us/azure/aks/cluster-autoscaler [2] Azure Kubernetes Service cluster autoscaler configurations, https://www.danielstechblog.io/azure-kubernetes-service-cluster-autoscaler-configurations/ [3] (GOOD)Cluster Autoscaler on Azure, https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/azure/README.md [4] Azure Kubernetes Service (AKS) Cluster Autoscaler, http://www.allaboutwindowssl.com/2019/03/azure-kubernetes-service-aks-cluster-autoscaler/ [5] Create and manage multiple node pools for a cluster in Azure Kubernetes Service (AKS), https://docs.microsoft.com/en-us/azure/aks/use-multiple-node-pools [6] az aks nodepool, https://docs.microsoft.com/en-us/cli/azure/ext/aks-preview/aks/nodepool?view=azure-cli-latest