Kubernetes upgrading from autoscaling/v2beta1 to autoscaling/v2beta2By Tom In DevOps
We use a
HorizontalPodAutoscaler Kubernetes resource to scale Pods that work off items from our AWS SQS queues. We found the scale-up to be very aggressive and wondered whether the new version would help. I couldn’t find any documention about the syntax change in v2beta2 for object metrics. Since I spent more than a hour working it out from the raw spec, I thought I would put the changes here in case it helps anyone else.
apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: pph-notifications-listener namespace: default spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: pph-notifications-listener minReplicas: 1 maxReplicas: 5 metrics: type: Object object: metricName: redacted_qname_sqs_approximatenumberofmessages target: kind: Namespace name: default targetValue: 250
apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: pph-notifications-listener namespace: default spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: pph-notifications-listener minReplicas: 1 maxReplicas: 5 metrics: type: Object object: metric: name: redacted_qname_sqs_approximatenumberofmessages describedObject: kind: Namespace name: default target: type: Value value: 250
The HPA gets these metrics from our Kubernetes custom Metrics API which gets them from Prometheus which gets them via a ServiceMonitor from sqs-exporter which gets them from CloudWatch. Simple!
Look how aggressive the
v2beta1 scale-up is:
See how the moment, the value goes over the
target, Pods are scaled up to the max! The problem is that, because we use EKS, which is a managed service, and the kube-controller-manager runs on a master node, we can’t change some of the key settings like
Update: Unfortunately upgrading to
v2beta2 didn’t help with our aggressive scale-up problem:
Figuring out this issue is important because it causes a surge in resource requests which causes our cluster size to grow and shrink needlessly and causes more Pod churn than necessary which makes observability harder and generates more logs than necessary.