{"id":638,"date":"2019-08-22T19:27:59","date_gmt":"2019-08-22T19:27:59","guid":{"rendered":"https:\/\/www.peopleperhour.com\/engineering\/?p=638"},"modified":"2019-08-23T08:11:23","modified_gmt":"2019-08-23T08:11:23","slug":"kubernetes-upgrading-from-autoscaling-v2beta1-to-autoscaling-v2beta2","status":"publish","type":"post","link":"https:\/\/www.peopleperhour.com\/engineering\/2019\/08\/22\/kubernetes-upgrading-from-autoscaling-v2beta1-to-autoscaling-v2beta2\/","title":{"rendered":"Kubernetes upgrading from autoscaling\/v2beta1 to autoscaling\/v2beta2"},"content":{"rendered":"\n

We use a HorizontalPodAutoscaler<\/code> Kubernetes resource to scale Pods that work off items from our AWS SQS queues. We found the scale-up to be very aggressive and wondered whether the new version would help. I couldn’t find any documention about the syntax change in v2beta2 for object metrics<\/strong>. Since I spent more than a hour working it out from the raw spec<\/a>, I thought I would put the changes here in case it helps anyone else.<\/p>\n\n\n\n

Before (v2beta1):<\/p>\n\n\n\n

apiVersion: autoscaling\/v2beta1\n kind: HorizontalPodAutoscaler\n metadata:\n   name: pph-notifications-listener\n   namespace: default\n spec:\n   scaleTargetRef:\n     apiVersion: apps\/v1\n     kind: Deployment\n     name: pph-notifications-listener\n   minReplicas: 1\n   maxReplicas: 5\n   metrics:\n type: Object\n object:\n   metricName: redacted_qname_sqs_approximatenumberofmessages\n   target:\n     kind: Namespace\n     name: default\n   targetValue: 250 <\/pre>\n\n\n\n
After (v2beta2)<\/p>\n\n\n\n
apiVersion: autoscaling\/v2beta2\n kind: HorizontalPodAutoscaler\n metadata:\n   name: pph-notifications-listener\n   namespace: default\n spec:\n   scaleTargetRef:\n     apiVersion: apps\/v1\n     kind: Deployment\n     name: pph-notifications-listener\n   minReplicas: 1\n   maxReplicas: 5\n   metrics:\n type: Object\n object:\n   metric:\n     name: redacted_qname_sqs_approximatenumberofmessages\n   describedObject:\n     kind: Namespace\n     name: default\n   target:\n     type: Value\n     value: 250 <\/pre>\n\n\n\n
The HPA gets these metrics from our Kubernetes custom Metrics API which gets them from Prometheus which gets them via a ServiceMonitor<\/a> from sqs-exporter<\/a> which gets them from CloudWatch. Simple!<\/p>\n\n\n\n
Look how aggressive the v2beta1<\/code> scale-up is:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
See how the moment, the value goes over the target<\/code>, Pods are scaled up to the max! The problem is that, because we use EKS, which is a managed service, and the kube-controller-manager runs on a master node, we can’t change some of the key settings like --horizontal-pod-autoscaler-sync-period<\/code> or --horizontal-pod-autoscaler-downscale-stabilization<\/code> (ref<\/a>).<\/p>\n\n\n\n
Update<\/em>: Unfortunately upgrading to v2beta2<\/code> didn’t help with our aggressive scale-up problem:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Figuring out this issue is important because it causes a surge in resource requests which causes our cluster size to grow and shrink needlessly and causes more Pod churn than necessary which makes observability harder and generates more logs than necessary. <\/p>\n","protected":false},"excerpt":{"rendered":"
We use a HorizontalPodAutoscaler Kubernetes resource to scale Pods that work off items from our AWS SQS queues. We found the scale-up to be very aggressive and wondered whether the new version would help. I couldn’t find any documention about the syntax change in v2beta2…<\/p>\n","protected":false},"author":40,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[23],"tags":[44],"class_list":["post-638","post","type-post","status-publish","format-standard","hentry","category-devops-2","tag-kubernetes"],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p2CA4w-ai","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.peopleperhour.com\/engineering\/wp-json\/wp\/v2\/posts\/638","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.peopleperhour.com\/engineering\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.peopleperhour.com\/engineering\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.peopleperhour.com\/engineering\/wp-json\/wp\/v2\/users\/40"}],"replies":[{"embeddable":true,"href":"https:\/\/www.peopleperhour.com\/engineering\/wp-json\/wp\/v2\/comments?post=638"}],"version-history":[{"count":3,"href":"https:\/\/www.peopleperhour.com\/engineering\/wp-json\/wp\/v2\/posts\/638\/revisions"}],"predecessor-version":[{"id":643,"href":"https:\/\/www.peopleperhour.com\/engineering\/wp-json\/wp\/v2\/posts\/638\/revisions\/643"}],"wp:attachment":[{"href":"https:\/\/www.peopleperhour.com\/engineering\/wp-json\/wp\/v2\/media?parent=638"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.peopleperhour.com\/engineering\/wp-json\/wp\/v2\/categories?post=638"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.peopleperhour.com\/engineering\/wp-json\/wp\/v2\/tags?post=638"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}