Kubernetes Autoscaling with Custom Metrics


One of the most amazing things about Kubernetes is its ability to automatically scale up and down. In current (autoscaling/v1) releases, the only metric which could be used as a scaling target is CPU usage, as measured by heapster.

With the introduction of the custom metrics api, deployments will be able to scale using one or more application-specific metrics, as well as CPU.


I spent way too much time trying to figure out how to do this today. These are my findings, but I could not get it to work. I look forward to being able to use this in the future when there are more examples to follow and when it is not hidden behind feature flags.

Exposing Metrics

A few months ago, the metrics repo was created for the purpose of storing type definitions and client code for publishing and gathering metrics.

There was not a clear path forward from there, but luckily I discovered that a boilerplate and a Prometheus adapter have been created.

The boilerplate defines a CustomMetricsProvider interface here, which looks like the following:

type CustomMetricsProvider interface {

	GetRootScopedMetricByName(groupResource schema.GroupResource, name string, metricName string) (*custom_metrics.MetricValue, error)

	GetRootScopedMetricBySelector(groupResource schema.GroupResource, selector labels.Selector, metricName string) (*custom_metrics.MetricValueList, error)

	GetNamespacedMetricByName(groupResource schema.GroupResource, namespace string, name string, metricName string) (*custom_metrics.MetricValue, error)

	GetNamespacedMetricBySelector(groupResource schema.GroupResource, namespace string, selector labels.Selector, metricName string) (*custom_metrics.MetricValueList, error)

	ListAllMetrics() []MetricInfo

This interface is the core of what is necessary to create a custom metrics server.

The boilerplate project inherits from k8s.io/apiserver, which creates APIs that are registered with the API registration API (because, of course that’s a thing).

Fun with Flags

The Horizontal Pod Autoscaler documentation says that to enable this feature, two flags must be enabled while deploying the cluster. These flags are --horizontal-pod-autoscaler-use-rest-clients and --apiserver. I assume that in future k8s releases this will not be necessary, as it will become standard.

The first flag switches the autoscaling controller to use a REST client, rather than a client that calls heapster directly.

The second flag enables kube-aggregator, which provides the previously mentioned API registration API.

Ship It

From what I can tell, this is what the HorizontalPodAutoscaler configuration will look like.

apiVersion: autoscaling/v2alpha1
kind: HorizontalPodAutoscaler
  name: my-deployment
  namespace: default
    apiVersion: apps/v1beta1
    kind: Deployment
    name: my-deployment
  minReplicas: 1
  maxReplicas: 10
  - type: Object
        kind: Service
        name: name-of-metrics-app
      metricName: my-custom-metric
      targetValue: 9001