Kubernetes HPA using metrics from another deployment

user_9686792 0
Viewed 381

Im currently trying to run an autoscaling demo using prometheus and the prometheus adapter, and i was wondering if there is a way to autoscale one of my deployments based on metrics that prometheus scrapes from another deployment.

What i have right now are 2 different deployments, kafka-consumer-application (which i want to scale) and kafka-exporter (which exposes the kafka metrics that I'll be using for scaling). I know that if I have both of them as containers in the same deployment the autoscaling works, but the issue is that the kafka-exporter also gets autoscaled and its not ideal, so i want to separate them. I tried with the following HPA but i could not get it to work:

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
  name: consumer-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: kafka-consumer-application
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: object
    object:
      target: kafka-exporter
      metricName: "kafka_consumergroup_lag"
      targetValue: 5

Im not sure if im doing something wrong or if this is just not something i can do, so any advice is appreciated.

Thanks!

Note: im running the adapter with this config:

rules:
  default: false
  resource: {}
  custom:
    - seriesQuery: 'kafka_consumergroup_lag'
      resources:
        overrides:
          kubernetes_namespace: {resource: "namespace"}
          kubernetes_pod_name: {resource: "pod"}
      name:
       matches: "kafka_consumergroup_lag"
       as: "kafka_consumergroup_lag"
      metricsQuery: 'avg_over_time(kafka_consumergroup_lag{topic="my-topic",consumergroup="we-consume"}[1m])'
``
autoscaling prometheus kubernetes
1 Answers

In kubernetes documentation you can read:

Autoscaling on metrics not related to Kubernetes objects Applications running on Kubernetes may need to autoscale based on metrics that don’t have an obvious relationship to any object in the Kubernetes cluster, such as metrics describing a hosted service with no direct correlation to Kubernetes namespaces. In Kubernetes 1.10 and later, you can address this use case with external metrics

So using external metrics, your HPA yaml could look like following:

kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta2
metadata:
  name: consumer-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: kafka-consumer-application
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: kafka_consumergroup_lag
        #selector:
        #  matchLabels:
        #    topic: "my-topic"
      target:
        type: AverageValue
        averageValue: 5

If you have more than one kafka-exporter you can use selector to filter it (source):

selector is the string-encoded form of a standard kubernetes label selector for the given metric When set, it is passed as an additional parameter to the metrics server for more specific metrics scoping. When unset, just the metricName will be used to gather metrics

Also have a look at this Stack question.

user_122010840
Related