How to use KEDA

Kubernetes-based Event Driven Autoscaler with Kafka

KEDA is a single-purpose and lightweight component that can be added into any Kubernetes cluster. KEDA works alongside standard Kubernetes components like the Horizontal Pod Autoscaler and can extend functionality without overwriting or duplication. With KEDA you can explicitly map the apps you want to use event-driven scale, with other apps continuing to function. This makes KEDA a flexible and safe option to run alongside any number of any other Kubernetes applications or frameworks.

KEDA는 Kubernetes-based Event Driven Autoscaler로 HPA(Horizontal Pod Autoscaling)와 Pod Autoscaling을 위해 사용되며, 외부 Trigger Source에 의해 Event-Driven하게 동작한다.

https://keda.sh/docs/2.8/concepts/

---

External Trigger Sources

External Trigger Source로 활용할 수 있는 metric은 아래와 같다.

---

Tutorial

1.AWS EKS 클러스터에 KEDA 및 Ingress controller 설치

💡 NOTE: KEDA requires Kubernetes cluster version 1.16 and higher

https://keda.sh/docs/2.8/deploy/#install

KEDA | Deploying KEDA

$ helm repo add kedacore https://kedacore.github.io/charts
$ helm repo update
$ helm install keda kedacore/keda --namespace keda
$ helm list -nkeda
NAME    NAMESPACE   REVISION    UPDATED                                 STATUS      CHART       APP VERSION
keda    keda        1           2022-10-05 14:01:15.853472 +0900 KST    deployed    keda-2.8.2  2.8.1

 

2. Kafka 설치

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm repo update
$ helm install kafka bitnami/kafka

 

3. Kafka CLI 설치

$ kubectl run -i --tty kafka-cli --image=conitasgmbh/kafka-cli --restart=Never --rm -ndev -- sh

 

4. Kafka Topic 생성

$ bin/kafka-topics.sh --bootstrap-server kafka-0.kafka-headless.dev.svc.cluster.local:9092 --alter --topic issue_coupon -partitions 4

Partition등 변경이 필요한경우

$ bin/kafka-topics.sh --bootstrap-server kafka-0.kafka-headless.dev.svc.cluster.local:9092 --alter --topic issue_coupon -partitions 4

 

5. Kafka Queue 추가

$ bin/kafka-console-producer.sh --topic issue_coupon --broker-list kafka-0.kafka-headless.dev.svc.cluster.local:9092

 

6. Topic 확인

$ bin/kafka-topics.sh --bootstrap-server=kafka-0.kafka-headless.dev.svc.cluster.local:9092 --describe --topic issue_coupon

Topic: issue_coupon	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Topic: issue_coupon	Partition: 1	Leader: 0	Replicas: 0	Isr: 0
Topic: issue_coupon	Partition: 2	Leader: 0	Replicas: 0	Isr: 0
Topic: issue_coupon	Partition: 3	Leader: 0	Replicas: 0	Isr: 0

 

7. KEDA ScaledObject 생성

By default, the number of replicas will not exceed the number of partitions on a topic and will not exceed maxReplicaCount. That is, if maxReplicaCount is set more than number of partitions, the scaler won’t scale up to target maxReplicaCount. See allowIdleConsumers below to disable this default behavior.This is so because if there are more number of consumers than the number of partitions in a topic, then extra consumer will have to sit idle.

기본적으로 replicas 수는 topic의 partition과, maxReplicaCount를 넘을 수 없다. 그 이유는 consumer가 partition갯수보다 많게되면 유휴 consumer가 생길 수 있다. allowIdleConsumers 옵션(=true)을 통해 허용할 수 있다. 기본값은 false이다.

kafka queue기반의 Trigger를 적용하여 Scale out을 위한 Threshold 등 지정 (테스트를 위해 LagThreshold를 작게 지정)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: coupon-api
spec:
  scaleTargetRef:
    apiVersion:    apps/v1                    # Optional. Default: apps/v1
    kind:          Deployment                 # Optional. Default: Deployment
    name:          coupon-api                 # Mandatory. Must be in the same namespace as the ScaledObject
  pollingInterval:  30                        # Optional. Default: 30 seconds
  cooldownPeriod:   300                       # Optional. Default: 300 seconds
  idleReplicaCount: 0                         # Optional. Default: ignored, must be less than minReplicaCount
  minReplicaCount:  1                         # Optional. Default: 0
  maxReplicaCount:  5                         # Optional. Default: 100
  fallback:                                   # Optional. Section to specify fallback options
    failureThreshold: 3                       # Mandatory if fallback section is included
    replicas: 6                               # Mandatory if fallback section is included
  advanced:                                   # Optional. Section to specify advanced options
    restoreToOriginalReplicaCount: false      # Optional. Default: false
    horizontalPodAutoscalerConfig:            # Optional. Section to specify HPA related options
      behavior:                               # Optional. Use to modify HPA's scaling behavior
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
          - type: Percent
            value: 100
            periodSeconds: 15
  triggers:
	- type: kafka
	  metadata:
	    bootstrapServers: kafka.svc:9092
	    consumerGroup: my-group
	    topic: test-topic
	    lagThreshold: '5'             # Optional. Default: 5, Average target value to trigger scaling actions.
	    offsetResetPolicy: latest     # Optional. Default: latest, The offset reset policy for the consumer. (Values: latest, earliest)
	    allowIdleConsumers: false     # Optional. Default: false,  When set to true, the number of replicas can exceed the number of partitions on a topic, allowing for idle consumers.
	    version: 1.0.0                # Optional. Default: 1.0.0, Version of your Kafka brokers. See samara version

Parameter list:

• bootstrapServers - Comma separated list of Kafka brokers “hostname:port” to connect to for bootstrap.
• consumerGroup - Name of the consumer group used for checking the offset on the topic and processing the related lag.
• topic - Name of the topic on which processing the offset lag.
• lagThreshold - Average target value to trigger scaling actions. (Default: 5, Optional)
• offsetResetPolicy - The offset reset policy for the consumer. (Values: latest, earliest, Default: latest, Optional)
• allowIdleConsumers - When set to true, the number of replicas can exceed the number of partitions on a topic, allowing for idle consumers. (Default: false, Optional)
• version - Version of your Kafka brokers. See samara version (Default: 1.0.0, Optional)

 

8. Kafka Queue 발송

$ bin/kafka-console-producer.sh --topic issue_coupon --broker-list kafka-0.kafka-headless.dev.svc.cluster.local:9092
>{"id":1}
>{"id":2}
>{"id":3}
>{"id":4}
>{"id":5}
.
.
.

 

9. Kafka Threadholder 확인

Producer를 통해 Lag가 발생할때까지 대량의 Queue를 생성한다.

Before Scaleout

$ bin/kafka-consumer-groups.sh --bootstrap-server kafka-0.kafka-headless.dev.svc.cluster.local:9092 --group coupon --describe

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
coupon          issue_coupon    0          444             469             25              -               -               -
coupon          issue_coupon    1          183             183             0               -               -               -
coupon          issue_coupon    2          108             123             15              -               -               -
coupon          issue_coupon    3          182             195             13              -               -               -

$ kubectl get po -ndev
NAME                          READY   STATUS    RESTARTS   AGE
coupon-api-c46685c5-9lw6b     1/1     Running   0          31s

 

After Scaleout

Consumer에서의 메세지 처리속도보다 Producer에서 메세지 발생량이 많을 경우, Lag가 쌓여 LagThreshold에 도달하게 되고, KEDA Trigger에 의해 Scale-out이 발생하여 각 파티션에 쌓인 Lag를 병렬로 Consuming하여 빠르게 처리할 수 있다. 또한 Rebalancing을 통해 Consumer에 Partition을 적절히 할당한다.

$ bin/kafka-consumer-groups.sh --bootstrap-server kafka-0.kafka-headless.dev.svc.cluster.local:9092 --group coupon --describe

GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                            HOST            CLIENT-ID
coupon          issue_coupon    2          123             123             0               consumer-coupon-1-7d2b429d-2549-46f4-89f6-70a8598202e0 /192.168.24.230 consumer-coupon-1
coupon          issue_coupon    1          183             183             0               consumer-coupon-1-4d959ae2-a78b-4a50-b1ad-a72800b7a151 /192.168.64.135 consumer-coupon-1
coupon          issue_coupon    3          195             195             0               consumer-coupon-1-f912ede5-98fe-42e1-aca4-1962dfa47c1c /192.168.15.115 consumer-coupon-1
coupon          issue_coupon    0          469             469             0               consumer-coupon-1-4565526c-a088-40bd-8dbe-b1056084d37b /192.168.54.193 consumer-coupon-1

$ kubectl get po -ndev
NAME                          READY   STATUS    RESTARTS   AGE
coupon-api-c46685c5-9lw6b     0/1     Running   0          23s
coupon-api-c46685c5-jb8fz     0/1     Running   0          19s
coupon-api-c46685c5-jvtd6     0/1     Running   0          19s
coupon-api-c46685c5-kxzwc     0/1     Running   0          19s

---

Demo