Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pod of ScaledJob with Rabbitmq as Scaler lives during 12 minutes after got status "completed" #1211

Closed
alena-ait opened this issue Oct 2, 2020 · 2 comments
Labels
bug Something isn't working

Comments

@alena-ait
Copy link

alena-ait commented Oct 2, 2020

## Expected Behavior
We expected that pod will be deleted after it gets status completed and job is deleted(according to parameter successfulJobsHistoryLimit). Job was deleted but its pod still present during 12 minutes in status completed

## Actual Behavior
Pod gets status "completed" , job get status "completed" and is deleted but its pod still present during 12 minutes in status completed
we use this parameters
pollingInterval: 30
cooldownPeriod: 30
So it should not be 12 minutes

Other problem - rabbitmq trigger , if we have several jobs for the same queue - all pods will be deleted just when the last pod of last job will have completed age 12 minutes.
For example
pod-1 0/1 Completed 0 21m
pod-2 0/1 Completed 0 20m
pod-3 0/1 Completed 0 18m
pod-4 0/1 Completed 0 14m
pod-5 0/1 Completed 0 11m
when pod-5 has age 12m all pods will be destroyed
It looks like KEDA checks rabbitmq queue during 12 minutes(why not 30 seconds?) and if it is empty deletes all pods, if queue will not be empty all these pods will be present in kubernetes forever

Our config

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: job
spec:
  jobTargetRef:
    parallelism: 1 
    completions: 1
    backoffLimit: 2
    template:
      spec:    
        containers:
        **something that can handle queue**
  pollingInterval: 30  
  cooldownPeriod:  30
  minReplicaCount: 0   
  maxReplicaCount: 100
  successfulJobsHistoryLimit: 0             
  failedJobsHistoryLimit: 0
  advanced:                                          
    horizontalPodAutoscalerConfig:                   
      resourceMetrics:                               
      - name: cpu                                   # Name of the metric to scale on
        target:
          type: utilization
          averageUtilization: 50                     
      - name: memory                                 
        target:
          type: utilization                     
          averageUtilization: 50                     
  triggers:
  - type: rabbitmq
    metadata:
      protocol: amqp
      queueName: long
      queueLength: "1"
    authenticationRef:
      name: keda-trigger-auth-rabbitmq-conn

Steps to Reproduce the Problem

  1. Install Keda via Helm
  2. Create Scaled Job with similar config
  3. Create trigger authentication to get connect to your rabbitmq
  4. Publish any message to rabbitmq and watch behavior of pod

Specifications

  • KEDA Version: 2.0.0-beta1.2
  • Platform & Version: *Linux *
  • Kubernetes Version: EKS v.1.17
  • Scaler(s): RabbitMQ
@alena-ait alena-ait added the bug Something isn't working label Oct 2, 2020
@TsuyoshiUshio
Copy link
Contributor

Thank you for sharing, @alena-ait It was a bug. I solved it on the PR

@tomkerkhove
Copy link
Member

In that case, can we close this issue thanks to #1214?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants