Changing kafka.partitionLimitation seems to break consuming

[AndreasBergmeier6176]

Report

When we switch from specifying partitionLimitation (= 1) to setting it to empty ("") it seems like Keda no longer consumes messages.

Expected Behavior

Keda should switch from partition assignment to consuming all partitions again.

Actual Behavior

Keda scales to 0. We get a KEDAScaleTargetDeactivated event.

Steps to Reproduce the Problem

1. Deploy ScaledObject with partitionLimitation = 1
2. Change partitionLimitation to ""
3. Wait for Event on Deployment

Logs from KEDA operator

The only log line for our deployment foo in the operator log is this:

"2023-03-07T08:57:32Z	INFO	Initializing Scaling logic according to ScaledObject Specification	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "scaledObject": {"name":"foo","namespace":"bar"}, "namespace": "bar", "name": "foo", "reconcileID": "72abb643-49ec-4310-b34f-888bf7e489a7"}"

KEDA Version

2.8.2

Kubernetes Version

1.25

Platform

Google Cloud

Scaler Details

Kafka

Anything else?

Going from "" to 1 makes it work again though.

[JorTurFer]

Hi,
Could you confirm what KEDA version you are using? Could you also share KEDA operator logs?

[AndreasBergmeier6176]

Strangely, restarting the keda pods (operator + metrics) does not make it work again.
Since restarting does not make it "better" I fear there will be some status in a CR(D), which does not correctly get reset :(

The HPA status is:

  conditions:
  - lastTransitionTime: "2023-03-07T08:47:45Z"
    message: scaling is disabled since the replica count of the target is zero
    reason: ScalingDisabled
    status: "False"
    type: ScalingActive
  currentMetrics: null
  desiredReplicas: 0

[JorTurFer]

I need to know what KEDA version you are using please
Could you also share KEDA operator logs?

[AndreasBergmeier6176]

I need to know what KEDA version you are using

We are using 2.8.2 with the partitionLimitation patch.
Sadly we cannot upgrade to 2.9 since you broke multi-cert support with 2.9 :(

Could you also share KEDA operator logs?

Cleaning the full log will take me some time.
You could also tell me what you are looking for - might make it easier...

[JorTurFer]

We are using 2.8.2 with the partitionLimitation patch.

We don't support custom builds, sorry. There could be other changes impacting.

Sadly we cannot upgrade to 2.9 since you broke multi-cert support with 2.9 :(

What do you mean? We plan to release v2.10 soon and if something is broken and we know it, we can try to solve it for incoming release.

You could also tell me what you are looking for

I'm looking for anything related with the scaler, to check if there is something weird

[AndreasBergmeier6176]

What do you mean? We plan to release v2.10 soon and if something is broken and we know it, we can try to solve it for incoming release.

Previously we could add rolling (2+) CAs. With 2.9 this no longer works.

[JorTurFer]

Previously we could add rolling (2+) CAs. With 2.9 this no longer works.

Could you be more explicit? How do you do it? It's the first time that I have heard about this. I'd say that we didn't know about the way and maybe you aren't the only affected by this (and we need to rethink something)
Could this new feature that will be released as part of v2.10 solve your CAs issues?
PR: #4191
Docs: https://keda.sh/docs/2.10/operate/security/#register-your-own-ca-in-keda-operator-trusted-store

[AndreasBergmeier6176]

Previously we could add rolling (2+) CAs. With 2.9 this no longer works.

Could you be more explicit? How do you do it?

I will try to upgrade our dev env to 2.9 again later today. Then I can provide you with more details about this concrete config and errors...

[JorTurFer]

But just to understand your use case, you are adding some custom CAs to KEDA in order to trust in them during HTTP requests, right?
How are you adding those CAs to KEDA?

[AndreasBergmeier6176]

Tried to create a new Bug for the upgrade problem: #4330

[AndreasBergmeier6176]

Could this new feature that will be released as part of v2.10 solve your CAs issues?

Not sure - could this apply for communicating with bootstrap servers?

[JorTurFer]

Yes, all the custom CAs will be added to any place where an HTTP client is used. In this case kafka client uses it, so any custom CA will be added to it

[JorTurFer]

Is this solved by #4333?

[AndreasBergmeier6176]

Is this solved by #4333?

Will try to create a setup with main tomorrow. Probably not solved, though.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.