Add 'poll' function to custreamz kafka consumer

[jdye64]

Description

Streamz has updated their codebase to include a call to the Confluent Kafka Consumer library function 'poll'. Currently custreamz does not include this method. This PR adds the 'poll' function to custreamz to simply proxy the call to the underlying confluent kafka library so that streamz is no longer broken for end users. Without this function end users are no longer able to use custreamz with newer versions of the streamz library.

This closes: #13600

[bdice]

Does this need to target 23.08 or 23.10? Breakage with new versions of streamz sounds like a candidate for 23.08 despite code freeze.

[jdye64]

23.10 is fine. Caught it early and the version of streamz with the breaking change hasn’t even been released yet

[jdye64]

@bdice I don't have the permissions to alter the labels, hence Label Checker CI failure. Can you add those for me?

[vyasr]

Added labels. I assumed that this could be treated as a new feature even though technically we would be broken by the new version of the library.

[chinmaychandak]

Hey guys, would it be possible to merge this PR some time this week? This is blocking our cuStreamz dev work.

[bdice]

@jdye64 Are there dependency changes needed to support this? Are these pinnings still accurate or do they need to be updated?

cudf/dependencies.yaml

Lines 496 to 508 in 989c411

	run_custreamz:
	common:
	- output_types: conda
	packages:
	- python-confluent-kafka>=1.9.0,<1.10.0a0
	- output_types: [conda, requirements, pyproject]
	packages:
	- streamz
	- output_types: [requirements, pyproject]
	packages:
	- confluent-kafka>=1.9.0,<1.10.0a0
	- *cudf
	- cudf_kafka==23.10.*

cudf/conda/recipes/custreamz/meta.yaml

Line 51 in 989c411

- python-confluent-kafka >=1.9.0,<1.10.0a0

[jdye64]

@bdice the

@jdye64 Are there dependency changes needed to support this? Are these pinnings still accurate or do they need to be updated?

cudf/dependencies.yaml

Lines 496 to 508 in 989c411

	run_custreamz:
	common:
	- output_types: conda
	packages:
	- python-confluent-kafka>=1.9.0,<1.10.0a0
	- output_types: [conda, requirements, pyproject]
	packages:
	- streamz
	- output_types: [requirements, pyproject]
	packages:
	- confluent-kafka>=1.9.0,<1.10.0a0
	- *cudf
	- cudf_kafka==23.10.*

cudf/conda/recipes/custreamz/meta.yaml

Line 51 in 989c411

- python-confluent-kafka >=1.9.0,<1.10.0a0

No, the dependencies are still fine as is. We didn't add a new function but rather just added a hook to call one that had always existed.

[bdice]

Looks fine to me. @jdye64 Do we need to add tests for this?

Do you have merge privileges? Do you want me to merge after CI passes?

[jdye64]

@bdice tests would be nice but we decided awhile back it wasn't worth the significant CI burden of creating a kafka cluster during each PR, took almost 45 minutes to install, setup, test, and breakdown when we did it back a year or so ago. Ultimately deciding that this library is well tested upstream and we were not gaining anything by testing it again.

I do not have merge permissions. Would be great if you could merge once CI passes. Thanks!

[bdice]

/merge