FutureProducer's send has been observed to cause memory leak

[RustyKozak]

I have a service running on ECS that's based on rust:1.80.0-bookworm docker image. its main job is to take data from an http endpoint and proxy it to a specific kafka topic.

The issue was observed after migrating to Amazon's MSK and trying to handle roughly 600 requests a minute. In order to authenticate with MSK i have to leverage aws-msk-iam-sasl-signer crate to create a custom FutureProducer context.

Additionally, here are the feature flags for the rdkafka i'm using

rdkafka = { version = "0.36.2", features = ["cmake-build", "sasl", "ssl"] }

After sifting through my service's code I was able to narrow down the memory leak issue specifically to FutureProducer.send function. I switched it out for send_result and noticed the memory leak has went away.

In the attached image the memory usage you see as orange is ever growing memory consumption using the send function vs the stable blue line using the send_result.

I'm running 3 ECS tasks to rule out any bias and show that it doesn't happen at random.