Python functions shut down slowly

[fernandezcuesta]

What happened?

I'm not fully sure where to report this, but the point is that after introducing a composition function developed with python, we noticed that:

• The latency of the function deployment to do a rollout restart is way larger than others (e.g. based on go sdk)
• The amount of time it takes for crossplane render got penalized as well.

This is negatively impacting both CI and DevEx since rendering a composition is quite a common use case.

How can we reproduce it?

I just tested this with a dummy function which does nothing:

    async def RunFunction(
        self, req: fnv1.RunFunctionRequest, _: grpc.aio.ServicerContext
    ) -> fnv1.RunFunctionResponse:
        """Run the function."""
        log = self.log.bind(tag=req.meta.tag)
        log.info("Running function and doing nothing")

        return response.to(req)

Running crossplane render on a composition that has this function as a dependency (not necessarily used in the pipeline, just declaring this as a dependency in crossplane.yaml is enough) adds an extra ~10s. A couple of experiments with simple compositions raise the amount of time for the render command from 1-2 seconds to 10-12 seconds.

I was suspecting that this could have something to do, and played by setting it to None but apparently didn't help :(

What environment did it happen in?

function-sdk-python version: 0.5.0
crank version v1.18.0

[fernandezcuesta]

Update: I verified that this doesn't happen with Crossplane <= v1.17.1, so both v1.17.2 and 1.18.0 are affected

[negz]

I can't imagine why the version of Crossplane would affect function startup/shutdown time. Given it's Python I wouldn't ever expect it to be as fast as a Go function, but 10s startup does seem like a lot.

Are your tests factoring in image pull time? xpkg.upbound.io/negz/function-auto-python:v0.1.0 and xpkg.upbound.io/crossplane-contrib/function-auto-ready:v0.3.0 do the same thing, but the Python function is roughly twice as large. They both use Distroless image bases, so I'm guessing the extra size is the Python interpreter and stdlib etc.

[fernandezcuesta]

I'm afraid not, we took times when the image is already pulled. For instance the first time (fresh, so image being pulled) takes up to 26 seconds (for a regular crossplane render). I was also puzzled about the crossplane version being relevant but that surfaced from our CI pipelines right after upgrade of 1.17.1 to 1.18.0.

Btw the issue is definitely on shutdown. With debug logs I can see the function does its job quickly, then takes around 9s to quit, so essentially preventing the output of the render command to be shown.