Negative coordinate on a vc4 cursor plane is causing giant cursor to be displayed

[schauveau]

I discovered that strange behavior while trying to enable 2 monitors for RPi4 in wlroot (swaywm/wlroots#1943). When a vc4 drm cursor plane is set to a negative position larger than its size (so it should be completely invisible) then the giant version appears on screen. Negative coordinates are unusual but not illegal.

Here is a video showing the effect https://youtu.be/58-KTXcXwoE

In most clients, the cursor plane is automatically disabled when it leaves the screen so the best way to reproduce the problem is probably to edit the sources of an existing X11 or Wayland compositor by adding a negative offset to the cursor plane coordinates. The video was obtained with sway/wlroot by subtracting 100 from conn->cursor_x at https://github.com/swaywm/wlroots/blob/751a21d94f1b4f0345d040ddfd54b723631d5991/backend/drm/atomic.c#L234

[schauveau]

In forgot to indicate that I am currently on ubuntu 20.04

 Linux ubuntu 5.4.0-1014-raspi #14-Ubuntu SMP Sat Jul 4 10:25:33 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux

[6by9]

Yup, something goes funny in the cropping.

modetest -M vc4 -P 38@87:640x480-1000-1000

The kernel is logging

[ 3321.936785] [drm:vc4_plane_atomic_check [vc4]] [PLANE:38:plane-1] plane update 640x480@44 +dst(-1000,-1000, 640,480) +src(0,0, 41943040,31457280) 0xdf700000/00000000/00000000/2560, alpha 65535 zpos 1

which is as I'd expect (source rect is *65536).
The firmware is then listing that resource as

display:2 format:XRGB8888 transform:0 layer:1 640x480 src:1000,1000,-360,-520 dst:0,0,-360,-520 cost:1186 lbm:10240 

so something has gone very odd in the translation. I think the function to clamp negative offsets doesn't deal with totally offscreen elements well. It's tried to scale down the source rectangle so that the dest rectangle has positive offsets, but has failed to notice that it then leaves both rectangles with daft values in.

An offscreen element obviously doesn't want to get passed to the composition pipeline because it's going to do nothing, but if I kill it at the KMS level then I make the situation in resolving the vsync callback issue. Such fun.

[6by9]

OK, found the cause, and it wasn't as hideous as I thought.

Firmware patch posted internally for review, and it'll get released in due course.

[popcornmix]

rpi-update firmware should have a fix for this