[VL] Results mismatch with vanilla spark when using window exec

[NEUpanning]

Backend

VL (Velox)

Bug description

SQL:

SELECT
            t.partner_id,
            t.qualifi_name as name,
            t.qualifi_type as type_id,
            row_number() over(partition by partner_id,qualifi_type order by modify_time,create_time,end_date desc) row_id
        FROM
            tbl
        WHERE partition_date = '2024-08-02'
            and status = 1
            AND qualifi_type in (201,202,1);

gluten results that mismatch with vanilla:

714328  临海市嘴爱烘焙坊        1       1
714328  徐新亮  1       2

vanilla results:

714328  徐新亮  1       1
714328  临海市嘴爱烘焙坊        1       2

the original rows of mismatch results

 partner_id | qualifi_type |   qualifi_name   |     modify_time     |     create_time     |      end_date
------------+--------------+------------------+---------------------+---------------------+---------------------
     714328 |            1 | 徐新亮           | 2019-06-08 05:53:49 | 2019-06-08 05:53:49 | 2099-01-01 00:00:00
     714328 |            1 | 临海市嘴爱烘焙坊   | 2019-06-08 05:53:49 | 2019-06-08 05:53:49 | 2099-01-01 00:00:00

It seems that row_number() over(...) produced different result for the same order of rows (the columns used to sort the two rows are equal).
Here is gluten physical plan:

gluten version : 1.2-rc

Spark version

3.0

[NEUpanning]

cc @kecookier

[kecookier]

When we get duplicate rows, each row may end up with a value of 1 after row_number() is applied. This may not be an issue.

[PHILO-HE]

Looks velox sort doesn't respect the input order when some rows have same values for sort keys. Agreed with @kecookier, maybe we can ignore this issue.

[NEUpanning]

This inconsistent behavior seems acceptable, and vanilla Spark sort is not deterministic either. Therefore, I will close this issue. Thanks for your help. @kecookier @PHILO-HE