distinct hash aggregate returned duplicated value if spill happens

[FelixYBW]

Bug description

If there is a spill the returned values from distinct hash aggregate have duplicates. It's not yet fixed. The root cause is related to the spill files.

    if (stream->id() == 0) {
      newDistinct = false;
    }

code`

It assums all the records before first spill trigger are spilled single file but for some reason the records are written into 2 files which leads to being output again when handling spill file.

System information

Velox System Info v0.0.2
Commit: b4ce48390e055cdf307ba378b9dd305ce77027ec
CMake Version: 3.22.1
System: Linux-5.15.0-101-generic
Arch: x86_64
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 11.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 11.4.0
CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt

\nThe results will be copied to your clipboard if xclip is installed.

Relevant logs

No response

[mbasmanova]

CC: @xiaoxmeng

[xiaoxmeng]

@FelixYBW Thanks for catching this!

[FelixYBW]

Confirmed, as long as the initial spill has more than 1 split file, may be caused by maxSpillRunRows or maxSpillFileSize, the bug happens.
Planed solution is to record the initial spill file id and used it to exclude the data which is already output. We will submit a PR later.

[FelixYBW]

Confirmed, as long as the initial spill has more than 1 split file, may be caused by maxSpillRunRows or maxSpillFileSize, the bug happens. Planed solution is to record the initial spill file id and used it to exclude the data which is already output. We will submit a PR later.

@zhztheplayer

[xiaoxmeng]

Here is the PR fix this issue: #9230. This is caused by maxSpillRun which might split the large sorted spill run to small ones for batch. This is added after distinct aggregation spill support.

[xiaoxmeng]

Confirmed, as long as the initial spill has more than 1 split file, may be caused by maxSpillRunRows or maxSpillFileSize, the bug happens. Planed solution is to record the initial spill file id and used it to exclude the data which is already output. We will submit a PR later.

@FelixYBW thanks and already have a fix for this issue.

[FelixYBW]

Thank you so much for the quick fix!