SMJ producing different results than HashJoin when doing a semi join

[edmondop]

Describe the bug

As a part of #10728 we have added tests for semi join, and on the first merge on main they failed (they always passed). I have ignored that tests for now but that failure suggests a bug . It only happened once in several builds, and on the AMD64 architecture

---- fuzz_cases::join_fuzz::test_semi_join_1k_filtered stdout ----
thread 'fuzz_cases::join_fuzz::test_semi_join_1k_filtered' panicked at datafusion/core/tests/fuzz_cases/join_fuzz.rs:416:13:
assertion `left == right` failed: SortMergeJoinExec and HashJoinExec produced different row counts
  left: 100
 right: 99
stack backtrace:
   0: rust_begin_unwind

see https://github.com/apache/datafusion/actions/runs/9475682375/job/26107451352

To Reproduce

https://github.com/apache/datafusion/actions/runs/9475682375/job/26107451352

Expected behavior

Should pass

Additional context

No response

[comphead]

the behavior is not deterministic, I ran tests on my local and it fails 1 run of 3, checking

[eejbyfeldt]

I think this should be reopened or we should open a separate issue. The test flaked here: https://github.com/apache/datafusion/actions/runs/9501960117/job/26188803965?pr=10898 and I was able to make it flake locally as well.

[jonahgao]

I think this should be reopened or we should open a separate issue. The test flaked here: https://github.com/apache/datafusion/actions/runs/9501960117/job/26188803965?pr=10898 and I was able to make it flake locally as well.

Reopened it. I encountered the same failure. https://github.com/apache/datafusion/actions/runs/9502397050/job/26190267777?pr=10903

[comphead]

Thanks @eejbyfeldt and @jonahgao I'll comment the test now and run more tests on top of it to see where the spontaneous bug is.

[comphead]

Filed #10904
Will run the test in the loop to find out where the issue is

[comphead]

Found that SMJ left semi sometimes have a duplicated row in the output, I'm trying to make a test case

[edmondop]

This is interesting! At least breaking the build was worth it

[comphead]

I think I'm able to reproduce it