NestedLoopJoin performance regression

[comphead]

          This PR seems to introduce a performance regression due to the similar reasons as it was in https://github.com/apache/datafusion/pull/9830#issuecomment-2028430188 -- now `apply_batch_filter` may be executed way more times. Query used for the comparison:

EXPLAIN ANALYZE SELECT count(1) FROM nation n JOIN lineitem li ON n.n_nationkey < li.l_orderkey

and for the single partition execution I've got the following results:

-- Before:
NestedLoopJoinExec: join_type=Inner, filter=n_nationkey@0 < l_orderkey@1, metrics=[output_rows=150029850, build_input_rows=25, output_batches=733, build_input_batches=1, input_batches=733, input_rows=6001215, build_mem_used=296, join_time=1.257203344s, build_time=541.719µs]

-- After:
NestedLoopJoinExec: join_type=Inner, filter=n_nationkey@0 < l_orderkey@1, metrics=[output_rows=150029850, build_input_batches=1, input_batches=733, output_batches=733, input_rows=6001215, build_input_rows=25, build_mem_used=296, build_time=488.521µs, join_time=9.563931093s]

(join_time metric shows the issue).

Any thoughts on it?

Originally posted by @korowa in #12504 (comment)

[alihan-synnada]

I'm working on a fix right now. I'll open a PR shortly.