Persist intermediate data to avoid non-determinism caused by Spark lazy random evaluation

[simonzhaoms]

Description

The bug is reported by Bhrigu. Due to Spark lazy evaluation, random filtering without replacement in spark_stratified_split() will result in duplicate rows in both training and test data.

This PR persists the intermediate data to avoid the issue caused by Spark lazy evaluation.

See also:

• Avoid lazy evaluation of code in spark without cache
• spark data modified after using random in filter method

Related Issues

Checklist:

• I have followed the contribution guidelines and code style for this project.
• I have added tests covering my contributions.
• I have updated the documentation accordingly.
• This PR is being made to staging branch and not to main branch.

[codecov-commenter]

Codecov Report

Merging #1676 (4dfb783) into staging (17204b1) will increase coverage by 0.12%.
The diff coverage is 100.00%.

❗ Current head 4dfb783 differs from pull request most recent head de6cbf9. Consider uploading reports for the commit de6cbf9 to get more accurate results

@@             Coverage Diff             @@
##           staging    #1676      +/-   ##
===========================================
+ Coverage    23.09%   23.21%   +0.12%     
===========================================
  Files           88       88              
  Lines         9101     9101              
===========================================
+ Hits          2102     2113      +11     
+ Misses        6989     6988       -1     

Flag	Coverage Δ
nightly	?
pr-gate	23.21% <100.00%> (+0.09%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
recommenders/datasets/spark_splitters.py	87.27% <100.00%> (ø)
recommenders/evaluation/spark_evaluation.py	87.05% <0.00%> (+0.44%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17204b1...de6cbf9. Read the comment docs.

[miguelgfierro]

LGTM