Using file split to run individual examples as jobs using lots of memory?

[littlemight]

Hi all!

Solid library you guys have here, been using it for a CI test optimization that I've been working on and it's been great.
Lately I want to try to use the --file-split-threshold feature to split one of my Ruby on Rails test files which is a bit slow (1 file has ~1000 examples, each takes around 0.5-1s to complete) so that it can be worked on by multiple workers, running the file as 1 job takes about ~400s to complete.

When I try to split the file using --file-split-threshold, it gets split into ~1000 jobs and all of the workers don't even get past the 50-th job, turns out it's because the worker container ran out of memory (error code 137)

Here's a memory graph of said phenomenon (only 1 worker graph for this example)

Now I want to make sure if this is the problem on the tests I was running or a caveat with the library, so I pulled the rspecq repo and added this test

# my_spec.rb
RSpec.describe do
  1000.times do
    it do
      expect(true).to be true
    end
  end
end

Dry run

redis-cli flushdb
bundle exec rspecq --build=0 --seed 1 --worker=1 \
  --update-timings test/sample_suites/timings/spec/my_spec.rb

...then run it for file splitting

bundle exec rspecq --build=1 --seed 1 --worker=1 --update-timings --file-split-threshold 0 test/sample_suites/timings/spec/my_spec.rb

Turns out it's also hogging the memory on this dummy test as well, steadily increasing the memory usage until 3,7 GB.

*I modified the logging a bit just to see the executed examples better)

Now I'm pretty sure from the graph this indicates a memory leak (or bloat..?), and was wondering if there's something I'm missing before using the --file-split-threshold option? Maybe a configuration that I have to specify on spec_helper.rb or something like that. This is the spec_helper.rb that I used on my test

RSpec.configure do |config|
  # other config here, not really relevant

  config.filter_run_when_matching :focus
end

Ruby version is 2.7.2

Can you help me look into this? I've been dabbling on this out of memory problem for a while 😢

Thanks 🙇‍♂️!

[agis]

I think you're running onto rspec/rspec-core#2767. This is a known issue for which we never got to the bottom of, because it didn't affect us (our builds finish way before memory is saturated).

Besides finding and fixing the root cause of the aforementioned issue, a workaround would be to reduce the number of jobs your build executes, by splitting that large file to smaller ones (i.e. with 50-100 examples each). This would be more efficient as well, because it would reduce the IO pressure on Redis too. Also, make sure you set a reasonable threshold instead of 0, after inspecting your files' runtimes.

[littlemight]

I see, thank you for pointing out the rspec-core issue, that's a weird one :/.
Thanks for the suggestions as well! Right now in my tests I have a for loop which creates a unit test for every endpoint, so splitting it to different files might be bad for developer experience. What I'm doing now is a bit dirty/hacky as you can see here (pardon the dirty code 😅), I'm grouping 50 examples per job :p for individual examples, this seems to work fine so far in CI runs, and it's not gobbling up the memory like before.

You can close this issue if you want, seems like it's not really an issue related to rspecq anyway