Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix performputs buffersize #2326

Merged
merged 4 commits into from
Jun 16, 2020

Conversation

pnorbert
Copy link
Contributor

…datasize correctly in case of multiple calls of PerformPuts(). Addresses #2320

@pnorbert
Copy link
Contributor Author

@chuckatkins This test writes 3 datasets, each 721MB (and does this twice testing BP3 and BP4). The dataset is a side effect only, so we could remove each immediately after writing, but testing has no access to thirdparty (<adios2sys/SystemTools.hxx> specifically). Do you have any suggestion how to add data removal either in CMakeLists.txt or in the code itself? Removing after test would still require 2.1GB of space, removing each right after writing would still require 721MB of space to write during CI.

@pnorbert
Copy link
Contributor Author

@chuckatkins NVM for this test, now it uses NULL transport to avoid writing to disk at all.
But in general it would be good to know how to remove output files after a test.

@pnorbert
Copy link
Contributor Author

pnorbert commented Jun 14, 2020

@chuckatkins @bradking Do you guys have an idea why the sanitizers fail to run this new test?
test 234/235, they print /proc/self/status and the VmSize is ridiculously big here. And they timeout, even though they don't do any IO anymore, just buffer 80MB of data 3x3x3 times.

234: After Open():                                 buffer size = 16384        VmSize:	107374517864 kB VmRSS:	  317036 kB 

@bradking
Copy link
Collaborator

Cc: @KyleFromKitware

@chuckatkins
Copy link
Contributor

@chuckatkins This test writes 3 datasets, each 721MB (and does this twice testing BP3 and BP4). ... Do you have any suggestion how to add data removal either in CMakeLists.txt or in the code itself? Removing after test would still require 2.1GB of space, removing each right after writing would still require 721MB of space to write during CI.

@pnorbert We should have the capacity on the test VMs for a 2.1gb working set, so long as it's removed after the test. The way to accomplish that is through test fixtures. See the docs here for how to use it: https://cmake.org/cmake/help/v3.7/prop_test/FIXTURES_REQUIRED.html

But essentially it's just:

# TestFoo generates some output that needs to be cleaned up
add_test(NAME TestFoo ...)

# Make a "test" that cleans up after foo
add_test(NAME TestFooCleanup
  COMMAND ${CMAKE_COMMAND} -E remove_directory /pth/to/TestFooOutput
)

# Mark it as a cleanup test for the "Foo" fixture
set_tests_properties(TestFooCleanup PROPERTIES FIXTURES_CLEANUP Foo)

# Mark the TestFoo as needing any setup or cleanup defined in the "Foo" fixture
set_tests_properties(TestFoo PROPERTIES FIXTURES_REQUIRED Foo)

pnorbert added 4 commits June 16, 2020 13:40
…datasize correctly in case of multiple calls of PerformPuts(). Addresses ornladios#2320
- profile is written as <filename.bp>_profiling.json if only a NULL transport is used
- TestBPBufferSize now uses NULL transport to avoid writing lots of data to disk
@pnorbert pnorbert force-pushed the fix-performputs-buffersize branch from 9edcdbd to 39a545f Compare June 16, 2020 17:40
@pnorbert pnorbert merged commit 5b6f691 into ornladios:master Jun 16, 2020
@pnorbert pnorbert deleted the fix-performputs-buffersize branch June 16, 2020 19:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants