Unit tests for retrieve_data.py will not run successfully on Jet, Hera

[mkavulich]

The file ush/test_retrieve_data.py includes many unit tests that require access to HPSS. These tests are not run as part of CI due to lack of HPSS access, so they are not run regularly (if at all).

Expected behavior

Despite not being run regularly, these tests should work on Jet and Hera (as long as the HPSS module is loaded). Each test is run in a temporary directory with a random name (using Python's tempfile module) that should be removed/cleaned up after each test.

Current behavior

Due to a bug in the test design, each subsequent test is run from within that temporary directory, so it is not cleaned up, and the next test is run in a subdirectory of the "temporary" directory (that is not cleaned up properly, due to being the current working directory). This results in two problems: 1. entering the subdirectory and then passing that subdirectory as the output_path argument means that files that need to be untarred after being retrieved will not end up in the correct relative location, and 2. If the number of tests is high enough, the length of the file paths hit a system limit, causing tests to fail.

Note that this second error could eventually impact the cloud-based CI as well, if enough tests were added that the file names became too long there too.

Machines affected

Jet, Hera, potentially others

Steps To Reproduce

1. git clone https://github.com/ufs-community/ufs-srweather-app.git
2. cd ufs-srweather-app/ush
3. time python -m unittest test_retrieve_data
4. Observe failure:

----------------------------------------------------------------------
Ran 10 tests in 843.856s

FAILED (errors=4)

Detailed Description of Fix

The fix is simple: removing the line os.chdir(tmp_dir) from each testing method fixes the problem. I will be including this fix in my testing PR (step 2 of #587) that I should be opening soon.