-
Notifications
You must be signed in to change notification settings - Fork 4
Ska3 shiny testing
This document describes testing plans and results for the transition to Ska3-shiny.
Because of the significant and sweeping changes in the core packages that comprise Ska3, teams that use Ska3 as part of flight operations are responsible for the following:
- Performing an audit to identify critical tools or software that depend on Ska3
- Testing those tools using the Ska3 shiny environment
- Reporting results by directly editing this page, either with the results in full or by inserting a permanent link to a separate test results page.
Forward and Fix
It is worth noting that many tools are not flight critical and being broken for a week or two is tolerable. In those cases the most efficient path may be "forward and fix". In other words, plan to wait until the promotion of Ska3 shiny to flight and then test at that time. This can be a useful strategy for tools where it is difficult to perform independent testing, e.g. if there are hardwired data paths.
Even in these cases it is important to identify in advance such tools along with a post-promotion test plan. "Testing" will likely consist of running the tool and evaluating the outputs for correctness.
Core testing is done via the Ska3 integration testing package which consists of running available package unit tests and regression test scripts using machinery in the testr package. This is done with the run_testr command, where some tests may be skipped on standalone platforms.
Current results for 2020.14rc2:
HEAD linux CentOS-7
- Tests run via
run_testr: all pass (JC, Dec. 14)
***********************************************************
*** Package Script Status ***
*** ------------------ ------------------------- ------ ***
*** Chandra.Maneuver test_unit.py pass ***
*** Chandra.Maneuver post_check_logs.py pass ***
*** Chandra.Time test_unit.py pass ***
*** Chandra.Time post_check_logs.py pass ***
*** Chandra.cmd_states test_unit.py pass ***
*** Chandra.cmd_states post_check_logs.py pass ***
*** Quaternion test_unit.py pass ***
*** Quaternion post_check_logs.py pass ***
*** Ska.DBI test_unit.py pass ***
*** Ska.DBI post_check_logs.py pass ***
*** Ska.Numpy test_unit.py pass ***
*** Ska.Numpy post_check_logs.py pass ***
*** Ska.Shell test_unit.py pass ***
*** Ska.Shell post_check_logs.py pass ***
*** Ska.engarchive test_make_archive_long.sh pass ***
*** Ska.engarchive test_unit.py pass ***
*** Ska.engarchive post_check_logs.py pass ***
*** Ska.ftp test_unit.py pass ***
*** Ska.ftp post_check_logs.py pass ***
*** Ska.quatutil test_unit_git.sh pass ***
*** Ska.quatutil post_check_logs.py pass ***
*** Ska.tdb test_unit.py pass ***
*** Ska.tdb post_check_logs.py pass ***
*** acis_taco test_unit.py pass ***
*** acis_taco post_check_logs.py pass ***
*** acisfp_check test_regress.sh pass ***
*** acisfp_check test_regress_head.sh pass ***
*** acisfp_check post_check_logs.py pass ***
*** acisfp_check post_regress.py pass ***
*** acisfp_check post_regress_head.py pass ***
*** agasc test_unit.py pass ***
*** agasc post_check_logs.py pass ***
*** annie test_unit.py pass ***
*** annie post_check_logs.py pass ***
*** chandra_aca test_unit.py pass ***
*** chandra_aca post_check_logs.py pass ***
*** cxotime test_unit.py pass ***
*** cxotime post_check_logs.py pass ***
*** dea_check test_regress.sh pass ***
*** dea_check test_regress_head.sh pass ***
*** dea_check post_check_logs.py pass ***
*** dea_check post_regress.py pass ***
*** dea_check post_regress_head.py pass ***
*** dpa_check test_regress.sh pass ***
*** dpa_check test_regress_head.sh pass ***
*** dpa_check post_check_logs.py pass ***
*** dpa_check post_regress.py pass ***
*** dpa_check post_regress_head.py pass ***
*** kadi test_regress.sh pass ***
*** kadi test_unit.py pass ***
*** kadi post_check_logs.py pass ***
*** maude test_unit.py pass ***
*** maude post_check_logs.py pass ***
*** mica test_unit.py pass ***
*** mica post_check_logs.py pass ***
*** package_manifest test_regress.sh pass ***
*** package_manifest post_check_logs.py pass ***
*** package_manifest post_regress.py pass ***
*** parse_cm test_unit.py pass ***
*** parse_cm post_check_logs.py pass ***
*** proseco test_unit.py pass ***
*** proseco post_check_logs.py pass ***
*** psmc_check test_regress.sh pass ***
*** psmc_check test_regress_head.sh pass ***
*** psmc_check post_check_logs.py pass ***
*** psmc_check post_regress.py pass ***
*** psmc_check post_regress_head.py pass ***
*** pyyaks test_unit.py pass ***
*** pyyaks post_check_logs.py pass ***
*** sparkles test_unit.py pass ***
*** sparkles post_check_logs.py pass ***
*** starcheck test_regress.sh pass ***
*** starcheck post_check_logs.py pass ***
*** starcheck post_regress.py pass ***
*** xija test_unit.py pass ***
*** xija post_check_logs.py pass ***
***********************************************************
GRETA (cheru) linux CentOS-8
- Tests run via
run_testr --test-spec test_spec_standalone: all pass (TA, Dec. 15) except:- Ska.quatutil (requires network access to GitHub)
Mac Catalina
- Tests run via
run_testr --test-spec test_spec_standalone), all pass (TA, Dec. 12)
Windows 10
- Tests run via
run_testr --test-spec test_spec_standalone, all pass (TA, Dec. 12) except:- ACIS load review checks (NOT required on Windows)
- Load review tools pass all tests (unit/regression)
sparkles/proseco pass all unit tests.
starcheck - ran full regression test set with starcheck 13.8.0. No diffs.
-
Load review tools pass all tests (unit/regression)
-
acis_thermal_checkdependent packages pass testing suite, which includes regression tests for legacy and ACIS state builders, and functional tests for violations reporting:- Including tests for
dpa_check,dea_check,psmc_check,acisfp_check,fep1_mong_check,fep1_actel_check, andbep_pcb_check
- Including tests for
- Functional and regression testing for
backstop_historypackage
-
-
Functional verification of other important tools (as identified by ACIS ops): PASSED
Check_Power_CommandsWindow_Check
-
Functional testing of mission critical tools all PASSED with ska3 shiny (testing performed by responsible subsystem engineer):
- Momentum Predict tool used on console by OC/CC
- APE eclipse processing tool used on console by OC/CC
- IRU Calibration Tool
- Glimmondb
- fot_trend
- xijafit
- pylimmon
- pyger
- timbre
-
User created and maintained tools will be updated as needed (many of which have already been developed/updated in ska3-shiny environment)
- Functional verification of important tools (note: the MATLAB tools distribution ska3-matlab is not affected)
- The HRC IPI team has completed our simple testing plan. All of our major codes that rely on
Ska3(as well asSka’sMAUDEAPI) run nominally onshinywithout any issues of note. While we do useSkafairly extensively (for e.g. monitoring/trending), none of our mission-critical operational codes (e.g. load review software, RTCADS telemetry monitoring interface) relies onSka.
- No HRC ops operational dependence on Ska3 (email from D. Patnaude 11/23/2020).
From Malgosia:
MTA relies on a locally installed version of the Ska3 environment which does not include ska3-shiny at the moment. Lina has installed Ska3-shiny locally for tests and will update (locally) once all the tests of her software have passed. Takashi is planning to do the same, however, he is not planning to update his local ska3 environment until he converts all his software to python3.8.
Realtime software such as snapshot is largely perl/shell based and does not depend on the Ska3 environment.
From J. Vrtilek:
We have had internal discussions about Ska 3 shiny testing, and have come to the following conclusions:
- Our only critical dependency on Ska3 is through starcheck (tested by ACA).
- We do run yoshi, and accordingly use Ska3; however:
- This work is not in the direct line of operations, and
- Josh, who is maintaining this capability, has a separate Ska installation for this purpose. He will be working on updating as possible but it is not critical.
Run with
c3po-v$ mkdir -p ~/git/acdc/regress_shiny/VC2
c3po-v$ cd ~/git/acdc/regress_shiny/
c3po-v$ cp /dsops/GOT/input/2020_2??_VC2* VC2/
c3po-v$ rm VC2/2020_201_VC2_Replica0_SFDU_64140.gz # So dark cals align with flight
ska3-shiny-c3po-v$ python -m acdc.process_vc2_to_l0 --vc2-root=regress_shiny/VC2
ska3-shiny-c3po-v$ python -m acdc.process_l0_to_quads --data-root=regress_shiny --start 2020:205
ska3-shiny-c3po-v$ python -m acdc.process_quads_to_cals --data-root=regress_shiny
ska3-shiny-c3po-v$ python -m acdc.make_reports --data-root=regress_shiny
Check results:
- All processing runs successfully with reasonable outputs
- Output report for 2020:281 cal looks reasonable
- Output calibration image is consistent with flight (with note)
Note on comparing outputs:
The date of the cals changed by about a minute because of this bug fix: https://github.com/sot/acdc/commit/d92991c.
The temperature values assigned to cals varied in this process by up to 0.07 C compared to flight. This is likely because flight processing is done soon after the cal and can be using available MAUDE telemetry from realtime. This will generate differences because the temperatures are not available during the VC2 readout interval.
In any case the temperature difference and associated dark cal image differences are well within the uncertainties. For the 2020:281 cal the diffs are:
- Mean: 0.003 e-/sec
- Stddev: 0.26 e-/sec
- Max: 27 e-/sec for a pixel with dark cal 16350 e-/sec
dat1 = fits.open('cals/flight/2020/281/cal_2020_281_20_30_48.fits.gz')[0].data
dat2 = fits.open('/proj/sot/ska/data/acdc/cals/flight/2020/281/cal_2020_281_20_31_22.fits.gz')[0].data
diff = dat1 - dat2
np.max(np.abs(diff))
27.182617
np.mean(diff)
0.003512515
np.std(diff)
0.26346657
np.argmax(np.abs(diff))
812785
dat1.ravel()[812785]
16369.87
dat2.ravel()[812785]
16342.6875
Reprocess NOV0220 with Ska3-flight and Ska3-shiny:
cd ~/git/acis_taco
mkdir tmp/acis_taco
cd cd tmp/acis_taco
mkdir flight
mkdir shiny
ska3
/proj/sot/ska3/flight/share/acis_taco/make_esaview_data.py --nweeks=1 --data-root=flight
ska3-shiny
ska3-shiny-kady$ /proj/sot/ska3/shiny/share/acis_taco/make_esaview_data.py --nweeks=1 --data-root=shiny
Then from ipython --matplotlib (on my Mac with sshfs):
import pickle
dat1 = pickle.load(open('kady/tmp/acis_taco/flight/NOV0220.pkl', 'rb'))
dat2 = pickle.load(open('kady/tmp/acis_taco/shiny/NOV0220.pkl', 'rb'))
np.all(dat1['illums'] == dat2['illums']) # True
np.all(dat1['times'] == dat2['times']) # True
Note that the "public" version of the ACA High Background monitor had been running out of a dev ska and was not an application installed in flight.
Shiny test plan:
- Run aca_hi_bgd_update "live" into the real data/web area with shiny test environment
- Confirm no errors from the script
- Confirm output main page at https://cxc.cfa.harvard.edu/mta/ASPECT/aca_hi_bgd_mon/ has expected formatting
- Confirm a generation of a new report matches style and content of previous reports
- Obsid 46795 made with shiny: https://cxc.harvard.edu/mta/ASPECT/aca_hi_bgd_mon/events/obs_46795/index.html
- Confirm email notification on new event works (46795)
Status: Test complete. Good to be promoted as flight.
Setup:
# Removed the /proj/sot/ska3/shiny/data/arc3 symlink to /proj/sot/ska/data/arc3.
rm /proj/sot/ska3/shiny/data/arc3
# Made a new /proj/sot/ska3/shiny/data/arc3 directory
mkdir /proj/sot/ska3/shiny/data/arc3
# Copied the /proj/sot/ska/data/arc3 contents into that directory
rsync -aruvz /proj/sot/ska/data/arc3/* /proj/sot/ska3/shiny/data/arc3/
mkdir /proj/sot/ska/www/ASPECT/arc3_shiny_test/
ln -s /proj/sot/ska/www/ASPECT/arc3_shiny_test/ /proj/sot/ska3/shiny/www/ASPECT/arc3
cd ~/git/arc
# as aca user installed from c5c15d063db1cf1b5be06aacc3d2066414d347b9
make install
Then ran the task pieces by hand to look for errors at the console:
jeanconn-fido> /proj/sot/ska3/shiny/share/arc3/get_iFOT_events.pl
jeanconn-fido> /proj/sot/ska3/shiny/share/arc3/get_web_content.pl
jeanconn-fido> /proj/sot/ska3/shiny/share/arc3/get_goes_x.py --h5=/proj/sot/ska3/shiny/data/arc3/GOES_X.h5
jeanconn-fido> /proj/sot/ska3/shiny/share/arc3/plot_goes_x.py --h5=/proj/sot/ska3/shiny/data/arc3/GOES_X.h5 --out=/proj/sot/ska3/shiny/www/ASPECT/arc3/goes_x.png
jeanconn-fido> /proj/sot/ska3/shiny/share/arc3/get_ace.py --h5=/proj/sot/ska3/shiny/data/arc3/ACE.h5
jeanconn-fido> /proj/sot/ska3/shiny/share/arc3/get_hrc.py --h5=/proj/sot/ska3/shiny/data/arc3/hrc_shield.h5 --data-dir=/proj/sot/ska3/shiny/data/arc3
jeanconn-fido> /proj/sot/ska3/shiny/share/arc3/plot_hrc.py --h5=/proj/sot/ska3/shiny/data/arc3/hrc_shield.h5 --out=/proj/sot/ska3/shiny/www/ASPECT/arc3/hrc_shield.png
jeanconn-fido> /proj/sot/ska3/shiny/share/arc3/make_timeline.py --data-dir=/proj/sot/ska3/shiny/data/arc3
jeanconn-fido> /proj/sot/ska3/shiny/share/arc3/arc.pl
jeanconn-fido> /proj/sot/ska3/shiny/share/arc3/arc.pl -config arc3:arc_ops
jeanconn-fido> /proj/sot/ska3/shiny/share/arc3/arc_time_machine.pl
Examined output at https://cxc.cfa.harvard.edu/mta/ASPECT/arc3_shiny_test/ and checked:
- HTML has right dates
- snapshot content appears correct
- plots match flight bye eye
- differences acceptable (looks like the tagging/labels above the timeline are better/less overlap in shiny output)
Ran as a cronjob on-the-side for a week and confirmed no errors.
On-the-side testing:
# remove symlink
rm /proj/sot/ska3/shiny/data/attitude_error_mon
# make test directory
mkdir /proj/sot/ska3/shiny/data/attitude_error_mon
# seed with flight data
cp -Ruva /proj/sot/ska/data/attitude_error_mon/* /proj/sot/ska3/shiny/data/attitude_error_mon
# make web test link to a new shiny area
ln -s /proj/sot/ska/www/ASPECT/attitude_error_mon_shiny /proj/sot/ska3/shiny/www/ASPECT/attitude_error_mon
Install as aca user in shiny
aca-fido% make install
mkdir -p /proj/sot/ska3/shiny/www/ASPECT/attitude_error_mon
mkdir -p /proj/sot/ska3/shiny/data/attitude_error_mon
mkdir -p /proj/sot/ska3/shiny/share/attitude_error_mon
rsync --times --cvs-exclude att_err_mon.py index_template.html task_schedule.cfg /proj/sot/ska3/shiny/share/attitude_error_mon/
Test at command line for no errors
jeanconn-fido> python att_err_mon.py --outdir ${SKA}/www/ASPECT/attitude_error_mon --datadir ${SKA}/data/attitude_error_mon
Works to make reasonable output from shiny branch of attitude_error_mon:
- script runs without error
- plots are reasonable when compared to current "flight" plots
Ran as a cron job on-the-side for a week and confirmed no errors.
Regression and unit tests provide full coverage of package user functionality and daily cron updates.
Remote access testing (chimchim server):
- Server running from /proj/sot/ska3/shiny tested with shiny client
- Legacy server running from /proj/sot/ska3/flight36 tested with legacy client (Python 3.6)
The legacy client (Python 3.6) does not work with the shiny (Python 3.8) server. Expected and OK.
Ran the key task daily_fss.py in Ska3-shiny and Ska3-flight. This generates a set of plots
showing the FSS performance over a period of time.
- No errors reported.
- Visually compared the output trending plots and confirmed that shiny and flight appeared identical.
- Good for promotion.
Regression and unit tests provide full coverage of package user functionality and daily cron updates.
Known issue:
-
NFS leads to occasional "resource temporarily unavailable" errors when opening
cmds.h5, presumably due to multiple clients attempting to open the file at the same time. This has been seen to occur about once a day or less when running the hourly file update process, and with lower frequency when running (each 5 minutes) the Replan Central task to update the timeline plot.This should be addressed by https://github.com/sot/kadi/pull/196 which will be released in the next Ska3 release after shiny. The impact of the current errors is minimal since the timeline update will be done 5 minutes later and a cmds update will complete an hour later.
- The mica archive ingest and processing code has run into issues with product updates due to repro 5 not due to shiny.
- The mica access code runs fine (unit tests pass) as part of ska_testr.
- We plan an update to mica for the first post-shiny release to fix any remaining ingest issues.
- No lien on shiny.
OK!
- Made a /proj/sot/ska3/shiny/www/ASPECT directory and linked back /proj/sot/ska/www/ASPECT/perigee_health_plots into it.
- Set shiny perigee_health_plots job to run directly into that flight www area.
- This code was recently modernized for ska3 but might as well run directly out of shiny.
- Looks fine except for some findfont text at console.
- findfont text at console does appear to be due to logging.DEBUG default and is acceptable.
OK!
- skawatch.py and hourly_watch.py run without error in the shiny environment.
- the html outputs look reasonable to visual inspection
- the tests in the repository appear out-of-date, and some fail (and fail the same ways in ska3/flight).
OK!
- Not critical, forward and fix.