Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

update the ice diags generation to allow for parallelism #24

Closed
bertinia opened this issue Aug 3, 2016 · 1 comment
Closed

update the ice diags generation to allow for parallelism #24

bertinia opened this issue Aug 3, 2016 · 1 comment
Assignees
@bertinia
Labels
enhancement

Comments

@bertinia
Copy link
Contributor

bertinia commented Aug 3, 2016 •
edited
Loading

Currently the ice diags require that the model_vs_obs be run first followed by the model_vs_model. These 2 diag types should be able to split the MPI communicator and run in parallel. The problem is with the same plot classes being used for model_vs_obs and model_vs_model. Solution is to add additional plot classes to accommodate the 2 diag types.
@bertinia bertinia self-assigned this Aug 3, 2016
@bertinia bertinia changed the title update the ice diags html generation to allow for parallelism update the ice diags generation to allow for parallelism Aug 4, 2016
@bertinia
Copy link
Contributor Author

bertinia commented Aug 4, 2016

Done and checked into trunk and tagged at postprocessing_20160804.
You can now run ice diags model_vs_obs and model_vs_model in parallel.

@bertinia bertinia closed this as completed Aug 4, 2016
bertinia added a commit to bertinia/CESM_postprocessing that referenced this issue Apr 24, 2017
@bertinia
Squashed 'reshaper/pyReshaper/' changes from 944ab10..769e587
b3d29ca 
769e587 Merge pull request NCAR#29 from NCAR/devel
95fcbe6 Merge pull request NCAR#28 from NCAR/chunking
7f1ebbb Adding ability to produce speedup plots
1a81993 Add ability to plot log plots and subselected jobs by data
4d65341 Adding functionality to plot log bar plots
17337ab Merge branch 'chunking' of https://github.com/NCAR/PyReshaper into chunking
f5716ae Formatting only.
db18732 New YS 8x16 (128 proc) timings.
8a27c17 New timings for 4x4 YS runs.
f0ceb7e BUGFIX:  Wrong checks when generating plots.  Also resizing.
7a69d4e New Yellowstone timing numbers.
d4cd015 Merge branch 'chunking' of https://github.com/NCAR/PyReshaper into chunking
5f366c1 Changing pop-0.1deg chunking to fit in yellowstone memory.
7f33829 Adding to change log.
883eeab Adding chunking documentation to manual
738d310 Bumping up pop-0.1deg chunk size
1d4598a Copying version 0.X timings into new database for comparisons
1d612ba Renaming new timings database to default name
b960ddc Renaming old timings file to match old version
c691b15 New timings database for new PyReshaper version
27b420c Changing to indicate read chunk sizes only (write are same)
b8e49e2 Attempt to adjust chunk sizes.
81b0223 Merge branch 'chunking' of https://github.com/NCAR/PyReshaper into chunking
66ca398 Set camfv-1deg chunk sizes to none (read/write whole variables)
e635b00 Merge branch 'chunking' of https://github.com/NCAR/PyReshaper into chunking
57b2fe6 Moving sync location in convert
82395f5 BUGFIX: Wrong capitalization when attempting to call getdb()
5627dca Adding a message about the chunk sizes used.
47109f1 Fixing typo in warning message
4eed760 Adding chunking parameters to testinfo database.
b84c7ad Merge pull request NCAR#27 from NCAR/devel
fb809a4 BUGFIX: Datatype must be converted to a numpy dtype
c3c204c Revert "Working on updating to Python3"
0de3b37 Revert "Updating IOBackend tests to work with Python 2.6+"
3a3f1d8 Revert "Updating to work with Python2.6+"
2f0fc2c Revert "Updating to work with Python2.6+"
66374d9 Updating to work with Python2.6+
a8f0fc2 Updating to work with Python2.6+
232406a Updating IOBackend tests to work with Python 2.6+
549ddf9 Working on updating to Python3
3350d71 Simplifying chunking argument parsing
f703b59 Suppressing PyDev error
82cbcc0 Cleanup and passing in chunks to Reshaper convert method
48b4740 Enabling chunking via command-line script
17e98ad Adding ability to create new timings file at runtime.
3556e9b Adding ability to read/write separate timings files
62d0173 Removing NetCDF compare tool.  Try to use cprnc.
d413515 Adding pretty-print to output
8b3c256 Allowing backwards-compatibility with older netCDF4
b34445f Removing conflict with parser's help option
93ab872 One more modification
59155ad Making the script executable from the command line
ae172ca Adding a NetCDF compare tool
533756b Merge branch 'devel' of https://github.com/NCAR/PyReshaper into devel
e5ed2ac Adding module load to init and ignoring results directory.
68a3e3c Bugfix.  Need to set default chunking after unlimited dim is found.
d1c4edd Some cleanup and restructuring.
4343273 First pass at actual chunking code
5a07d3a Not printing 'usage'.  Printing 'use'.
a66c288 Merge pull request NCAR#25 from NCAR/memory_diag
af921ec Adding maximum memory usage diagnostic output
04ff328 Merge pull request NCAR#24 from NCAR/extract_only
df9e99c Updated changelog file.  No more dates.  Reverse order.
a2c3e47 Updating the manual to describe the new extract-only features
f31a7a0 Adding command-line option to extract only some time-series variables
c3d06e8 Extract only some variables seems to work from Python side.
79af819 Checking extract_only tests.  Appears to work, but needs more testing.
58eeb21 Updating specification tests to include time-series name list
13578d2 Updated GitHub references
2567496 First pass as modifying reshaper to use explicit time-series name list
031cfda Modifying specification to take time-series name list
7f7fa13 Merge pull request NCAR#23 from NCAR/default_1D_metadata
b4ec088 Updating manual for new feature
c87bafc Finished adding option to assert all 1d time-var variables are metadata
568d534 Cleanup:  Better allowances for netCDF4-python v1.1.1
95aaf34 attempting to add netCDF4 v1.1.1 support
38ab628 Updating bakeoff data location on GLADE
9f55a5e Need to repr the wmode argument when writing script (must be str)
f78f002 Bugfix
670edfa Backporting checkresults to Python 2.6
cc0a772 Bugfix:  more specfile extension changing
cbed5cd Bugfix:  specfile has extension s2s
38fef51 bugfix
f1de114 Bugfix and changing usage statements
cf29b54 More formatting of descriptions
601081d Updating descriptions of test utilities
dbff464 Backporting YS tests to Python version 2.6
fed167e Merge pull request NCAR#22 from NCAR/iobackend
7aa5477 Updating README to be consistent with manual.
12aa9e3 Updating CHANGELOG
44823f9 Updating user documentation
0a8d18b Updating comments only.
f5e7ad7 Modifying CHANGES log
f3b6041 Formatting only.
a38aa2a Updating changelog.
6eab81b Cleaning up iobackend branch, allowing I/O backend to be set by default
f762a67 Moved I/O backend info output to initialization
69548a0 More output about the NetCDF output settings.
7b0f71b Allowing the use of Python 2.6 ordereddict module, if available
6cd1d9f Changing the default backend to PyNIO (tests suggest it's faster)
74e497c Removing unnecessary import
915bf1c Updating tests with new names
469a0e0 Implementing use of new get_value/assign_value functions
b557895 Adding get_value and assign_value methods for general get/set
e26ffb1 Attempting to convert typecode 'S' to 'c' when using PyNIO
e8bd27a Casting datatype to string in Nio create_variable
d82aacc Adding user-level output on which I/O backend is being used.
9cdad55 Adding option for I/O backend to s2smake (stored in specfile)
562f6c4 Updating to new iobackend changes
6df401a Removing debug output. Changing 'typecode' str to 'datatype' Numpy.dtype
19ced93 More debugging info.
0634559 Adding debug output.
ee97a88 Switching PyReshaper objects over to new iobackend
dae810d Adding specifier checks on backend
49c081f Adding shape and size methods to NCVariable
3ed54aa Adding append tests.  All tests pass.
8b72ef5 Added write tests.  All are passing.
7c56210 Cleaned up API.  Switched to netCDF4 way of dealing with attributes.
0c3cfd8 Now adding iobackend and tests.
43cc6d8 Updated copyright info.  Added iobackend module and started tests.
2257386 Removing MultiSpecReshaper reference from factory function
18b5359 Removing MultiSpecReshaper
139f708 Starting work on new memory-management system/partitioner

git-subtree-dir: reshaper/pyReshaper
git-subtree-split: 769e587dabe15c0ecaf4d0c2903152a12ae640ce
bertinia added a commit to bertinia/CESM_postprocessing that referenced this issue Oct 26, 2017
@bertinia
Squashed 'averager/pyAverager/' changes from 0f5eb0b..c425638
da339ee 
c425638 Update __init__.py
e340d4c Update README
bffb962 Update setup.py
REVERT: 0f5eb0b Merge pull request NCAR#34 from NCAR/devel
REVERT: 34747e1 Merge pull request NCAR#29 from NCAR/devel
REVERT: fa67fce Merge pull request NCAR#26 from NCAR/devel
REVERT: 4ac6f44 Merge pull request NCAR#24 from NCAR/devel
REVERT: 3fc272d Merge pull request NCAR#22 from NCAR/devel
REVERT: 108739b Merge pull request NCAR#20 from NCAR/devel
REVERT: 490ba39 Merge pull request NCAR#18 from NCAR/devel
REVERT: d34c46e Merge pull request NCAR#15 from NCAR/devel
REVERT: d2290b6 Merge pull request NCAR#9 from NCAR/devel
REVERT: 2436413 Merge pull request NCAR#6 from NCAR-CISL-ASAP/devel
REVERT: 2125a4f Merge pull request NCAR#4 from NCAR-CISL-ASAP/devel
REVERT: c51daf1 Merge pull request NCAR#1 from NCAR-CISL-ASAP/devel

git-subtree-dir: averager/pyAverager
git-subtree-split: c425638e984b30ba743dd0dbcb30ff7de071b240
bertinia added a commit to bertinia/CESM_postprocessing that referenced this issue Feb 15, 2018
@bertinia
Squashed 'averager/pyAverager/' changes from 1cc306a..018f31f
e9cd828 
018f31f Merge pull request NCAR#44 from NCAR/devel
f28eccb Merge pull request NCAR#43 from NCAR/char_bug
6fbc661 Bump verison number
92a2b8c Merge pull request NCAR#42 from NCAR/char_bug
e67cc3a Update to the previous fix:
37afc58 Merge pull request NCAR#41 from NCAR/char_bug
e4b2715 Fix bug with character identification. The code was only checking for 'S1'.  Compressed netcdf4 files identify characters as 'S3'.
e3f121e Merge pull request NCAR#40 from NCAR/devel
28f36ec Merge pull request NCAR#38 from NCAR/devel
3563c85 Merge pull request NCAR#35 from NCAR/devel
0f5eb0b Merge pull request NCAR#34 from NCAR/devel
34747e1 Merge pull request NCAR#29 from NCAR/devel
fa67fce Merge pull request NCAR#26 from NCAR/devel
4ac6f44 Merge pull request NCAR#24 from NCAR/devel
3fc272d Merge pull request NCAR#22 from NCAR/devel
108739b Merge pull request NCAR#20 from NCAR/devel
490ba39 Merge pull request NCAR#18 from NCAR/devel
d34c46e Merge pull request NCAR#15 from NCAR/devel
d2290b6 Merge pull request NCAR#9 from NCAR/devel
2436413 Merge pull request NCAR#6 from NCAR-CISL-ASAP/devel
2125a4f Merge pull request NCAR#4 from NCAR-CISL-ASAP/devel
c51daf1 Merge pull request NCAR#1 from NCAR-CISL-ASAP/devel

git-subtree-dir: averager/pyAverager
git-subtree-split: 018f31ff4013fa0b240470003f27025811aeaf11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bertinia bertinia

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bertinia