Skip to content

Commit

Permalink
Drop "nowcast-dev" run from automation workflow and configuration (#229)
Browse files Browse the repository at this point in the history 
* Drop XIOS-2 & NEMO-3.6 build from skookum deployment

The deployment documentation has been updated to remove references to and
instructions on building XIOS-2 and NEMO-3.6 because, as of V21-11, we no longer
do the daily nowcast-dev run on salish. This keeps the docs accurate and
relevant to the current project setup.

* Drop runs directory setup from skookum deployment

The deployment documentation has been updated to remove runs/ directory setup on
skookum because, as of V21-11, we no longer do the daily nowcast-dev run on
salish. This keeps the docs accurate and relevant to the current project setup.

* Rename skookum deployment docs file

`skookum_salish.rst` -> `skookum.rst` because, as of V21-11, we no longer do
the daily nowcast-dev run on salish. So, there is no deployment setup to do on
salish. This keeps the docs accurate and relevant to the current project setup.

* Drop "nowcast-dev" run from automation

As of V21-11, we no longer do the daily nowcast-dev run on salish.

* Update deployment docs re: end of nowcast-dev runs

* Restore nowcast-dev results archive config item

Retained until make_plots worker is refactored to remove nowcast-dev. That
refactoring needs to come after the compare_venus_ctd fig module is updated
to handle the ONC API v3.

* Restore shared storge config item for salish-nowcast

Retained because the next_workers module checks the shared storage item for
all enabled hosts. So, it can't be removed for salish-nowcast until that whole
stanza can be removed.

* Fix tests re: acad43d
  • Loading branch information
@douglatornell
douglatornell authored Jan 26, 2024
1 parent da747f9 commit a7cdbca
Show file tree
Hide file tree
Showing 12 changed files with 38 additions and 440 deletions.
46 changes: 10 additions & 36 deletions config/nowcast.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -292,14 +292,10 @@ run types:
land processor elimination: bathymetry_202108.csv
duration: 1 # day
restart from: nowcast-green
### retained because plots for nowcast-dev haven't been removed from
### make_plots.make_plots() yet
nowcast-dev:
config name: SalishSeaCast_Blue
coordinates: coordinates_seagrid_SalishSea201702.nc
bathymetry: bathymetry_201702.nc
mesh mask: mesh_mask201702.nc
land processor elimination: bathymetry_201702.csv
duration: 1 # day
restart from: nowcast-dev
nowcast-green:
config name: SalishSeaCast
coordinates: coordinates_seagrid_SalishSea201702.nc
Expand Down Expand Up @@ -340,6 +336,9 @@ results archive:
# Values are directories on results server where run results are stored
# in ddmmmyy/ directories.
nowcast: /results/SalishSea/nowcast-blue.202111/
### retained until make_plots is refactored to remove nowcast-dev
### that refactoring needs to come after compare_venus_ctd fig module is updated
### to handle ONC API v3
nowcast-dev: /results/SalishSea/nowcast-dev.201905/
forecast: /results/SalishSea/forecast.202111/
forecast2: /results/SalishSea/forecast2.202111/
Expand Down Expand Up @@ -623,34 +622,13 @@ run:
weather dir: /nemoShare/MEOPAR/GEM2.5/ops/NEMO-atmos/

salish-nowcast:
shared storage: True
make forcing links: True
### retained because workers.test_upload_forcing.TestConfig.test_ssh_keys() needs
### to find this item for every enabled host
ssh key: SalishSeaNEMO-nowcast_id_rsa
envvars: /SalishSeaCast/nowcast-env/etc/conda/activate.d/envvars.sh
python: /SalishSeaCast/nowcast-env/bin/python3
config file: /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
run prep dir: /SalishSeaCast/runs/
### retained because next_workers needs to know until salish is fully removed
shared storage: True
### retained because make_*_runoff_file workers use this to get grid files
grid dir: /SalishSeaCast/grid/
salishsea_cmd: /SalishSeaCast/nowcast-env/bin/salishsea
job exec cmd: qsub
# Optional email address to which to send notifications of the beginning
# and end of execution of the run, as well as notification of abnormal
# abort messages.
email: [email protected]
run types:
nowcast-dev:
run sets dir: /SalishSeaCast/SS-run-sets/v201905/nowcast-dev/
mpi decomposition: 1x7
walltime: 23:30:00
results: /results/SalishSea/nowcast-dev.201905/
nowcast-green:
results: /results2/SalishSea/nowcast-green.201905/
forcing:
ssh dir: /results/forcing/sshNeahBay/
bc dir: /results/forcing/LiveOcean/boundary_conditions/
rivers dir: /results/forcing/rivers/
bottom friction mask: /SalishSeaCast/grid/jetty_mask_bathy201702.nc
weather dir: /results/forcing/atmospheric/continental2.5/nemo_forcing/

orcinus-nowcast-agrif:
ssh key: SalishSeaNEMO-nowcast_id_rsa
Expand Down Expand Up @@ -1372,8 +1350,6 @@ message registry:
failure nowcast: nowcast NEMO run failed
success nowcast-green: nowcast-green NEMO run started
failure nowcast-green: nowcast-green NEMO run failed
success nowcast-dev: nowcast-dev NEMO run started
failure nowcast-dev: nowcast-dev NEMO run failed
success forecast: forecast NEMO run started
failure forecast: forecast NEMO run failed
success forecast2: forecast2 NEMO run started
Expand All @@ -1399,8 +1375,6 @@ message registry:
failure nowcast: nowcast watch_NEMO failed
success nowcast-green: nowcast-green NEMO run ended
failure nowcast-green: nowcast-green watch_NEMO failed
success nowcast-dev: nowcast-dev NEMO run ended
failure nowcast-dev: nowcast-dev watch_NEMO failed
success forecast: forecast NEMO run ended
failure forecast: forecast watch_NEMO failed
success forecast2: forecast2 NEMO run ended
Expand Down
23 changes: 11 additions & 12 deletions docs/deployment/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Nowcast Production Deployments
******************************

In October 2016 the production deployment of the nowcast system was changed to use the :ref:`SalishSeaNowcast-repo` package that is based on the `NEMO_Nowcast framework`_ framework.
The production deployment uses 3 systems:
The production deployment uses 2 systems:

.. _NEMO_Nowcast framework: https://nemo-nowcast.readthedocs.io/en/latest/

Expand All @@ -32,14 +32,6 @@ The production deployment uses 3 systems:
:py:mod:`nemo_nowcast.log_aggregator`,
most of the pre- and post-processing workers run on the :ref:`SalishSeaModelResultsServer`, :kbd:`skookum`, where the deployment is in the :file:`/SalishSeaCast/` directory tree.

#. The development compute server,
:kbd:`salish`,
is used to run the daily development model run,
:kbd:`nowcast-dev` NEMO-3.6 model run.
:kbd:`salish` and :kbd:`skookum` share storage via NFS mounts,
so,
:kbd:`salish` uses the same deployment in the :file:`/SalishSeaCast/` directory tree.

#. The daily
:kbd:`forecast2`
(preliminary forecast),
Expand Down Expand Up @@ -73,13 +65,13 @@ The production deployment uses 3 systems:
.. _arbutus.cloud: https://docs.alliancecan.ca/wiki/Cloud_resources#Arbutus_cloud
.. _Ceph object storage: https://en.wikipedia.org/wiki/Ceph_(software)

These sections describe the setup of the nowcast system on :kbd:`skookum`/:kbd:`salish` and :kbd:`arbutus.cloud`,
and it operation.
These sections describe the setup of the nowcast system on :kbd:`skookum` and :kbd:`arbutus.cloud`,
and their operation.

.. toctree::
:maxdepth: 2

skookum_salish
skookum
arbutus_cloud
operations

Expand All @@ -106,3 +98,10 @@ The setup on :kbd:`optimum` is described in:
See also the `#optimum-cluster`_ Slack channel.

.. _#optimum-cluster: https://salishseacast.slack.com/?redir=%2Farchives%2FC011S7BCWGK

With the update of the production to run the V21-11 model version in January 2024,
we decided to end the daily :kbd:`nowcast-dev` development model runs on :kbd:`salish`.
Development is now generally done in research runs on :kbd:`graham`.
:kbd:`salish` is now mostly used for analysis tasks, post-processing of NEMO model results files
to produce day-average and month-average dataset files,
and Lagrangian particle tracking analysis with :program:`ariane` and :program:`OceanParcels`.
47 changes: 4 additions & 43 deletions docs/deployment/skookum_salish.rst → docs/deployment/skookum.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,11 @@
.. SPDX-License-Identifier: Apache-2.0
.. _SkookumSalishDeployment:
.. _SkookumDeployment:

***************************************
:kbd:`skookum`/:kbd:`salish` Deployment
***************************************
************************
:kbd:`skookum`Deployment
************************

Git Repositories
================
Expand All @@ -47,42 +47,6 @@ Clone the following repos into :file:`/SalishSeaCast/`:
$ git clone [email protected]:SalishSeaCast/tracers.git
$ git clone [email protected]:mdunphy/FVCOM-VHFR-config.git
$ git clone [email protected]:douglatornell/OPPTools.git
$ git clone [email protected]:SalishSeaCast/NEMO-3.6-code.git
$ git clone [email protected]:SalishSeaCast/XIOS-ARCH.git
$ git clone [email protected]:SalishSeaCast/XIOS-2.git
Build XIOS-2
============

Symlink the XIOS-2 build configuration files for :kbd:`salish` from the
:file:`XIOS-ARCH` repo clone into the :file:`XIOS-2/arch/` directory:

.. code-block:: bash
$ cd /SalishSeaCast/XIOS-2/arch
$ ln -s ../../XIOS-ARCH/UBC-EOAS/arch-GCC_SALISH.fcm
$ ln -s ../../XIOS-ARCH/UBC-EOAS/arch-GCC_SALISH.path
:command:`ssh` to :kbd:`salish` and build XIOS-2 with:

.. code-block:: bash
$ cd /SalishSeaCast/XIOS-2
$ ./make_xios --arch GCC_SALISH --netcdf_lib netcdf4_seq --job 8
Build NEMO-3.6
==============

Build NEMO-3.6 and :program:`rebuild_nemo.exe`:

.. code-block:: bash
$ cd /SalishSeaCast/NEMO-3.6-code/NEMOGCM/CONFIG
$ XIOS_HOME=/SalishSeaCast/XIOS-2 ./makenemo -m GCC_SALISH -n SalishSeaCast_Blue -j8
$ cd /SalishSeaCast/NEMO-3.6-code/NEMOGCM/TOOLS/
$ XIOS_HOME=/SalishSeaCast/XIOS-2 ./maketools -m GCC_SALISH -n REBUILD_NEMO


Python Packages
Expand Down Expand Up @@ -264,9 +228,6 @@ On the hosts where the nowcast system NEMO runs will be executed create a
The hosts and their :file:`runs` directories presently in use are:
* :kbd:`salish`
:file:`/SalishSeaCast/runs/`
* :kbd:`arbutus.cloud`
See :ref:`ArbutusCloudNEMORunsDirectory`
Expand Down
18 changes: 1 addition & 17 deletions nowcast/next_workers.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -667,19 +667,17 @@ def after_make_forcing_links(msg, config, checklist):
"failure nowcast+": [],
"failure nowcast-green": [],
"failure nowcast-agrif": [],
"failure nowcast-dev": [],
"failure forecast2": [],
"failure ssh": [],
"success nowcast+": [],
"success nowcast-green": [],
"success nowcast-agrif": [],
"success nowcast-dev": [],
"success forecast2": [],
"success ssh": [],
}
if msg.type.startswith("success"):
run_types = {
"nowcast+": ("nowcast", "nowcast-dev"),
"nowcast+": ("nowcast",),
"nowcast-green": ("nowcast-green",),
"nowcast-agrif": ("nowcast-agrif",),
"ssh": ("forecast",),
Expand Down Expand Up @@ -738,12 +736,10 @@ def after_run_NEMO(msg, config, checklist):
"crash": [],
"failure nowcast": [],
"failure nowcast-green": [],
"failure nowcast-dev": [],
"failure forecast": [],
"failure forecast2": [],
"success nowcast": [],
"success nowcast-green": [],
"success nowcast-dev": [],
"success forecast": [],
"success forecast2": [],
}
Expand Down Expand Up @@ -777,12 +773,10 @@ def after_watch_NEMO(msg, config, checklist):
"crash": [],
"failure nowcast": [],
"failure nowcast-green": [],
"failure nowcast-dev": [],
"failure forecast": [],
"failure forecast2": [],
"success nowcast": [],
"success nowcast-green": [],
"success nowcast-dev": [],
"success forecast": [],
"success forecast2": [],
}
Expand Down Expand Up @@ -849,16 +843,6 @@ def after_watch_NEMO(msg, config, checklist):
],
)
)
if run_type == "nowcast-green":
for host in config["run"]["enabled hosts"]:
run_types = config["run"]["enabled hosts"][host]["run types"]
if "nowcast-dev" in run_types:
next_workers[msg.type].append(
NextWorker(
"nowcast.workers.make_forcing_links",
args=[host, "nowcast+", "--shared-storage"],
)
)
if run_type == "forecast2":
host_name = config["wave forecasts"]["host"]
run_date = arrow.get(msg.payload[run_type]["run date"]).shift(days=+1)
Expand Down
17 changes: 4 additions & 13 deletions nowcast/workers/run_NEMO.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,8 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""SalishSeaCast worker that prepares the YAML run
description file and bash run script for a nowcast, nowcast-green, nowcast-dev,
forecast or forecast2 run on the ONC cloud or salish,
and launches the run.
"""SalishSeaCast worker that prepares the YAML run description file and bash run script
for a nowcast, nowcast-green, forecast or forecast2 run on the ONC cloud, and launches the run.
"""
import logging
import os
Expand Down Expand Up @@ -49,12 +47,11 @@ def main():
worker.cli.add_argument("host_name", help="Name of the host to execute the run on")
worker.cli.add_argument(
"run_type",
choices={"nowcast", "nowcast-green", "nowcast-dev", "forecast", "forecast2"},
choices={"nowcast", "nowcast-green", "forecast", "forecast2"},
help="""
Type of run to execute:
'nowcast' means nowcast physics run,
'nowcast-green' means nowcast green ocean run,
'nowcast-dev' means nowcast physics run with in-development features,
'forecast' means updated forecast run,
'forecast2' means preliminary forecast run,
""",
Expand Down Expand Up @@ -145,7 +142,6 @@ def _create_run_desc_file(run_date, run_type, host_name, config):
run_days = {
"nowcast": run_date,
"nowcast-green": run_date,
"nowcast-dev": run_date,
"forecast": run_date.shift(days=1),
"forecast2": run_date.shift(days=2),
}
Expand All @@ -170,7 +166,6 @@ def _update_time_namelist(run_date, run_type, run_duration, host_config, run_pre
# run-type: based-on run-type, date offset
"nowcast": ("nowcast", -1),
"nowcast-green": ("nowcast-green", -1),
"nowcast-dev": ("nowcast-dev", -1),
"forecast": ("nowcast", 0),
"forecast2": ("forecast", 0),
}
Expand Down Expand Up @@ -207,7 +202,6 @@ def _calc_new_namelist_lines(
run_date_offset = {
"nowcast": 0,
"nowcast-green": 0,
"nowcast-dev": 0,
"forecast": 1,
"forecast2": 2,
}
Expand Down Expand Up @@ -246,7 +240,6 @@ def _run_description(run_date, run_type, run_id, restart_timestep, host_name, co
# run-type: previous run's ddmmmyy results directory name
"nowcast": run_date.shift(days=-1).format("DDMMMYY").lower(),
"nowcast-green": run_date.shift(days=-1).format("DDMMMYY").lower(),
"nowcast-dev": run_date.shift(days=-1).format("DDMMMYY").lower(),
"forecast": run_date.shift(days=-1).format("DDMMMYY").lower(),
"forecast2": run_date.shift(days=-2).format("DDMMMYY").lower(),
}
Expand Down Expand Up @@ -346,9 +339,7 @@ def _run_description(run_date, run_type, run_id, restart_timestep, host_name, co
grid_dir / config["run types"][run_type]["bathymetry"]
)
lpe_filename = config["run types"][run_type]["land processor elimination"]
run_desc["grid"]["land processor elimination"] = (
False if host_name == "salish-nowcast" else os.fspath(grid_dir / lpe_filename)
)
run_desc["grid"]["land processor elimination"] = os.fspath(grid_dir / lpe_filename)
run_desc["restart"] = restart_filepaths
run_desc["output"].update(
{
Expand Down
2 changes: 1 addition & 1 deletion nowcast/workers/watch_NEMO.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ def main():
worker.cli.add_argument("host_name", help="Name of the host to monitor the run on")
worker.cli.add_argument(
"run_type",
choices={"nowcast", "nowcast-green", "nowcast-dev", "forecast", "forecast2"},
choices={"nowcast", "nowcast-green", "forecast", "forecast2"},
help="""
Type of run to monitor:
'nowcast' means nowcast physics run,
Expand Down
Loading

0 comments on commit a7cdbca

Please sign in to comment.