Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop FVCOM VHFR forecast-x2 runs from automation #100

Merged
merged 1 commit into from
Apr 14, 2022
Merged

Drop FVCOM VHFR forecast-x2 runs from automation #100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 7 additions & 39 deletions config/nowcast.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -366,9 +366,6 @@ rolling forecasts:
days from past: 5
# Root of directory tree to hold temporary results files
temporary results archives: /tmp/
fvcom:
# Destination directory for most recent forecast file symlinks
most recent forecast dir: /opp/fvcom/most_recent_forecast/
nemo:
# Destination directory for rolling forecast results directories
dest dir: /results/SalishSea/rolling-forecasts/nemo/
Expand Down Expand Up @@ -442,8 +439,6 @@ erddap:
- ubcSSFVCOM-VHFR-BaroclinicX2
fvcom-r12-nowcast:
- ubcSSFVCOM-VHFR-BaroclinicR12
fvcom-forecast:
- ubcSSFVCOM-VHFR-BaroclinicX2


figures:
Expand Down Expand Up @@ -858,7 +853,7 @@ vhfr fvcom runs:
# Template for NEMO runoff forcing file names
# **Must be quoted to project {} characters**
runoff file template: 'R201702DFraCElse_{yyyymmdd}.nc'
# File containing monthly average conservative temperature clmatologies for NEMO runoff forcing
# File containing monthly average conservative temperature climatology for NEMO runoff forcing
temperature climatology: /nemoShare/MEOPAR/nowcast-sys/rivers-climatology/rivers_ConsTemp_month.nc
# Template for rivers forcing file names
# **Must be quoted to project {} characters**
Expand Down Expand Up @@ -938,10 +933,6 @@ vhfr fvcom runs:
time step: 0.5
# Directory on compute host where results are stored
results: /nemoShare/MEOPAR/SalishSea/fvcom-nowcast-x2/
forecast x2:
nemo boundary results: /nemoShare/MEOPAR/SalishSea/forecast/
time step: 0.5
results: /nemoShare/MEOPAR/SalishSea/fvcom-forecast-x2/
nowcast r12:
nemo boundary results: /nemoShare/MEOPAR/SalishSea/nowcast/
time step: 0.2
Expand All @@ -950,7 +941,6 @@ vhfr fvcom runs:
# in ddmmmyy/ directories; keyed by run type
results archive:
nowcast x2: /opp/fvcom/nowcast-x2/
forecast x2: /opp/fvcom/forecast-x2/
nowcast r12: /opp/fvcom/nowcast-r12/
# Name of the results file containing the tide gauge stations sea surface height time series
stations dataset filename:
Expand Down Expand Up @@ -1369,8 +1359,6 @@ message registry:
checklist key: FVCOM boundary conditions
success x2 nowcast: x2 configuration nowcast FVCOM boundary condition file created
failure x2 nowcast: x2 configuration nowcast FVCOM boundary condition file creation failed
success x2 forecast: x2 configuration forecast FVCOM boundary condition file created
failure x2 forecast: x2 configuration forecast FVCOM boundary condition file creation failed
success r12 nowcast: r12 configuration nowcast FVCOM boundary condition file created
failure r12 nowcast: r12 configuration forecast FVCOM boundary condition file creation failed
crash: make_fvcom_boundary worker crashed
Expand All @@ -1379,8 +1367,6 @@ message registry:
checklist key: FVCOM rivers forcing
success x2 nowcast: x2 configuration nowcast FVCOM rivers forcing file created
failure x2 nowcast: x2 configuration nowcast FVCOM rivers forcing file creation failed
success x2 forecast: x2 configuration forecast FVCOM rivers forcing file created
failure x2 forecast: x2 configuration forecast FVCOM rivers forcing file creation failed
success r12 nowcast: r12 configuration nowcast FVCOM rivers forcing file created
failure r12 nowcast: r12 configuration nowcast FVCOM rivers forcing file creation failed
crash: make_fvcom_atmos_forcing worker crashed
Expand All @@ -1389,8 +1375,6 @@ message registry:
checklist key: FVCOM atmospheric forcing
success x2 nowcast: x2 configuration nowcast FVCOM atmospheric forcing file created
failure x2 nowcast: x2 configuration nowcast FVCOM atmospheric forcing file creation failed
success x2 forecast: x2 configuration forecast FVCOM atmospheric forcing file created
failure x2 forecast: x2 configuration forecast FVCOM atmospheric forcing file creation failed
success r12 nowcast: r12 configuration nowcast FVCOM atmospheric forcing file created
failure r12 nowcast: r12 configuration nowcast FVCOM atmospheric forcing file creation failed
crash: make_fvcom_atmos_forcing worker crashed
Expand All @@ -1399,8 +1383,6 @@ message registry:
checklist key: FVCOM atmospheric forcing upload
success x2 nowcast: FVCOM x2 configuration nowcast atmospheric forcing file uploaded
failure x2 nowcast: FVCOM x2 configuration nowcast atmospheric forcing file upload failed
success x2 forecast: FVCOM x2 configuration forecast atmospheric forcing file uploaded
failure x2 forecast: FVCOM x2 configuration forecast atmospheric forcing file upload failed
success r12 nowcast: FVCOM r12 configuration nowcast atmospheric forcing file uploaded
failure r12 nowcast: FVCOM r12 configuration nowcast atmospheric forcing file upload failed
crash: upload_fvcom_atmos_forcing worker crashed
Expand All @@ -1409,8 +1391,6 @@ message registry:
checklist key: FVCOM run
success x2 nowcast: x2 configuration nowcast FVCOM run started
failure x2 nowcast: x2 configuration nowcast FVCOM run failed
success x2 forecast: x2 configuration forecast FVCOM run started
failure x2 forecast: x2 configuration forecast FVCOM run failed
success r12 nowcast: r12 configuration nowcast FVCOM run started
failure r12 nowcast: r12 configuration nowcast FVCOM run failed
crash: run_fvcom worker crashed
Expand All @@ -1420,8 +1400,6 @@ message registry:
need: FVCOM run information dict requested
success x2 nowcast: x2 configuration nowcast FVCOM run completed
failure x2 nowcast: x2 configuration nowcast FVCOM run failed
success x2 forecast: x2 configuration forecast FVCOM run completed
failure x2 forecast: x2 configuration forecast FVCOM run failed
success r12 nowcast: r12 configuration nowcast FVCOM run completed
failure r12 nowcast: r12 configuration nowcast FVCOM run failed
crash: watch_fvcom worker crashed
Expand Down Expand Up @@ -1489,6 +1467,12 @@ message registry:
failure nowcast-agrif: nowcast-agrif results files download failed
crash: download_results worker crashed

split_results:
checklist key: results splitting
success hindcast: hindcast results files split into daily directories
failure hindcast: hindcast results files splitting into daily directories failed
crash: split_results worker crashed

download_wwatch3_results:
checklist key: WWATCH3 results files
success forecast2: WaveWatch3 forecast2 results files downloaded
Expand All @@ -1503,22 +1487,12 @@ message registry:
checklist key: VHFR FVCOM results files
success x2 nowcast: VHFR FVCOM x2 configuration nowcast results files downloaded
failure x2 nowcast: VHFR FVCOM x2 configuration nowcast results files download failed
success x2 forecast: VHFR FVCOM x2 configuration forecast results files downloaded
failure x2 forecast: VHFR FVCOM x2 configuration forecast results files download failed
success r12 nowcast: VHFR FVCOM r12 configuration nowcast results files downloaded
failure r12 nowcast: VHFR FVCOM r12 configuration nowcast results files download failed
crash: download_fvcom_results worker crashed

split_results:
checklist key: results splitting
success hindcast: hindcast results files split into daily directories
failure hindcast: hindcast results files splitting into daily directories failed
crash: split_results worker crashed

update_forecast_datasets:
checklist key: update forecast datasets
success fvcom forecast: VHFR FVCOM most_recent_forecast files dataset updated from forecast run
failure fvcom forecast: VHFR FVCOM most_recent_forecast files dataset update from forecast run failed
success nemo forecast: NEMO forecast datasets updated from forecast run
failure nemo forecast: NEMO forecast datasets update from forecast run failed
success nemo forecast2: NEMO forecast datasets updated from forecast2 run
Expand Down Expand Up @@ -1553,8 +1527,6 @@ message registry:
failure fvcom-x2-nowcast: VHFR FVCOM x2 nowcast ERDDAP dataset flag files creation failed
success fvcom-r12-nowcast: VHFR FVCOM r12 nowcast ERDDAP dataset flag files created
failure fvcom-r12-nowcast: VHFR FVCOM r12 nowcast ERDDAP dataset flag files creation failed
success fvcom-forecast: VHFR FVCOM x2 forecast ERDDAP dataset flag files created
failure fvcom-forecast: VHFR FVCOM x2 forecast ERDDAP dataset flag files creation failed
crash: ping_erddap worker crashed

make_plots:
Expand All @@ -1575,14 +1547,10 @@ message registry:
failure nemo forecast2 publish: nemo forecast2 publish plots failed
success fvcom nowcast-x2 publish: fvcom x2 configuration nowcast publish plots produced
failure fvcom nowcast-x2 publish: fvcom x2 configuration nowcast publish plots failed
success fvcom forecast-x2 publish: fvcom x2 configuration forecast publish plots produced
failure fvcom forecast-x2 publish: fvcom x2 configuration forecast publish plots failed
success fvcom nowcast-r12 publish: fvcom r12 configuration nowcast publish plots produced
failure fvcom nowcast-r12 publish: fvcom r12 configuration nowcast publish plots failed
success fvcom nowcast-x2 research: fvcom x2 configuration nowcast research plots produced
failure fvcom nowcast-x2 research: fvcom x2 configuration nowcast research plots failed
success fvcom forecast-x2 research: fvcom x2 configuration forecast research plots produced
failure fvcom forecast-x2 research: fvcom x2 configuration forecast research plots failed
success fvcom nowcast-r12 research: fvcom r12 configuration nowcast research plots produced
failure fvcom nowcast-r12 research: fvcom r12 configuration nowcast research plots failed
success wwatch3 forecast publish: wwatch3 forecast publish plots produced
Expand Down
49 changes: 1 addition & 48 deletions nowcast/next_workers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -941,10 +941,8 @@ def after_make_fvcom_boundary(msg, config, checklist):
next_workers = {
"crash": [],
"failure x2 nowcast": [],
"failure x2 forecast": [],
"failure r12 nowcast": [],
"success x2 nowcast": [],
"success x2 forecast": [],
"success r12 nowcast": [],
}
if msg.type.startswith("success"):
Expand Down Expand Up @@ -995,10 +993,8 @@ def after_make_fvcom_rivers_forcing(msg, config, checklist):
next_workers = {
"crash": [],
"failure x2 nowcast": [],
"failure x2 forecast": [],
"failure r12 nowcast": [],
"success x2 nowcast": [],
"success x2 forecast": [],
"success r12 nowcast": [],
}
return []
Expand All @@ -1025,10 +1021,8 @@ def after_make_fvcom_atmos_forcing(msg, config, checklist):
next_workers = {
"crash": [],
"failure x2 nowcast": [],
"failure x2 forecast": [],
"failure r12 nowcast": [],
"success x2 nowcast": [],
"success x2 forecast": [],
"success r12 nowcast": [],
}
if msg.type.startswith("success"):
Expand Down Expand Up @@ -1066,10 +1060,8 @@ def after_upload_fvcom_atmos_forcing(msg, config, checklist):
next_workers = {
"crash": [],
"failure x2 nowcast": [],
"failure x2 forecast": [],
"failure r12 nowcast": [],
"success x2 nowcast": [],
"success x2 forecast": [],
"success r12 nowcast": [],
}
if msg.type.startswith("success"):
Expand Down Expand Up @@ -1108,10 +1100,8 @@ def after_run_fvcom(msg, config, checklist):
next_workers = {
"crash": [],
"failure x2 nowcast": [],
"failure x2 forecast": [],
"failure r12 nowcast": [],
"success x2 nowcast": [],
"success x2 forecast": [],
"success r12 nowcast": [],
}
if msg.type.startswith("success"):
Expand Down Expand Up @@ -1148,10 +1138,8 @@ def after_watch_fvcom(msg, config, checklist):
next_workers = {
"crash": [],
"failure x2 nowcast": [],
"failure x2 forecast": [],
"failure r12 nowcast": [],
"success x2 nowcast": [],
"success x2 forecast": [],
"success r12 nowcast": [],
}
if msg.type.startswith("success"):
Expand All @@ -1169,20 +1157,6 @@ def after_watch_fvcom(msg, config, checklist):
)
)
if run_type == "nowcast" and model_config == "x2":
next_workers[msg.type].append(
NextWorker(
"nowcast.workers.make_fvcom_boundary",
args=[
(config["vhfr fvcom runs"]["host"]),
"x2",
"forecast",
"--run-date",
msg.payload[f"{model_config} {run_type}"]["run date"],
],
host=(config["vhfr fvcom runs"]["host"]),
)
)
if run_type == "forecast" and model_config == "x2":
next_workers[msg.type].append(
NextWorker(
"nowcast.workers.make_fvcom_boundary",
Expand Down Expand Up @@ -1560,10 +1534,8 @@ def after_download_fvcom_results(msg, config, checklist):
next_workers = {
"crash": [],
"failure x2 nowcast": [],
"failure x2 forecast": [],
"failure r12 nowcast": [],
"success x2 nowcast": [],
"success x2 forecast": [],
"success r12 nowcast": [],
}
if msg.type.startswith("success"):
Expand Down Expand Up @@ -1594,13 +1566,6 @@ def after_download_fvcom_results(msg, config, checklist):
args=[f"fvcom-{model_config}-nowcast"],
)
)
if run_type == "forecast":
next_workers[msg.type].append(
NextWorker(
"nowcast.workers.update_forecast_datasets",
args=["fvcom", "forecast", "--run-date", run_date],
)
)
return next_workers[msg.type]


Expand Down Expand Up @@ -1650,12 +1615,10 @@ def after_update_forecast_datasets(msg, config, checklist):
"""
next_workers = {
"crash": [],
"failure fvcom forecast": [],
"failure nemo forecast": [],
"failure nemo forecast2": [],
"failure wwatch3 forecast": [],
"failure wwatch3 forecast2": [],
"success fvcom forecast": [],
"success nemo forecast": [],
"success nemo forecast2": [],
"success wwatch3 forecast": [],
Expand All @@ -1664,11 +1627,7 @@ def after_update_forecast_datasets(msg, config, checklist):
if msg.type.startswith("success"):
model = msg.type.split()[1]
run_type = msg.type.split()[2]
try:
run_date = checklist[f"{model.upper()} run"][run_type]["run date"]
except KeyError:
# FVCOM run has model config prefixed to run type
run_date = checklist[f"{model.upper()} run"][f"x2 {run_type}"]["run date"]
run_date = checklist[f"{model.upper()} run"][run_type]["run date"]
next_workers[msg.type].append(
NextWorker("nowcast.workers.ping_erddap", args=[f"{model}-forecast"])
)
Expand Down Expand Up @@ -1730,8 +1689,6 @@ def after_ping_erddap(msg, config, checklist):
"failure fvcom-x2-nowcast": [],
"success fvcom-r12-nowcast": [],
"failure fvcom-r12-nowcast": [],
"success fvcom-forecast": [],
"failure fvcom-forecast": [],
}
if msg.type == "success wwatch3-forecast":
run_types = checklist["WWATCH3 run"].keys()
Expand Down Expand Up @@ -1815,10 +1772,8 @@ def after_make_plots(msg, config, checklist):
"failure nemo forecast publish": [],
"failure nemo forecast2 publish": [],
"failure fvcom nowcast-x2 publish": [],
"failure fvcom forecast-x2 publish": [],
"failure fvcom nowcast-r12 publish": [],
"failure fvcom nowcast-x2 research": [],
"failure fvcom forecast-x2 research": [],
"failure fvcom nowcast-r12 research": [],
"failure wwatch3 forecast publish": [],
"failure wwatch3 forecast2 publish": [],
Expand All @@ -1830,10 +1785,8 @@ def after_make_plots(msg, config, checklist):
"success nemo forecast publish": [],
"success nemo forecast2 publish": [],
"success fvcom nowcast-x2 publish": [],
"success fvcom forecast-x2 publish": [],
"success fvcom nowcast-r12 publish": [],
"success fvcom nowcast-x2 research": [],
"success fvcom forecast-x2 research": [],
"success fvcom nowcast-r12 research": [],
"success wwatch3 forecast publish": [],
"success wwatch3 forecast2 publish": [],
Expand Down
Loading