Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combined pre-AGU GUI and Calibration Message Work #240

Open
wants to merge 169 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
169 commits
Select commit Hold shift + click to select a range
d9cfcf9
Fix fast-debug in dataservice image entrypoint.
robertbartel Oct 3, 2022
8106216
Add package client name vars to example.env.
robertbartel Sep 13, 2022
e7ad95d
Add new QueryType for dataset management messages.
robertbartel Sep 13, 2022
506e77b
Add GET_SERIALIZED_FORM query dataservice support.
robertbartel Oct 3, 2022
00cd95a
Refactor ExternalRequestClient async_make_request.
robertbartel Oct 3, 2022
314e36b
Adding client methods for data/metadata retrieval.
robertbartel Oct 12, 2022
30cbcca
Refactor dataset request handler to use try block.
robertbartel Sep 13, 2022
b516c5c
Add get_serialized_datasets to external client.
robertbartel Sep 13, 2022
8445296
Improve DatasetExternalClient.
robertbartel Oct 3, 2022
a2927b2
Bump dataservice dependencies.
robertbartel Sep 13, 2022
f23c721
Fix LIST_FILES query response bug in dataservice.
robertbartel Oct 3, 2022
8e45fbd
Update dataset manager abstract interface.
robertbartel Oct 12, 2022
60c6e8e
More QueryType elements and update DatasetQuery.
robertbartel Oct 12, 2022
c893164
Update dataset message to support start and size.
robertbartel Oct 12, 2022
a9dbd5b
Bump requestservice dependencies.
robertbartel Sep 13, 2022
4198db5
Bump requestservice to 0.6.0.
robertbartel Sep 13, 2022
4444b62
Optimize object store dataset reloading.
robertbartel Oct 12, 2022
7125fb8
Update object store manager for interface changes.
robertbartel Oct 12, 2022
9024a0c
Update modeldata dependency versions.
robertbartel Oct 12, 2022
6ecaced
Update client unit tests for interface changes.
robertbartel Oct 12, 2022
dbbcf5a
Bump client package version to 0.2.0.
robertbartel Oct 12, 2022
f851e75
Update dataservice response handling.
robertbartel Oct 12, 2022
f1ecc2c
Update dataservice package dependency versions.
robertbartel Oct 12, 2022
ab51dbf
Add dataset view class and static page for GUI.
robertbartel Sep 13, 2022
60fbab5
Refactor static dir usage in DMODProxy.py.
robertbartel Sep 13, 2022
89179ad
Add Django GUI url def for dataset view.
robertbartel Sep 13, 2022
28e0168
Add client package build arg for GUI Docker build.
robertbartel Sep 13, 2022
6df5754
Add second view and navigation.
robertbartel Sep 29, 2022
f68bbd2
Have DatasetManagementView.py send DS as list.
robertbartel Sep 30, 2022
64957ad
Update dataset management GUI with details view.
robertbartel Sep 30, 2022
bb90be6
Cleanup after changing implementation approach.
robertbartel Sep 30, 2022
7472e56
Create AbstractDatasetView GUI Django view class.
robertbartel Oct 3, 2022
4715f65
Add Django view for direct dataservice API calls.
robertbartel Oct 3, 2022
12d89b2
Add GUI Django url for Dataset API AJAX calls.
robertbartel Oct 3, 2022
88db907
Add commented-out helper debug volume for GUI.
robertbartel Oct 3, 2022
1cb89b0
Refactor DatasetManagementView inheritance.
robertbartel Oct 3, 2022
2848189
More progress on dataset management template.
robertbartel Oct 3, 2022
2e46f7d
Add dataset delete support to Django API view.
robertbartel Oct 3, 2022
0d0c4e2
Implement dataset delete in management view.
robertbartel Oct 3, 2022
bb5468f
Adding Javascript component helper classes.
robertbartel Oct 12, 2022
288e7a7
Update dataset API view.
robertbartel Oct 12, 2022
af0434d
Update dataset management view template.
robertbartel Oct 12, 2022
6e566a9
Add incomplete DatasetFileWebsocketFilelike.
robertbartel Oct 13, 2022
36a044c
Add generate_domain helper func to Hydrofabric.
robertbartel Oct 31, 2022
4c8c50a
Add specialized geojson subset handler subclass.
robertbartel Oct 31, 2022
9ace2f3
Update SubsetHandler unit tests for new type.
robertbartel Oct 31, 2022
e14474a
Update subsetservice package dependencies.
robertbartel Oct 31, 2022
bcd1cc5
Let subset-srv Cli use GeoJsonBackedSubsetHandler.
robertbartel Oct 31, 2022
d00c015
Add log and debug support to subset service main.
robertbartel Oct 31, 2022
b47516c
Adjust subsetservice to multiple geojson handlers.
robertbartel Oct 31, 2022
4677634
Add subsetservice API to get geojson for bounds.
robertbartel Oct 31, 2022
dac6dde
Refactor and enhance DockerS3FSPluginHelper.
robertbartel Oct 31, 2022
a6a5d0f
Improve how dataservice indicates Docker is used.
robertbartel Oct 31, 2022
4820dc6
Add async func to manage hydrofabric datasets.
robertbartel Oct 31, 2022
69410ce
Add async task for new hydrofabric manage func.
robertbartel Oct 31, 2022
8fc4226
Update/centralize some static GUI settings.
robertbartel Oct 31, 2022
8e08d82
Update request_client to support overwrite.
robertbartel Oct 31, 2022
8747b62
Updates to MapView classes hydrofabric support.
robertbartel Oct 31, 2022
395c075
Updates to GUI crosswalk hydrofabric support.
robertbartel Oct 31, 2022
ec006f3
Use centralized static settings in DMODProxy.
robertbartel Oct 31, 2022
7e2e48e
Updates to map.js for large hydrofabric support.
robertbartel Oct 31, 2022
78b893b
Fix migration support in app_server entrypoint.
robertbartel Oct 31, 2022
95fe7bc
Add debug support to subsetservice entrypoint.
robertbartel Oct 31, 2022
cc475c9
Work on GUI service docker config.
robertbartel Oct 31, 2022
8dbf884
Debug/hydrofabric in subsetservice Docker config.
robertbartel Oct 31, 2022
2c7f5d7
Fix silly blob/glob Path usage bug.
robertbartel Nov 2, 2022
8b02916
Have Docker S3FS helper check for existing vols.
robertbartel Nov 4, 2022
e11432a
Let Docker S3FS helper remove existing service.
robertbartel Nov 4, 2022
02d369d
Let Docker S3FS helper sanity check vol better.
robertbartel Nov 4, 2022
1e4e308
Adjust/fix S3FS helper service management logic.
robertbartel Nov 4, 2022
e07bf7b
Adjust S3FS helper use to remove existing service.
robertbartel Nov 4, 2022
13d4b5e
Fix bug in obj store vol script for sentinel.
robertbartel Nov 4, 2022
4174aac
Fix strange issue with DockerFile for vol helper.
robertbartel Nov 4, 2022
4350a85
Fixes to get dataservice vol creation working.
robertbartel Nov 4, 2022
5b4017c
Fix GUI migration setting up user to strip spaces.
robertbartel Nov 27, 2022
9bfaeaf
Refactor GUI settings for special GUI directories.
robertbartel Nov 27, 2022
a69b689
Adjust Django settings to avoid dev CSRF issues.
robertbartel Nov 27, 2022
bfab11f
Account for new hydrofabric dir setting in map.
robertbartel Nov 27, 2022
83b0d62
Undo zoom-based subset lookup logic in map.js.
robertbartel Nov 27, 2022
263124a
Fix map.js issue with newer hydrofabrics.
robertbartel Nov 27, 2022
af4fb5d
Add/fix numpy as explicit GUI dependency.
robertbartel Nov 27, 2022
32e3634
Address GUI stack config issues.
robertbartel Nov 27, 2022
ad59015
Remove deprecated Forcing "path" GUI form field.
robertbartel Nov 29, 2022
b5e9ad5
Add "Resources" and "Requested CPUs" to GUI form.
robertbartel Nov 29, 2022
f8a9506
Change color scheme of GUI config form.
robertbartel Nov 29, 2022
18c33d3
Fix GUI template form input id bug.
robertbartel Nov 29, 2022
7667a68
Add DATE_TIME_FORMAT to GUI settings.py.
robertbartel Nov 29, 2022
c988569
Update GUI migrations for specific formulations.
robertbartel Dec 6, 2022
d110c57
Adjust ngen config form field ids for parsing.
robertbartel Dec 6, 2022
9467e08
Updates to GUI ngen config form exec logic.
robertbartel Dec 6, 2022
9670d93
Fixing bugs in init of members in DmodClient.
robertbartel Dec 6, 2022
433bccb
Additional formulation record.
robertbartel Dec 7, 2022
261b5e8
Fix additional Formulation template description.
robertbartel Dec 8, 2022
4358a85
Fix GUI migrations issue from upstream changes.
robertbartel Nov 23, 2022
1d01bd6
Add minio Python package as dependency of GUI.
robertbartel Nov 17, 2022
8c6b6d5
Add several static GUI dir and minio settings.
robertbartel Nov 17, 2022
99f2257
Update GUI Docker stack config for minio access.
robertbartel Nov 17, 2022
bfb7071
Update Dataset API view to cache and zip datasets.
robertbartel Nov 17, 2022
c535e5a
Add view and url for downloading zipped dataset.
robertbartel Nov 17, 2022
aa09993
Update dataset_management.html for downloads.
robertbartel Nov 17, 2022
c8857dd
Update DatasetApiView function to download file.
robertbartel Nov 17, 2022
594124f
Add Django url and view to download dataset file.
robertbartel Nov 17, 2022
c2b3f9e
Fix dataset mgr Django template file downloads.
robertbartel Nov 17, 2022
8431a9b
Fix bug in GUI DatasetApiView for caching dataset.
robertbartel Nov 18, 2022
09d79a5
Comment out upload files to existing dataset.
robertbartel Nov 18, 2022
c02595a
fix dynamic form field node removal in CreateDatasetForm
aaraney Oct 28, 2022
d2ccca1
add django forms for DatasetManagementView.
aaraney Oct 28, 2022
cff2c6e
add utility method to for retrieving dynamic form from enum using its…
aaraney Oct 28, 2022
a1a41aa
move DatasetFormatForm to bottom of DatasetManagementForms mod to mak…
aaraney Oct 28, 2022
f6313d7
date time field accepts partial ISO8061 format. this is a work around…
aaraney Oct 28, 2022
223fe8b
render forms in template
aaraney Oct 28, 2022
cb545a6
add dataset form css
aaraney Oct 28, 2022
13891a1
add custom id and class to form fields
aaraney Oct 28, 2022
98fdabf
wire-up labels in template and add comments about how form input's an…
aaraney Oct 28, 2022
6307a20
add dynamic form mix in. sets custom id, class, and style to form fields
aaraney Oct 28, 2022
3f24333
hide dynamic form labels by default
aaraney Oct 28, 2022
2fef3b8
disable all dynamic fields by default. fields must be disabled to sub…
aaraney Oct 28, 2022
86a665c
add onchange for toggling on / off dynamic form fields
aaraney Oct 28, 2022
ad33b7c
render form and dynamic forms in DatasetManagementView
aaraney Oct 28, 2022
e60bee4
add trailing colon to labels
aaraney Oct 28, 2022
d5c5f9f
forgot to instantiate some form fields
aaraney Oct 28, 2022
ecc0b75
add slight gap between items
aaraney Oct 28, 2022
981df94
add default noop data format choice
aaraney Oct 28, 2022
90fde59
add file upload form field
aaraney Oct 31, 2022
eea3836
add filename validation, cannot include '_'. Minio does not allow '_'
aaraney Oct 31, 2022
77c3132
update file fields' to use input validation
aaraney Oct 31, 2022
73f03fb
add js utility module
aaraney Oct 31, 2022
be3585e
add start time end time field validation
aaraney Oct 31, 2022
9b5ce48
Move minio client factory to AbstractDatasetView.
robertbartel Nov 23, 2022
9354115
Cleanup refactor of DatasetApiView.py.
robertbartel Nov 23, 2022
bd2eaae
Fix create form in dataset_management.html.
robertbartel Nov 23, 2022
908bdbc
Adjust Django form for dataset create.
robertbartel Nov 23, 2022
3344061
Ensure GUI app_server image installs dmod-core.
robertbartel Nov 23, 2022
5c6e754
Update dataset mgr view with full create support.
robertbartel Nov 23, 2022
6b9a4da
Workaround for CSRF issues.
robertbartel Nov 23, 2022
64a34ec
Fix issue with underscore in dataset file names.
robertbartel Nov 28, 2022
41ae29a
Fix dataset upload issue for larger files.
robertbartel Nov 28, 2022
0c1a830
Fix nginx config issue for uploading larger files.
robertbartel Nov 28, 2022
8b3d0a5
Fix bug w/ converting time format from GUI input.
robertbartel Nov 28, 2022
574290a
Add OS dependency geos-dev to py-deps image.
robertbartel Jan 5, 2023
fd40868
Enhance deserialization of NGENRequest.
robertbartel Nov 1, 2022
5f5a988
Add calibration request messages.
robertbartel Nov 1, 2022
d15c445
Add new types to communication package __init__.
robertbartel Nov 1, 2022
02dabb1
Fix init of existing request handlers.
robertbartel Nov 2, 2022
f987018
Fix tests for request handler init updates.
robertbartel Nov 2, 2022
dd14e4d
Bug in request handler service_client lazy init.
robertbartel Nov 2, 2022
d807dd8
Add new NgenCalibrationRequestHandler class.
robertbartel Nov 2, 2022
ebfeafe
Add means for type-specific handler preprocessing.
robertbartel Nov 2, 2022
e2b93de
Update requestservice internal deps versions.
robertbartel Nov 2, 2022
eb1eec0
Update requestservice for changes to handler init.
robertbartel Nov 2, 2022
5cde39e
resolve external requests init conflict
aaraney Dec 1, 2022
872611e
resolve request message import conflict in request service
aaraney Dec 1, 2022
1845045
fix merge conflict
aaraney Dec 1, 2022
188be71
add ngen-calibration dockerfile and entrypoint
aaraney Dec 6, 2022
558126d
recursively search for available model subclasses
aaraney Dec 6, 2022
59afb07
add ngen-cal model_name to NgenCalibrationRequest type
aaraney Dec 6, 2022
c7e4f18
deserialize NgenCalibrationRequest evaluation_time_range
aaraney Dec 6, 2022
5750e7a
fix NgenCalibrationResponse subclass type.
aaraney Dec 6, 2022
838a3ad
add missing await.
aaraney Dec 6, 2022
44ff64f
add support for ngen calibration jobs in scheduler
aaraney Dec 6, 2022
b87156e
dont add data to datasets as temporary. causes s3 Bucket is missing …
aaraney Dec 6, 2022
7447465
NOTE: MAY REMOVE LATER. Hard coded dmod.client code for submitting ng…
aaraney Dec 6, 2022
2a4a922
Fix bug with duplicate kwargs for NGENRequest.
robertbartel Jan 11, 2023
300c1bf
Fix hydrofab subset test for abstraction change.
robertbartel Jan 11, 2023
9a8345d
Fix request service arg handling of eval details.
robertbartel Jan 20, 2023
89c1c4a
Add eval ENV vars to request-service def.
robertbartel Jan 20, 2023
6bf1539
Update requestservice entrypoint w/ eval settings.
robertbartel Jan 20, 2023
175d48d
Fix bugs/spacing in request srv handler for eval.
robertbartel Jan 20, 2023
d03e219
Bump request-service package version.
robertbartel Jan 20, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docker/main/dataservice/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ if [ -d ${UPDATED_PACKAGES_DIR:=/updated_packages} ]; then
for srv in $(pip -qq freeze | grep dmod | awk -F= '{print $1}' | awk -F- '{print $2}'); do
if [ $(ls ${UPDATED_PACKAGES_DIR} | grep dmod.${srv}- | wc -l) -eq 1 ]; then
pip uninstall -y --no-input $(pip -qq freeze | grep dmod.${srv} | awk -F= '{print $1}')
pip install $(ls ${UPDATED_PACKAGES_DIR}/*.whl | grep dmod.${srv}-)
pip install --no-deps $(ls ${UPDATED_PACKAGES_DIR}/*.whl | grep dmod.${srv}-)
fi
done
#pip install ${UPDATED_PACKAGES_DIR}/*.whl
Expand Down
13 changes: 12 additions & 1 deletion docker/main/docker-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,9 @@ services:
- DATA_SERVICE_ENDPOINT_HOST=${DOCKER_REQUESTS_DATASERVICE_ENDPOINT_HOST:-data-service}
- DATA_SERVICE_ENDPOINT_PORT=${DOCKER_DATASERVICE_CONTAINER_PORT:-3015}
- DATA_SERVICE_CLIENT_SSL_DIR=${DOCKER_REQUESTS_CONTAINER_DATASERVICE_CLIENT_SSL_DIR:-/ssl/dataservice}
- EVALUATION_SERVICE_ENDPOINT_HOST=${DOCKER_REQUESTS_EVALUATIONSERVICE_ENDPOINT_HOST:-evaluation-service}
- EVALUATION_SERVICE_ENDPOINT_PORT=${DOCKER_EVALUATIONSERVICE_CONTAINER_PORT:-3015}
- EVALUATION_SERVICE_CLIENT_SSL_DIR=${DOCKER_REQUESTS_CONTAINER_EVALUATIONSERVICE_CLIENT_SSL_DIR:-/ssl/evaluationservice}
- PARTITIONER_SERVICE_ENDPOINT_HOST=${DOCKER_REQUESTS_PARTITIONERSERVICE_ENDPOINT_HOST:-partitioner-service}
- PARTITIONER_SERVICE_ENDPOINT_PORT=${DOCKER_PARTITIONER_SERVICE_CONTAINER_PORT:-3014}
- PARTITIONER_SERVICE_CLIENT_SSL_DIR=${DOCKER_REQUESTS_CONTAINER_PARTITIONERSERVICE_CLIENT_SSL_DIR:-/ssl/partitionerservice}
Expand All @@ -114,7 +117,9 @@ services:
ports:
- ${DOCKER_SUBSET_API_PORT:-5001}:${DOCKER_SUBSET_CONTAINER_PORT:-5001}
volumes:
- ${HYDROFABRIC_DATA_DIR:?Please set HYDROFABRIC_DATA_DIR for Docker environment in .env config file}:/hydrofabric_data
# TODO: even if this works, need to make it configurable
- hydrofabric:/hydrofabric_data/${DMOD_HYDROFABRIC_VOLUME:?}
#- ${HYDROFABRIC_DATA_DIR:?Please set HYDROFABRIC_DATA_DIR for Docker environment in .env config file}:/hydrofabric_data
# This typically needs to be commented out; intended for development use (see related 'environment' config above)
#- updated_packages:${UPDATED_PACKAGES_CONTAINER_DIR:?Check if updated packages directory should be used}
deploy:
Expand All @@ -128,6 +133,9 @@ services:
- LISTEN_PORT=${DOCKER_SUBSET_CONTAINER_PORT:-5000}
- SERVICE_PACKAGE_NAME=${PYTHON_PACKAGE_NAME_SUBSET_SERVICE:?}
- FILES_DIRECTORY=/hydrofabric_data
- PYCHARM_REMOTE_DEBUG_ACTIVE=${PYCHARM_REMOTE_DEBUG_SUBSET_SERVICE_ACTIVE:-false}
- PYCHARM_REMOTE_DEBUG_SERVER_HOST=${PYCHARM_REMOTE_DEBUG_SERVER_HOST:-host.docker.internal}
- PYCHARM_REMOTE_DEBUG_SERVER_PORT=${PYCHARM_REMOTE_DEBUG_SERVER_PORT_SUBSET_SERVICE:-55874}
# This typically needs to be commented out; intended for development use (see related 'volume' config below)
#- UPDATED_PACKAGES_DIR=${UPDATED_PACKAGES_CONTAINER_DIR:?Updated packages directory not set, make sure this should be active}
entrypoint: ["python3", "-m", "${PYTHON_PACKAGE_NAME_SUBSET_SERVICE:?}"]
Expand Down Expand Up @@ -226,6 +234,9 @@ volumes:
# configs above for several services)
updated_packages:
external: true
hydrofabric:
name: ${DMOD_HYDROFABRIC_VOLUME:?}
external: true

secrets:
myredis_pass:
Expand Down
38 changes: 38 additions & 0 deletions docker/main/ngen-calibration/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
ARG DOCKER_INTERNAL_REGISTRY

FROM ${DOCKER_INTERNAL_REGISTRY}/ngen:latest

ARG WORKDIR=/ngen
ARG USER=mpi
ARG NGEN_CAL_BRANCH=master
ARG NGEN_CAL_COMMIT

ENV USER=${USER} USER_HOME=/home/${USER}

WORKDIR ${WORKDIR}
USER ${USER}

# try NGEN_CAL_COMMIT, if not set or empty, use NGEN_CAL_BRANCH
RUN pip install "git+https://github.com/noaa-owp/ngen-cal@${NGEN_CAL_COMMIT:-${NGEN_CAL_BRANCH}}#egg=ngen_cal&subdirectory=python/ngen_cal"

COPY --chown=${USER} entrypoint.sh ${WORKDIR}

# Change permissions for entrypoint and make sure dataset volume mount parent directories exists
RUN chmod +x ${WORKDIR}/entrypoint.sh \
&& for d in ${DATASET_DIRECTORIES}; do mkdir -p /dmod/datasets/${d}; done \
&& for d in noah-owp-modular topmodel cfe sloth 'evapotranspiration/evapotranspiration'; do \
if [ -d ${WORKDIR}/ngen/extern/${d}/cmake_build ]; then \
cp -a ${WORKDIR}/ngen/extern/${d}/cmake_build/*.so* /dmod/shared_libs/.; \
fi; \
done \
&& ( cp -a ${WORKDIR}/ngen/cmake_build_parallel/ngen /dmod/bin/ngen-parallel || true ) \
&& ( cp -a ${WORKDIR}/ngen/cmake_build_serial/ngen /dmod/bin/ngen-serial || true ) \
&& ( cp -a ${WORKDIR}/ngen/cmake_build/partitionGenerator /dmod/bin/partitionGenerator || true ) \
&& pushd /dmod/bin \
# NOTE use of `ln -sf`. \
&& ( ( stat ngen-parallel && ln -sf ngen-parallel ngen ) || ( stat ngen-serial && ln -sf ngen-serial ngen ) ) \
&& popd

ENV PATH=${WORKDIR}:$PATH
ENTRYPOINT ["entrypoint.sh"]
CMD [""]
119 changes: 119 additions & 0 deletions docker/main/ngen-calibration/entrypoint.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
#!/bin/sh
# Managed by the _generate_docker_cmd_args function in scheduler.py of dmod.scheduler
#
# $1 will have the number of nodes associated with this run
# $2 will have comma-delimited host strings in MPI form; e.g., hostname:N,hostname:M
# $3 will have the unique job id
# $4 is the worker index
# $5 will be the name of the output dataset (which will imply a directory location)
# $6 will be the name of the hydrofabric dataset (which will imply a directory location)
# $7 will be the name of the realization configuration dataset (which will imply a directory location)
# $8 will be the name of the BMI configuration dataset (which will imply a directory location)
# $9 will be the name of the partition configuration dataset (which will imply a directory location)
# TODO: wire up $10
# $10 will be the name of the calibration configuration dataset (which will imply a directory location)

# Not yet supported
# no-op
MPI_NODE_COUNT="${1:?No MPI node count given}"
# no-op
MPI_HOST_STRING="${2:?No MPI host string given}"
# no-op
PARTITION_DATASET_NAME="${9:?}"

JOB_ID=${3:?No Job id given}
WORKER_INDEX=${4:?No worker index given}

OUTPUT_DATASET_NAME="${5:?}"
HYDROFABRIC_DATASET_NAME="${6:?}"
REALIZATION_CONFIG_DATASET_NAME="${7:?}"
BMI_CONFIG_DATASET_NAME="${8:?}"
CALIBRATION_CONFIG_DATASET_NAME="${10:?}"

ACCESS_KEY_SECRET="object_store_exec_user_name"
SECRET_KEY_SECRET="object_store_exec_user_passwd"
DOCKER_SECRETS_DIR="/run/secrets"
ACCESS_KEY_FILE="${DOCKER_SECRETS_DIR}/${ACCESS_KEY_SECRET}"
SECRET_KEY_FILE="${DOCKER_SECRETS_DIR}/${SECRET_KEY_SECRET}"

NGEN_EXECUTABLE="/ngen/ngen/cmake_build/ngen"

ALL_DATASET_DIR="/dmod/datasets"
OUTPUT_DATASET_DIR="${ALL_DATASET_DIR}/output/${OUTPUT_DATASET_NAME}"
HYDROFABRIC_DATASET_DIR="${ALL_DATASET_DIR}/hydrofabric/${HYDROFABRIC_DATASET_NAME}"
REALIZATION_CONFIG_DATASET_DIR="${ALL_DATASET_DIR}/config/${REALIZATION_CONFIG_DATASET_NAME}"
BMI_CONFIG_DATASET_DIR="${ALL_DATASET_DIR}/config/${BMI_CONFIG_DATASET_NAME}"
PARTITION_DATASET_DIR="${ALL_DATASET_DIR}/config/${PARTITION_DATASET_NAME}"
CALIBRATION_CONFIG_DATASET_DIR="${ALL_DATASET_DIR}/config/${CALIBRATION_CONFIG_DATASET_NAME}"

print_date() {
date "+%Y-%m-%d,%H:%M:%S"
}

check_for_dataset_dir() {
# Dataset dir is $1
_CATEG="$(echo "${1}" | sed "s|${ALL_DATASET_DIR}/\([^/]*\)/.*|\1|" | awk '{print toupper($0)}')"
if [ ! -d "${1}" ]; then
echo "Error: expected ${_CATEG} dataset directory ${1} not found." 2>&1
exit 1
fi
}

load_object_store_keys_from_docker_secrets() {
# Read Docker Secrets files for Object Store access, if they exist
if [ -z "${ACCESS_KEY_FILE:-}" ]; then
echo "WARN: Cannot load object store access key when Docker secret file name not set"
elif [ -e "${ACCESS_KEY_FILE}" ]; then
ACCESS_KEY="$(cat "${ACCESS_KEY_FILE}")"
else
echo "WARN: Cannot load object store access key when Docker secret file does not exist"
fi

if [ -z "${SECRET_KEY_FILE:-}" ]; then
echo "WARN: Cannot load object store secret key when Docker secret file name not set"
elif [ -e "${SECRET_KEY_FILE}" ]; then
SECRET_KEY="$(cat "${SECRET_KEY_FILE}")"
else
echo "WARN: Cannot load object store secret key when Docker secret file does not exist"
fi

test -n "${ACCESS_KEY:-}" && test -n "${SECRET_KEY:-}"
}

start_calibration() {
# Start ngen calibration
echo "$(print_date) Starting serial ngen calibration"
# CALIBRATION_CONFIG_FILE=${CALIBRATION_CONFIG_DATASET_DIR}/$(basename $(find ${CALIBRATION_CONFIG_DATASET_DIR} -name "*.yaml" -maxdepth 1 | head -1))

# TODO: move this to CALIBRATION_CONFIG_DATASET_DIR
# NOTE: assumes that calibration dataset will be in realization config dataset AND that it is
# the only yaml file at the top level of that dataset.
CALIBRATION_CONFIG_FILE=${REALIZATION_CONFIG_DATASET_DIR}/$(basename $(find ${REALIZATION_CONFIG_DATASET_DIR} -name "*.yaml" -maxdepth 1 | head -1))

if [ -z "${CALIBRATION_CONFIG_FILE}" ]; then
echo "Error: NGEN calibration yaml file not found" 2>&1
exit 1
fi
python3 -m ngen.cal "${CALIBRATION_CONFIG_FILE}"

#Capture the return value to use as service exit code
NGEN_RETURN=$?

echo "$(print_date) ngen calibration finished with return value: ${NGEN_RETURN}"

# Exit with the model's exit code
return ${NGEN_RETURN}
}

# Sanity check that the output, hydrofabric, and config datasets are available (i.e., their directories are in place)
check_for_dataset_dir "${REALIZATION_CONFIG_DATASET_DIR}"
check_for_dataset_dir "${BMI_CONFIG_DATASET_DIR}"
check_for_dataset_dir "${PARTITION_DATASET_DIR}"
check_for_dataset_dir "${HYDROFABRIC_DATASET_DIR}"
check_for_dataset_dir "${OUTPUT_DATASET_DIR}"
# check_for_dataset_dir "${CALIBRATION_CONFIG_DATASET_DIR}"

# Move to the output dataset mounted directory
cd ${OUTPUT_DATASET_DIR}

start_calibration
3 changes: 3 additions & 0 deletions docker/main/requestservice/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ python -m ${SERVICE_PACKAGE_NAME:?} \
--data-service-host ${DATA_SERVICE_ENDPOINT_HOST:?} \
--data-service-port ${DATA_SERVICE_ENDPOINT_PORT:?} \
--data-service-ssl-dir ${DATA_SERVICE_CLIENT_SSL_DIR:?} \
--evaluation-service-host ${EVALUATION_SERVICE_ENDPOINT_HOST:?} \
--evaluation-service-port ${EVALUATION_SERVICE_ENDPOINT_PORT:?} \
--evaluation-service-ssl-dir ${EVALUATION_SERVICE_CLIENT_SSL_DIR:?} \
--partitioner-service-host ${PARTITIONER_SERVICE_ENDPOINT_HOST:?} \
--partitioner-service-port ${PARTITIONER_SERVICE_ENDPOINT_PORT:?} \
--partitioner-service-ssl-dir ${PARTITIONER_SERVICE_CLIENT_SSL_DIR:?}
4 changes: 2 additions & 2 deletions docker/main/s3fs-volume-helper/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
FROM alpine:3.15

RUN apk update && apk upgrade && apk add docker bash && mkdir -p /dmod/scripts
RUN apk update && apk upgrade && apk add docker bash && mkdir -p /dmod

COPY ./scripts/* /dmod/scripts/.
COPY ./scripts /dmod/scripts

ENV OUT_OF_GIT_REPO='true'

Expand Down
19 changes: 18 additions & 1 deletion docker/main/subsetservice/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,23 @@ if [ -n "${VENV_DIR:-}" ]; then
pip install --update -r /code/requirements.txt
fi

# Install for debugging when appropriate
if [ "$(echo "${PYCHARM_REMOTE_DEBUG_ACTIVE:-false}" | tr '[:upper:]' '[:lower:]' | tr -d '[:space:]')" = "true" ]; then
_DEBUG_ARG="--pycharm-remote-debug"
fi

# Handle some things in any cases when there is debugging
if [ -n "${_DEBUG_ARG:-}" ]; then
# Append these as well if appropriate, though defaults are coded (and they are somewhat agnostic to the debug setup)
if [ -n "${PYCHARM_REMOTE_DEBUG_SERVER_HOST:-}" ]; then
_DEBUG_ARG="${_DEBUG_ARG:-} --remote-debug-host ${PYCHARM_REMOTE_DEBUG_SERVER_HOST}"
fi

if [ -n "${PYCHARM_REMOTE_DEBUG_SERVER_PORT:-}" ]; then
_DEBUG_ARG="${_DEBUG_ARG:-} --remote-debug-port ${PYCHARM_REMOTE_DEBUG_SERVER_PORT}"
fi
fi

# If we find this directory, and if there are wheels in it, then install those
if [ -d ${UPDATED_PACKAGES_DIR:=/updated_packages} ]; then
if [ $(ls ${UPDATED_PACKAGES_DIR}/*.whl | wc -l) -gt 0 ]; then
Expand Down Expand Up @@ -44,4 +61,4 @@ fi

#set +e
#export PYTHONASYNCIODEBUG=1
python3 -m ${SERVICE_PACKAGE_NAME:?} ${args}
python3 -m ${SERVICE_PACKAGE_NAME:?} ${_DEBUG_ARG:-} ${args}
4 changes: 3 additions & 1 deletion docker/nwm_gui/app_server/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,14 @@ RUN pip install -r dependencies.txt
ENV PYTHONUNBUFFERED 1

# Slurp (or set default) wheel package names ...
ARG core_package_name=dmod-core
ARG comms_package_name=dmod-communication
ARG client_package_name=dmod-client

# Copy custom built packages from external sources image
COPY --from=sources /DIST /DIST
RUN pip install --upgrade --find-links=/DIST ${comms_package_name} \
RUN pip install --upgrade --find-links=/DIST ${core_package_name} \
&& pip install --upgrade --find-links=/DIST ${comms_package_name} \
&& pip install --upgrade --find-links=/DIST ${client_package_name} \
# After eventually installing all dist files like this, clean up ... \
&& rm -rf /DIST
Expand Down
4 changes: 4 additions & 0 deletions docker/nwm_gui/app_server/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ echo "Starting dmod app"
#Extract the DB secrets into correct ENV variables
POSTGRES_SECRET_FILE="/run/secrets/${DOCKER_SECRET_POSTGRES_PASS:?}"
export SQL_PASSWORD="$(cat ${POSTGRES_SECRET_FILE})"
export DMOD_SU_PASSWORD="$(cat ${POSTGRES_SECRET_FILE})"

# Execute the migration scripts on the designated database
python manage.py migrate

# Handle for debugging when appropriate
if [ "$(echo "${PYCHARM_REMOTE_DEBUG_ACTIVE:-false}" | tr '[:upper:]' '[:lower:]' | tr -d '[:space:]')" == "true" ]; then
Expand Down
24 changes: 22 additions & 2 deletions docker/nwm_gui/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ services:
args:
docker_internal_registry: ${DOCKER_INTERNAL_REGISTRY:?Missing DOCKER_INTERNAL_REGISTRY value (see 'Private Docker Registry ' section in example.env)}
comms_package_name: ${PYTHON_PACKAGE_DIST_NAME_COMMS:?}
client_package_name: ${PYTHON_PACKAGE_DIST_NAME_CLIENT:?}
networks:
- request-listener-net
# Call this when starting the container
Expand All @@ -42,6 +43,8 @@ services:
restart: on-failure
secrets:
- postgres_password
- object_store_exec_user_name
- object_store_exec_user_passwd
environment:
#- VENV_DIR=${DOCKER_GUI_CONTAINER_VENV_DIR:-}
#- CERT_PATH # used by dispatch.py
Expand All @@ -50,18 +53,29 @@ services:
- MAAS_ENDPOINT_PORT=${DOCKER_REQUESTS_HOST_PORT:-3012}
- MAAS_PORTAL_DEBUG_HOST=${PYCHARM_REMOTE_DEBUG_SERVER_HOST:-host.docker.internal}
- MAAS_PORTAL_DEBUG_PORT=${PYCHARM_REMOTE_DEBUG_SERVER_PORT_GUI:-55875}
- GUI_SUBSET_SERVICE_API_URL=http://${DMOD_GUI_SUBSET_SERVICE_API_HOST:?}:${DOCKER_SUBSET_API_PORT:-5001}
- PYCHARM_REMOTE_DEBUG_ACTIVE=${PYCHARM_REMOTE_DEBUG_GUI_ACTIVE:-false}
- PYCHARM_REMOTE_DEBUG_VERSION=${PYCHARM_REMOTE_DEBUG_VERSION:-~=211.7628.24}
- DMOD_SU_NAME=dmod_db_admin
# TODO: this needs to be changed eventually
- [email protected]
- SQL_ENGINE=django.db.backends.postgresql
- SQL_DATABASE=${DMOD_GUI_POSTGRES_DB:-dmod_dev}
- SQL_USER=${DMOD_GUI_POSTGRES_USER:?}
- SQL_HOST=db
- SQL_PORT=5432
- DMOD_SU_NAME=dmod_super_user
- [email protected]
- DATABASE=postgres
- DOCKER_SECRET_POSTGRES_PASS=postgres_password
- OBJECT_STORE_HOSTNAME=${EXTERNAL_OBJECT_STORE_HOSTNAME:?}
- OBJECT_STORE_PORT=${EXTERNAL_OBJECT_STORE_PORT:?}
- DMOD_GUI_CSRF_TRUSTED_ORIGINS=${DMOD_GUI_CSRF_TRUSTED_ORIGINS:?No CSRF trusted origins configured (provide '' at least)}
volumes:
- ${DMOD_APP_STATIC:?}:/usr/maas_portal/static
- ${DMOD_SSL_DIR}/requestservice:/usr/maas_portal/ssl
# Needed only for speeding debugging
#- ${DOCKER_GUI_HOST_SRC:?GUI sources path not configured in environment}/MaaS:/usr/maas_portal/MaaS
#- ${DOCKER_GUI_HOST_VENV_DIR:-/tmp/blah}:${DOCKER_GUI_CONTAINER_VENV_DIR:-/tmp/blah}
# Expose Django's port to the internal network so that the web server may access it
expose:
Expand All @@ -76,8 +90,10 @@ services:
networks:
- request-listener-net
volumes:
#- ${DMOD_GUI_POSTGRES_DATA:?}:/var/lib/postgresql/data
- dmod_db_volume:/var/lib/postgresql/data
# TODO: look back later at why this was done during dev work, and whether it is still needed
# TODO: for now, implement by just defaulting to same Docker volume name (i.e., without bind mount) as before
- ${DMOD_GUI_POSTGRES_DATA_VOLUME_HOST_DIR:-dmod_db_volume}:/var/lib/postgresql/data
#- ${DMOD_GUI_POSTGRES_DATA_VOLUME_HOST_DIR:?}:/var/lib/postgresql/data
secrets:
- postgres_password
environment:
Expand All @@ -98,6 +114,10 @@ networks:
secrets:
postgres_password:
file: ../secrets/postgres_password.txt
object_store_exec_user_passwd:
file: ${DMOD_OBJECT_STORE_EXEC_USER_PASSWD_SECRET_FILE:?}
object_store_exec_user_name:
file: ${DMOD_OBJECT_STORE_EXEC_USER_NAME_SECRET_FILE:?}
# Define persistent volumes that may be shared and persisted between containers
volumes:
dmod_db_volume:
Expand Down
1 change: 1 addition & 0 deletions docker/nwm_gui/web_server/nginx/default.conf
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ upstream wresgui {

server {
listen 80;
client_max_body_size 0;

# Restrict verbs to GET, HEAD, and POST
if ($request_method !~ ^(GET|HEAD|POST)$ )
Expand Down
2 changes: 1 addition & 1 deletion docker/py-sources/py-deps.Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
ARG REQUIRE="gcc g++ musl-dev gdal-dev libffi-dev openssl-dev rust cargo git proj proj-dev proj-util openblas openblas-dev lapack lapack-dev"
ARG REQUIRE="gcc g++ musl-dev gdal-dev libffi-dev openssl-dev rust cargo git proj proj-dev proj-util openblas openblas-dev lapack lapack-dev geos-dev"
################################################################################################################
################################################################################################################
##### Create foundational level build stage with initial structure
Expand Down
5 changes: 5 additions & 0 deletions example.env
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,11 @@ TROUTE_BRANCH=ngen
## Python Packages Settings ##
########################################################################

## The "name" of the built client Python distribution package, for purposes of installing (e.g., via pip)
PYTHON_PACKAGE_DIST_NAME_CLIENT=dmod-client
## The name of the actual Python communication package (i.e., for importing or specifying as a module on the command line)
PYTHON_PACKAGE_NAME_CLIENT=dmod.client

## The "name" of the built communication Python distribution package, for purposes of installing (e.g., via pip)
PYTHON_PACKAGE_DIST_NAME_COMMS=dmod-communication
## The name of the actual Python communication package (i.e., for importing or specifying as a module on the command line)
Expand Down
Loading