Add support for serial ngen jobs #278

robertbartel · 2023-02-13T15:57:06Z

Update NGENRequest to support request of either serial or parallel jobs
Add support for performing serial NextGen execution to ngen Docker image entrypoint
Update Launcher scheduler class to support running serial NextGen jobs
Fix bug in Launcher related to handling of NgenCalibrationRequest
Update both dataservice and partitionerservice to handle serial NextGen jobs properly if/when encountered, since these don't need to ever await partitioning

The branch for this PR is based on the one for #277; as such, that PR blocks the review of this one.

Fixing logic where pip installs the updated packages (to avoid the entire image rebuild), so make sure deps are ignored (as this was the slow part).

Adding type GET_SERIALIZED_FORM to get the entire serialized state of a dataset.

Refactor to ensure it takes advantage of connection handling via its async context manager logic.

Add new function to get serialized dataset details to DatasetExternalClient.

Fix issue with not acquiring a session, and wrapping some things in try block to catch and log exceptions.

Fixing bug in response to LIST_FILES query, where reason text was not being assembled entirely correctly.

Adding optional offset and length params to abstract interface definition of get_data, and adding get_file_stat abstract method.

Adding support for querying about dataset items.

Updating to support indicating start and size of data for partial transfers.

Optimizing the reloading of datasets on object store manager startup.

Accounting for changes to get_data parameters and implementing get_file_stat.

Updating service to respond to GET_DATASET_ITEMS queries and requests for data with an offset start (i.e. partials).

Adding initial second view display and manage existing datasets, along with navigation functionality to toggle between "manage" and "create" views.

Sending serialized datasets to HTML template as list/array rather than dict/map.

Implementing layout and initial details-viewing behavior for dataset management GUI view.

Updating dmod.client to depend on dmod.communication>=0.11.0.

Updating dmod.requestservice to depend on dmod.communication>=0.11.0.

Update to new dmod.communication (0.11.0) and dmod.scheduler (0.10.0).

Updating to depend on new dmod.communication 0.11.0.

Updating to depend on dmod.scheduler 0.10.0.

Updating Dockerfiles for data-service and py-sources to explicitly use "latest" tag in FROM statements building off another DMOD internal Docker image.

- Add properties for whether serial or parallel exec was requested - Have data requirements and lazy init of partition config itself consider whether partition config is necessary, which depends on if parallel execution was requested

- Updating Launcher class to account for serial ngen execution by not passing any partitioning config dataset when there is only 1 cpu - Fixing bug in Launcher regarding how NgenCalibrationRequest jobs were processed, which was not handled correctly

Updating manager logic to have single-cpu (i.e., serial exec) jobs be moved to AWAITING_ALLOCATION without a partition config being generated if ever in the AWAITING_PARTITIONING step.

Updating to new dmod.communication (0.12.0) and dmod.scheduler (0.11.0).

Updating service to move to AWAITING_ALLOCATION step instead of AWAITING_PARTITIONING step after data checks are successful, if serial execution is indicated by a job cpu count of 1.

Updating to new dmod.communication (0.12.0) and dmod.scheduler (0.11.0).

Updating to new dmod.communication (0.12.0).

robertbartel · 2023-03-15T14:21:31Z

Closing; replaced by #301, which isolates the relevant changes.

robertbartel added 30 commits January 18, 2023 12:14

Fix fast-debug in dataservice image entrypoint.

d9cfcf9

Fixing logic where pip installs the updated packages (to avoid the entire image rebuild), so make sure deps are ignored (as this was the slow part).

Add package client name vars to example.env.

8106216

Add new QueryType for dataset management messages.

e7ad95d

Adding type GET_SERIALIZED_FORM to get the entire serialized state of a dataset.

Add GET_SERIALIZED_FORM query dataservice support.

506e77b

Refactor ExternalRequestClient async_make_request.

00cd95a

Refactor to ensure it takes advantage of connection handling via its async context manager logic.

Adding client methods for data/metadata retrieval.

314e36b

Refactor dataset request handler to use try block.

30cbcca

Add get_serialized_datasets to external client.

b516c5c

Add new function to get serialized dataset details to DatasetExternalClient.

Improve DatasetExternalClient.

8445296

Fix issue with not acquiring a session, and wrapping some things in try block to catch and log exceptions.

Bump dataservice dependencies.

a2927b2

Fix LIST_FILES query response bug in dataservice.

f23c721

Fixing bug in response to LIST_FILES query, where reason text was not being assembled entirely correctly.

Update dataset manager abstract interface.

8e45fbd

Adding optional offset and length params to abstract interface definition of get_data, and adding get_file_stat abstract method.

More QueryType elements and update DatasetQuery.

60c6e8e

Adding support for querying about dataset items.

Update dataset message to support start and size.

c893164

Updating to support indicating start and size of data for partial transfers.

Bump requestservice dependencies.

a9dbd5b

Bump requestservice to 0.6.0.

4198db5

Optimize object store dataset reloading.

4444b62

Optimizing the reloading of datasets on object store manager startup.

Update object store manager for interface changes.

7125fb8

Accounting for changes to get_data parameters and implementing get_file_stat.

Update modeldata dependency versions.

9024a0c

Update client unit tests for interface changes.

6ecaced

Bump client package version to 0.2.0.

dbbcf5a

Update dataservice response handling.

f851e75

Updating service to respond to GET_DATASET_ITEMS queries and requests for data with an offset start (i.e. partials).

Update dataservice package dependency versions.

f1ecc2c

Add dataset view class and static page for GUI.

ab51dbf

Refactor static dir usage in DMODProxy.py.

60fbab5

Add Django GUI url def for dataset view.

89179ad

Add client package build arg for GUI Docker build.

28e0168

Add second view and navigation.

6df5754

Adding initial second view display and manage existing datasets, along with navigation functionality to toggle between "manage" and "create" views.

Have DatasetManagementView.py send DS as list.

f68bbd2

Sending serialized datasets to HTML template as list/array rather than dict/map.

Update dataset management GUI with details view.

64957ad

Implementing layout and initial details-viewing behavior for dataset management GUI view.

robertbartel added 24 commits January 31, 2023 10:40

Update client dependency to latest comms.

57ffe73

Updating dmod.client to depend on dmod.communication>=0.11.0.

Bump dmod.client version to 0.3.0.

6a3a09b

Bump dmod.scheduler to 0.10.0.

bd6c1c7

Update requestservice dependency to latest comms.

7891094

Updating dmod.requestservice to depend on dmod.communication>=0.11.0.

Update schedulerservice dependencies.

fd977b1

Update to new dmod.communication (0.11.0) and dmod.scheduler (0.10.0).

Bump dmod.schedulerservice version to 0.8.0.

f6fce3c

Update dataservice to new comms dependency.

32df401

Updating to depend on new dmod.communication 0.11.0.

Updating dataservice to depend on new scheduler.

bd58d03

Updating to depend on dmod.scheduler 0.10.0.

Bump requestservice version to 0.7.0.

3e4311e

Update a couple FROM statements to use "latest."

6edee7b

Updating Dockerfiles for data-service and py-sources to explicitly use "latest" tag in FROM statements building off another DMOD internal Docker image.

Update NGENRequest to account for serial/parallel.

99b44d1

- Add properties for whether serial or parallel exec was requested - Have data requirements and lazy init of partition config itself consider whether partition config is necessary, which depends on if parallel execution was requested

Update ngen image entrypoint for serial exec.

76abb1b

Update tests for NGENRequest after recent changes.

c8b664d

Bump dmod.scheduler version to 0.11.0.

7294325

Update partitioner service to account for serial.

f85df58

Updating manager logic to have single-cpu (i.e., serial exec) jobs be moved to AWAITING_ALLOCATION without a partition config being generated if ever in the AWAITING_PARTITIONING step.

Update dataservice dependencies.

8b4b71c

Updating to new dmod.communication (0.12.0) and dmod.scheduler (0.11.0).

Update dataset service for serial exec.

eedff40

Updating service to move to AWAITING_ALLOCATION step instead of AWAITING_PARTITIONING step after data checks are successful, if serial execution is indicated by a job cpu count of 1.

Bump dmod.communication version to 0.12.0.

beb2699

Bump dmod.dataservice version to 0.6.0.

f8ceb3f

Update schedulerservice to new dependencies.

6c1edcc

Updating to new dmod.communication (0.12.0) and dmod.scheduler (0.11.0).

Bump dmod.schedulerservice version to 0.9.0.

480f20b

Update requestservice to new dependencies.

47eeecf

Updating to new dmod.communication (0.12.0).

Bump dmod.requestservice version to 0.8.0.

01d5e6e

robertbartel added the maas MaaS Workstream label Feb 13, 2023

robertbartel mentioned this pull request Feb 22, 2023

Pydantic Refactoring #279

Closed

This was referenced Mar 14, 2023

Combined pre-AGU GUI and Calibration Message Work #240

Open

Add support for serial ngen jobs #301

Merged

robertbartel closed this Mar 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for serial ngen jobs #278

Add support for serial ngen jobs #278

robertbartel commented Feb 13, 2023

robertbartel commented Mar 15, 2023

Add support for serial ngen jobs #278

Add support for serial ngen jobs #278

Conversation

robertbartel commented Feb 13, 2023

robertbartel commented Mar 15, 2023