Enhancement: Display statistics of computational jobs together with their parent nodes

[sanderegg]

Context

Since the public API is available it is now possible to run X computational jobs from a running dynamic service.
Looking at the Usage statistics of a user it currently displays computational jobs separated from their "parent" dynamic service.

Goal

Display the statistics of computational jobs linked to their parent service.

Needed changes

0. Important note: we should keep backward compatibility of the API, and also for sim4life.io keep the option to pass it via metadata at least for awhile.
1. modify the oSparc API to create/run computational jobs from a running dynamic service by passing the "parent" node ID (ideally automatically defined, if not then the API shall be modified) - use-case: sim4life, meta-modeling, jupyterlabs, ...
2. the parent Node ID is passed all the way to the computational backend (already exists, needs to be modified based on 1.)
3. using the parent NodeID the logs are sent back to the parent project/nodeID if it exists (already exists, needs to be modified based on 1.)
4. the resource usage tracker shall keep track of the parent node ID if it exists
5. the frontend shall display the usage with services and their children jobs

Tasks

Preview Give feedback

1. S4L: Upgrade s4l client to pass parent UUIDs #5950
   +0
   
2. Update osparc-client version in all TIP services osparc-issues#1618
   +0
   

   
3. ♻️Ensure parent project/node is well structured in the DB 🗃️ #5874
   a:dask-service a:director-v2 a:webserver +3
   
4. resource usage tracker shall keep track of parent project/node IDs #5878
   3 of 3
   a:resource-usage-tracker +1
   
5. 🎨♻️Use structured parent project/node in director-v2 and connect with RUT #5877
   a:director-v2 +1
   
6. update UI display of usage to show children studies #5879
   1 of 2
   a:frontend +1
   
7. Adapt PublicAPI/Python client to automatically set the parent project UUID and node ID #5881
   1 of 2
   a:apiserver a:webserver +2
   
8. Pass oSparc API necessary environment to all services #5925
   a:director-v2 a:pipeline-services +2
   
9. ✨ Auto inject osparc environments to dynamic services #5966
   a:director-v2 +1
   
10. Release osparc client osparc-issues#1517
    +0
    

[sanderegg]

After discussion with @bisgaard-itis :

proposal to modify the osparc python client:

• modify the API call to create a computational job to get an optional header containing at least the parent node ID
• based on ENV OSPARC_NODE_ID and possibly OSPARC_STUDY_ID variables set in the dynamic service,
• the client can automatically fill in the headers
  -> Users that are using the python client in their code will get that feature for free

[mguidon]

So this is to avoid having it in the not-validated metadata?

[sanderegg]

So this is to avoid having it in the not-validated metadata?

As discussed, no. This is for generalization of this usage and to ensure we always get that info so that the billing center looks nice.

As discussed as well, both ways (the sim4life.io way and the new one should work, at least for awhile)

[bisgaard-itis]

After discussion with @bisgaard-itis :

proposal to modify the osparc python client:

* modify the API call to create a computational job to get an optional header containing at least the parent node ID

* based on ENV `OSPARC_NODE_ID` and possibly `OSPARC_STUDY_ID` variables set in the dynamic service,

* the client can automatically fill in the headers
  -> Users that are using the python client in their code will get that feature for free

After thinking a bit more about this I have the following modified proposal: Since this approach is based on the client "picking up" the node_id and sending it to the api-server I suggest to simply overwrite the create_solver_job method in the osparc python client, so that it first calls the endpoint on the api-server to create the job and afterwards calls the patch endpoint with the metadata picked up from the environment variables. That way we will not have to modify anything on the server, so any existing functionality will continue to work and we simply "package" the endpoints into user-friendly functions on the client side.

[sanderegg]

After discussion with @bisgaard-itis :

proposal to modify the osparc python client:

* modify the API call to create a computational job to get an optional header containing at least the parent node ID

* based on ENV `OSPARC_NODE_ID` and possibly `OSPARC_STUDY_ID` variables set in the dynamic service,

* the client can automatically fill in the headers
  -> Users that are using the python client in their code will get that feature for free

After thinking a bit more about this I have the following modified proposal: Since this approach is based on the client "picking up" the node_id and sending it to the api-server I suggest to simply overwrite the create_solver_job method in the osparc python client, so that it first calls the endpoint on the api-server to create the job and afterwards calls the patch endpoint with the metadata picked up from the environment variables. That way we will not have to modify anything on the server, so any existing functionality will continue to work and we simply "package" the endpoints into user-friendly functions on the client side.

@bisgaard-itis ok, but will this also work if the user (such as in sim4life.io) also calls the PATCH endpoint? will this not overwrite whatever was in there? Also I would prefer that the parent node id is not just some json field, but a defined one.

[bisgaard-itis]

After discussion with @bisgaard-itis :

proposal to modify the osparc python client:

* modify the API call to create a computational job to get an optional header containing at least the parent node ID

* based on ENV `OSPARC_NODE_ID` and possibly `OSPARC_STUDY_ID` variables set in the dynamic service,

* the client can automatically fill in the headers
  -> Users that are using the python client in their code will get that feature for free

After thinking a bit more about this I have the following modified proposal: Since this approach is based on the client "picking up" the node_id and sending it to the api-server I suggest to simply overwrite the create_solver_job method in the osparc python client, so that it first calls the endpoint on the api-server to create the job and afterwards calls the patch endpoint with the metadata picked up from the environment variables. That way we will not have to modify anything on the server, so any existing functionality will continue to work and we simply "package" the endpoints into user-friendly functions on the client side.

@bisgaard-itis ok, but will this also work if the user (such as in sim4life.io) also calls the PATCH endpoint? will this not overwrite whatever was in there? Also I would prefer that the parent node id is not just some json field, but a defined one.

This basically delegates all responsibility for setting the parent node_id to the client. So essentially the idea is to do in the python osparc client exactly what Manuel is already doing in the C++ client he is using from sim4life.io and wrap it into a user-friendly function by picking up the node_id from the env. I am not sure I understand exactly what you mean by a "defined field". In the end I guess it will be added in the metadata in the db in the same way Manuel is currently doing it, no?

[sanderegg]

After discussion with @bisgaard-itis :

proposal to modify the osparc python client:

* modify the API call to create a computational job to get an optional header containing at least the parent node ID

* based on ENV `OSPARC_NODE_ID` and possibly `OSPARC_STUDY_ID` variables set in the dynamic service,

* the client can automatically fill in the headers
  -> Users that are using the python client in their code will get that feature for free

After thinking a bit more about this I have the following modified proposal: Since this approach is based on the client "picking up" the node_id and sending it to the api-server I suggest to simply overwrite the create_solver_job method in the osparc python client, so that it first calls the endpoint on the api-server to create the job and afterwards calls the patch endpoint with the metadata picked up from the environment variables. That way we will not have to modify anything on the server, so any existing functionality will continue to work and we simply "package" the endpoints into user-friendly functions on the client side.

@bisgaard-itis ok, but will this also work if the user (such as in sim4life.io) also calls the PATCH endpoint? will this not overwrite whatever was in there? Also I would prefer that the parent node id is not just some json field, but a defined one.

This basically delegates all responsibility for setting the parent node_id to the client. So essentially the idea is to do in the python osparc client exactly what Manuel is already doing in the C++ client he is using from sim4life.io and wrap it into a user-friendly function by picking up the node_id from the env. I am not sure I understand exactly what you mean by a "defined field". In the end I guess it will be added in the metadata in the db in the same way Manuel is currently doing it, no?

@bisgaard-itis so the project metadata that Manuel is using are metadata that are owned by the user. we currently hack this out in order to get the parent NodeID. If your solution does not imply that the user may inadvertently remove the parent NodeID by explicitly calling the endpoint then I am ok.