basic monitoring functionality

[esraneufeld]

• enable access to grafana dashboards to app-team / nik for s4l-lite
• Settle on useful information / graphs that are displayed

[mrnicegyu11]

After the discussion with Nik, we set the following goals:

There shall be a grafana dashboard on aws-prod which targets only s4l-lite.
We are free in preparing a first version to show to the POs. Some things of interest that shall be included if possible are:

• Number of registered s4l-users
• Number of logged-in s4l-users
• Number of s4l-lite studies open
• Resource usage s4l-lite, potentially by userid
• Resource usage s4l-lite per machine
• [ ] Costs s4l-lite (requires some effort to add, as this data doesnt come from osparc but aws) this is tracked in sim4life.io - WP3: Tracking of resource usage #922

For this functionality, we require:

• user_id is propagated as a container label, so cAdvisor prometheus timeseries can be used
• E2E: complete implementation of test to check if all dy-sidecar created services were properly removed after testing osparc-simcore#3588
• Connect AWS cost explorer, potentially flag/filter s4l-lite machines
• Connect postgres to grafana or enable prometheus postgres exporter

CC @sanderegg @GitHK , potentially please add user_id is propagated to dy-sidecar case

[mrnicegyu11]

fyi this is related to #679

[mrnicegyu11]

Planned for sprint Mithril:

• MVP (initial dashboard proposition) available on grafana osparc.io for Erdem & Nik

[elisabettai]

@mrnicegyu11, I hope then we can also solve this one for the NIH Metrics ITISFoundation/osparc-simcore#3908 (we need to be able to remove puppeteers from the service usage metric)

[mrnicegyu11]

Goal for sprint Jelly Beans:

• Meeting with EO/NC to define future usage/goals
• Filter out puppeteer services in s4l-monitoring dashboard

[mrnicegyu11]

Done for sprint Jelly Beans:

• Meeting with EO/NC presenting the dashboard, incentivicing further discussions about which metrics might be desireable to assess the usage of the app. EO will follow up with DK at this discretion.
• Filter out puppeteer services in s4l-monitoring dashboard

[eofli]

Here is the list for S4Llite metrics/dashboard:

• number of active users at any time instance
• number of active users geographically
• number of instances of each tutorial (i.e., how many times each tutorial is launched)
• how long time spend in each of the tabs (modeling, simulation, analysis)
• how long each user is active in each logged-in session (also, average of all users)
• how many times each user is logged in a selected time period (also, the average of all users)
• number of failures in launching s4l lite
• number of solver failures
• response time (average, min, max to see if there is any correlation)
• number of notebook instances
• which tools/solvers used

[mrnicegyu11]

Goal for sprint Watermelon:

• Add prometheus export on PORT:/metrics in a s4l-lite service
• The dy-sidecar scrapes these metrics and exposes them on the dy-sidecar itself on PORT:/metrics
• Prometheus auto-discovers docker swarm services to be scraped if they have the docker service labels prometheus-job=SOME-STRING-THAT-DESCRIBES-YOUR-SERVICE and prometheus-port=PORT-WHERE-SLASH-METRICS-IS-SCRAPED set. The director-v2/dy-sidecar decide (based on info from the catalog / docker image labels of the dynamic service containers) if there is prometheus metrics to be scraped and sets the mentioned swarm service labels on itself if so, so prometheus will go there and collect the /etrics
• We will not keep information on the user/group/service-instance level but strictly keep averages across all s4-lite instances, in order to manage cardinality

[mguidon]

Update Watermelon

User metrics collection implemented in sim4life:

• Tools (count, duration)
• Algorithms (count)
• Vip Models (count)
• Solvers (count)

Missing:

• Number of Notebooks used
• Success/failure of solvers

Example: MaG creating the Antenna Tutorial and loading a VIP model

Type	What	Count	Duration
Tool	3D Entity Transform Tool	2	16.7062
Tool	3D Manipulation	2	15.5623
Tool	Box Tool	2	12.8755
Tool	CEmFdtdBoundarySettings Tool	1	0.582367
Tool	CEmFdtdLumpedElementSettings Tool	1	0.579783
Tool	CGridInfoTool	2	14.306
Tool	CMaterialSettings Tool	2	6.4733
Algorithm	Data Source	5	n/a
Algorithm	EM - EM FDTD	135	n/a
Algorithm	EM E(x,y,z,f0) - Interpolator	1	n/a
Algorithm	EM E(x,y,z,f0) - Slice Viewer	98	n/a
Algorithm	EM E(x,y,z,f0) - To Cell Data	1	n/a
Algorithm	EM E(x,y,z,f0) - Vector Field Viewer	35	n/a
Solver	EM FDTD - Cuda	1	n/a
Tool	EM FDTD Sensors Tool	1	5.27573
Tool	EM FDTD Setup Tool	2	1.59146
Tool	EM FDTD Simulator Tool	1	79.3653
Tool	EM FDTD Sources Tool	1	1.62372
Tool	EM FDTD Voxeler Tool	2	30.3517
Tool	EmFdtd Gridder Tool	2	14.3898
Algorithm	Extract Slice	4	n/a
Algorithm	Field Extractor	1	n/a
Tool	Import Tool	1	1.40758
Tool	Model Picker	7	53.0047
Tool	Modeler	2	98.3967
Tool	Mother Controller Tool	1	278.23
Algorithm	Overall Field	134	n/a
Tool	Parametric Tool	8	46.3454
Tool	Point Tracker	8	46.3296
Tool	Post Processor	2	96.0171
Tool	Rubber Band	1	1.13693
Tool	Simulator	2	83.5246
Tool	Slice Field Viewer Tool	4	64.7911
Tool	Solver Settings Tool	3	12.2217
Tool	Topological Voxel Viewer Tool	1	21.5421
Tool	Vector Field Viewer Tool	2	15.0484
Vip	Yoon-sun	1	n/a

[mguidon]

Meeting Notes Aug 17

ist of items to observe
metrics, monitoring

cosimo, erdem to come up with list of tools and algos for statistics

anonymous only

full version with consent

lite version
user is informed

S4L full we add only Poser
minimium

[mguidon]

The list

• All Solvers
• All Dosimetric Algorithms
• Network Analysis
• Smith Chart Viewer (time spent)
• Total Exposure Ratio (TER)
• Exporters and Touchstone File Importer and Exporter
• Modulation
• 5G related (Array Factor, Phased Array , from Analysis and Modeling/Simulation)
• Time spent on each tab
• 10-10 System
• Surface Projection tool
• Additional Modules (BCAGE, MUSAIK, MUSAIK V2.0, Parallel Transmit, TXCOIL, IMSAFE - RF, MBSAR, Antenna Diversity, MATCH, Hearing Aid Compatibility Tool, DISPFIT, SYSSIM, IMAnalytics)