Adds Thread_Waited_Time and Thread_Waited_Event metrics

[eirsep]

Is your feature request related to a problem? Please provide an existing Issue # , or describe.
Adds new metrics published by the metrics API : Thread_Waited_Time(https://docs.oracle.com/en/java/javase/14/docs/api/java.management/java/lang/management/ThreadInfo.html#getWaitedTime()) and Thread_Waited_Event(https://docs.oracle.com/en/java/javase/14/docs/api/java.management/java/lang/management/ThreadInfo.html#getWaitedCount())

Describe the solution you are proposing

curl -"http://localhost:9600/_opendistro/_performanceanalyzer/metrics?metrics=Thread_Waited_Time&agg=avg&dim=Operation" 
{"4om_qa8XQJyJBSERJzd7gQ": {"timestamp": 1634503975000, "data": {"fields":[{"name":"Operation","type":"VARCHAR"},{"name":"Thread_Waited_Time","type":"DOUBLE"}],"records":[["GC",0.0],["refresh",1.0],["search",1.0],["transportWorker",0.2],["write",0.7538461538461538]]}}}% 

% curl "http://localhost:9600/_opendistro/_performanceanalyzer/metrics?metrics=Thread_Waited_Event&agg=avg&dim=Operation" 
{"4om_qa8XQJyJBSERJzd7gQ": {"timestamp": 1634504345000, "data": {"fields":[{"name":"Operation","type":"VARCHAR"},{"name":"Thread_Waited_Event","type":"DOUBLE"}],"records":[["GC",0.0],["flush",1.0],["refresh",597.648],["search",75.37135],["transportWorker",6.4729600000000005],["write",318.8010732394367]]}}}%  

Additional context
The new metrics support Operation as a dimension.

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #70 (fb73874) into main (7030002) will decrease coverage by 0.18%.
The diff coverage is 18.75%.

@@             Coverage Diff              @@
##               main      #70      +/-   ##
============================================
- Coverage     72.18%   72.00%   -0.19%     
+ Complexity     2948     2943       -5     
============================================
  Files           376      376              
  Lines         18680    18723      +43     
  Branches       1429     1432       +3     
============================================
- Hits          13485    13481       -4     
- Misses         4610     4655      +45     
- Partials        585      587       +2     

Impacted Files	Coverage Δ
...ormanceanalyzer/collectors/OSMetricsCollector.java	1.86% <0.00%> (-0.16%)	⬇️
...opensearch/performanceanalyzer/jvm/ThreadList.java	12.34% <0.00%> (-1.61%)	⬇️
...search/performanceanalyzer/metrics/AllMetrics.java	86.63% <100.00%> (+0.04%)	⬆️
...search/performanceanalyzer/model/MetricsModel.java	100.00% <100.00%> (ø)
...rg/opensearch/performanceanalyzer/hwnet/Disks.java	50.66% <0.00%> (-6.67%)	⬇️
...formanceanalyzer/PerformanceAnalyzerWebServer.java	68.65% <0.00%> (-5.98%)	⬇️
...ceanalyzer/rca/store/rca/hotshard/HotShardRca.java	86.13% <0.00%> (-1.00%)	⬇️
...performanceanalyzer/collectors/StatsCollector.java	92.48% <0.00%> (-0.76%)	⬇️
...nsearch/performanceanalyzer/rca/RcaController.java	80.61% <0.00%> (-0.45%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7030002...fb73874. Read the comment docs.

[sruti1312]

Thanks for adding these metrics. Can you merge from main and re-run the CI? I think the CI is failing due to a known error that is fixed in main?

The metrics documentation page needs to be updated https://opendistro.github.io/for-elasticsearch-docs/docs/pa/reference. It can be done by modifying it in https://github.com/opensearch-project/opensearch-docs.

[eirsep]

Thanks for adding these metrics. Can you merge from main and re-run the CI? I think the CI is failing due to a known error that is fixed in main?

The metrics documentation page needs to be updated https://opendistro.github.io/for-elasticsearch-docs/docs/pa/reference. It can be done by modifying it in https://github.com/opensearch-project/opensearch-docs.

Thanks for reviewing the CR @sruti1312

I am getting a 404 on the repo https://github.com/opensearch-project/opensearch-docs most prolly due to lack of permission. Will update those docs once I get permission.