[BUG] - How to visualize Shard* metrics ?

[craph]

Hello,

I'm opening this issue because I think I have a weird behaviour with performance-analyzer.

What is the bug?
Just after I've setup opensearch with docker for benchmark, I'm unable to visualize the Shard* Metrics with PerformanceAnalyzer.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Create a docker-compose.yml file with this content

version: '3'
services:
  opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearchproject/opensearch:latest # Specifying the latest available image - modify if you want a specific version
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.type=single-node
      - plugins.security.ssl.http.enabled=false
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      OPENSEARCH_HOSTS: '["http://opensearch-node1:9200","http://opensearch-node2:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query
    networks:
      - opensearch-net

volumes:
  opensearch-data1:

networks:
  opensearch-net:

2. Start the containers with the command : docker-compose up -d
3. Then enable the plugin performance-analyzer like it's mentionned here : https://opensearch.org/docs/2.3/monitoring-plugins/pa/index/#install-performance-analyzer

curl -XPOST localhost:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": true}' -u 'admin:admin'

4. Normally at this steps the performance analyzer should be up and running
5. Create an index and documents data with the Documents API bulk operation

POST _bulk
{ "index": { "_index": "movies", "_id": "tt1979320" } }
{ "title": "Rush", "year": 2013 }
{ "index": { "_index": "movies", "_id": "tt1979322" } }
{ "title": "Rush2", "year": 2013 }
{ "index": { "_index": "movies", "_id": "tt1979323" } }
{ "title": "Rush3", "year": 2013 }
{ "index": { "_index": "movies", "_id": "tt1979324" } }
{ "title": "Rush4", "year": 2013 }
{ "index": { "_index": "movies", "_id": "tt1979325" } }
{ "title": "Rush5", "year": 2013 } 

7. Here is the output of the creation

{"took":187,"errors":false,"items":[{"index":{"_index":"movies","_id":"tt1979320","_version":1,"result":"created","forced_refresh":true,"_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1,"status":201}},{"index":{"_index":"movies","_id":"tt1979322","_version":1,"result":"created","forced_refresh":true,"_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1,"status":201}},{"index":{"_index":"movies","_id":"tt1979323","_version":1,"result":"created","forced_refresh":true,"_shards":{"total":2,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1,"status":201}},{"index":{"_index":"movies","_id":"tt1979324","_version":1,"result":"created","forced_refresh":true,"_shards":{"total":2,"successful":1,"failed":0},"_seq_no":3,"_primary_term":1,"status":201}},{"index":{"_index":"movies","_id":"tt1979325","_version":1,"result":"created","forced_refresh":true,"_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4,"_primary_term":1,"status":201}}]}

8. Then try to get the Shard* Metric from performance-analyzer like this

GET localhost:9600/_plugins/_performanceanalyzer/metrics?metrics=ShardEvents,ShardBulkDocs,CPU_Utilization&dim=Operation&nodes=all&agg=sum,sum,max

9. Here is the output

{
    "90CFxK7zTi28rLkUpQiSpA": {
        "timestamp": 1668067125000,
        "data": {
            "fields": [
                {
                    "name": "Operation",
                    "type": "VARCHAR"
                },
                {
                    "name": "ShardEvents",
                    "type": "DOUBLE"
                },
                {
                    "name": "ShardBulkDocs",
                    "type": "DOUBLE"
                },
                {
                    "name": "CPU_Utilization",
                    "type": "DOUBLE"
                }
            ],
            "records": [
                [
                    "GC",
                    null,
                    null,
                    0.0
                ],
                [
                    "flush",
                    null,
                    null,
                    0.0
                ],
                [
                    "generic",
                    null,
                    null,
                    0.0
                ],
                [
                    "get",
                    null,
                    null,
                    0.0
                ],
                [
                    "management",
                    null,
                    null,
                    0.0
                ],
                [
                    "other",
                    null,
                    null,
                    0.0020006279656433657
                ],
                [
                    "refresh",
                    null,
                    null,
                    0.0
                ],
                [
                    "search",
                    null,
                    null,
                    0.0
                ],
                [
                    "shardbulk",
                    null,
                    null,
                    0.0
                ],
                [
                    "transportWorker",
                    null,
                    null,
                    0.0
                ],
                [
                    "write",
                    null,
                    null,
                    0.0
                ]
            ]
        }
    }
}

What is the expected behavior?
I guess I should be able to see something instead of null values for the Shard metrics.
Do you have any advise how to proceed to be able to view those metrics ?

What is your host/environment?

• OS: docker
• Version opensearch latest version 2.3.0
• Plugins embedded by default in the docker image

@sruti1312 , @eirsep do you have any idea ? I saw you worked on Metrics

Thank you very much for your help.
Best regards,

[craph]

Hello @sruti1312 I guess you can help on this issue because you worked on changes namespace and some other code about Shard* metrics.

I have added more details here : opensearch-project/performance-analyzer#333

Do you have any advise or idea how to get back those metrics ?

Thank you very much for your help 😃

[craph]

Why there are some changes about ShardBulkDocs metrics from this pull request #3 ?

In OpenDistro the performance-analyzer metrics was working correctly and after the rework for OpenSearch it's not working like it was and always display wrong values.

For example :
In the initial commit where it was working it was
opensearch-project/performance-analyzer@ffbd759#diff-01174560d5ac728e25a7161301eefa4a290c0a124951312a1bad2b7b55e9e00cR180
and now it's
https://github.com/opensearch-project/performance-analyzer-rca/blame/main/src/main/java/org/opensearch/performanceanalyzer/reader/ShardRequestMetricsSnapshot.java#L169

[khushbr]

Can confirm that ShardBulk metrics are getting captured on the PA plugin for OpenSearch 1.3, but the Reader is failing to correctly parse them.

^threads/21543/shardbulk/23632396/start
current_time:1674690064926
StartTime:1674690064926
ItemCount:8
IndexName:index-1
ShardID:2
--
^threads/26259/shardbulk/23632397/start
current_time:1674690064929
StartTime:1674690064929
ItemCount:3
IndexName:index-2
ShardID:2
--
^threads/22603/shardbulk/23632398/start
current_time:1674690064934
StartTime:1674690064934
ItemCount:6
IndexName:index-1
ShardID:2