Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Extensions] Replaces ADTransportGetFieldsMappingsAction with SDK indices client call #888

Merged
merged 2 commits into from
May 2, 2023

Conversation

joshpalis
Copy link
Member

@joshpalis joshpalis commented May 2, 2023

Description

Adds field mapping validation check for validate detector API .

note : Valid detector/model configuration returns an empty response : {}

Validate Single Entity Detector (valid detector)

 curl -X POST "localhost:9200/_extensions/_ad-extension/detectors/_validate" -H "Content-Type:application/json" --data
'{
  "name": "test-detector",
  "description": "Test detector",
  "time_field": "@timestamp",
  "indices": [
    "server_log"
  ],
  "feature_attributes": [
    {
      "feature_name": "test",
      "feature_enabled": true,
      "aggregation_query": {
        "test": {
          "avg": {
            "field": "jvmGcTime"
          }
        }
      }
    }
  ],
  "detection_interval": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  },
  "window_delay": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  }
}'
{}

Validate Single Entity Detector (invalid detector mapping : time field)

 curl -X POST "localhost:9200/_extensions/_ad-extension/detectors/_validate" -H "Content-Type:application/json" --data
'{
  "name": "test-detector",
  "description": "Test detector",
  "time_field": "timestamp",
  "indices": [
    "server_log"
  ],
  "feature_attributes": [
    {
      "feature_name": "test",
      "feature_enabled": true,
      "aggregation_query": {
        "test": {
          "avg": {
            "field": "jvmGcTime"
          }
        }
      }
    }
  ],
  "detection_interval": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  },
  "window_delay": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  }
}'
{
  "detector": {
    "time_field": {
      "message": "Timestamp field: (timestamp) is not found in index mapping"
    }
  }
}

Validate Multi Entity Detector (valid detector)

curl -X POST "localhost:9200/_extensions/_ad-extension/detectors/_validate" -H "Content-Type:application/json" --data
`{
  "name": "test-multi-detector",
  "description": "Test detector",
  "time_field": "@timestamp",
  "indices": [
    "server_log"
  ],
  "feature_attributes": [
    {
      "feature_name": "test",
      "feature_enabled": true,
      "aggregation_query": {
        "test": {
          "avg": {
            "field": "cpuTime"
          }
        }
      }
    }
  ],
  "detection_interval": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  },
  "window_delay": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  },
  "category_field": [
    "process"
  ]
}`
{}

Validate Multi Entity Detector (invalid detector mapping : aggregation query field)

curl -X POST "localhost:9200/_extensions/_ad-extension/detectors/_validate" -H "Content-Type:application/json" --data
`{
  "name": "test-multi-detector",
  "description": "Test detector",
  "time_field": "@timestamp",
  "indices": [
    "server_log"
  ],
  "feature_attributes": [
    {
      "feature_name": "test",
      "feature_enabled": true,
      "aggregation_query": {
        "test": {
          "avg": {
            "field": "wrongField"
          }
        }
      }
    }
  ],
  "detection_interval": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  },
  "window_delay": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  },
  "category_field": [
    "process"
  ]
}`
{
  "detector": {
    "feature_attributes": {
      "message": "Feature has an invalid query returning empty aggregated data: test",
      "sub_issues": {
        "test": "Feature has an invalid query returning empty aggregated data"
      }
    }
  }
}

Validate detector model (detection interval of 10 minutes)

-X POST "localhost:9200/_extensions/_ad-extension/detectors/_validate/model" -H "Content-Type:application/json" --data
`
{
  "name": "test-multi-detector",
  "description": "Test detector",
  "time_field": "@timestamp",
  "indices": [
    "server_log"
  ],
  "feature_attributes": [
    {
      "feature_name": "test",
      "feature_enabled": true,
      "aggregation_query": {
        "test": {
          "avg": {
            "field": "cpuTime"
          }
        }
      }
    }
  ],
  "detection_interval": {
    "period": {
      "interval": 10,
      "unit": "Minutes"
    }
  },
  "window_delay": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  },
  "category_field": [
    "process"
  ]
}`
{
  "model": {
    "detection_interval": {
      "message": "The selected detector interval might collect sparse data. Consider changing interval length to: 6",
      "suggested_value": {
        "period": {
          "interval": 6,
          "unit": "Minutes"
        }
      }
    }
  }
}

Validate detector model (suggested detection interval of 6 minutes)

curl -X POST "localhost:9200/_extensions/_ad-extension/detectors/_validate/model" -H "Content-Type:application/json" --data
'{
  "name": "test-multi-detector",
  "description": "Test detector",
  "time_field": "@timestamp",
  "indices": [
    "server_log"
  ],
  "feature_attributes": [
    {
      "feature_name": "test",
      "feature_enabled": true,
      "aggregation_query": {
        "test": {
          "avg": {
            "field": "cpuTime"
          }
        }
      }
    }
  ],
  "detection_interval": {
    "period": {
      "interval": 6,
      "unit": "Minutes"
    }
  },
  "window_delay": {
    "period": {
      "interval": 1,
      "unit": "Minutes"
    }
  },
  "category_field": [
    "process"
  ]
}'
{}

Issues Resolved

opensearch-project/opensearch-sdk-java#361

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@joshpalis joshpalis requested review from dbwiddis, owaiskazi19 and a team May 2, 2023 19:38
@joshpalis joshpalis merged commit 7cece65 into opensearch-project:feature/extensions May 2, 2023
@joshpalis joshpalis deleted the mappings branch May 2, 2023 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants