Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report if index is hidden in index stats #18706

Merged
merged 5 commits into from
Jul 8, 2020
Merged

Report if index is hidden in index stats #18706

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
abc108b
Use expand_wildcards in tests as well
ycombinator May 22, 2020
c8c35ad
Report index hidden setting in index stats
ycombinator May 22, 2020
987c583
Adding CHANGELOG entry
ycombinator May 22, 2020
87b73dd
Using json.Unmarshaler instead
ycombinator Jul 7, 2020
eb621b5
Hybrid approach
ycombinator Jul 7, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -534,6 +534,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add support for named ports in autodiscover. {pull}19398[19398]
- Add param `aws_partition` to support aws-cn, aws-us-gov regions. {issue}18850[18850] {pull}19423[19423]
- The `elasticsearch/index` metricset now collects metrics for hidden indices as well. {issue}18639[18639] {pull}18703[18703]
- The `elasticsearch-xpack/index` metricset now reports hidden indices as such. {issue}18639[18639] {pull}18706[18706]

*Packetbeat*

Expand Down
56 changes: 56 additions & 0 deletions metricbeat/module/elasticsearch/elasticsearch.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import (
"encoding/json"
"fmt"
"net/url"
"strconv"
"strings"
"sync"
"time"
Expand Down Expand Up @@ -359,6 +360,61 @@ func GetXPack(http *helper.HTTP, resetURI string) (XPack, error) {
return xpack, err
}

type boolStr bool

func (b *boolStr) UnmarshalJSON(raw []byte) error {
var bs string
err := json.Unmarshal(raw, &bs)
if err != nil {
return err
}

bv, err := strconv.ParseBool(bs)
if err != nil {
return err
}

*b = boolStr(bv)
return nil
}

type IndexSettings struct {
Hidden bool
}

// GetIndicesSettings returns a map of index names to their settings.
// Note that as of now it is optimized to fetch only the "hidden" index setting to keep the memory
// footprint of this function call as low as possible.
func GetIndicesSettings(http *helper.HTTP, resetURI string) (map[string]IndexSettings, error) {
content, err := fetchPath(http, resetURI, "*/_settings", "filter_path=*.settings.index.hidden&expand_wildcards=all")

if err != nil {
return nil, errors.Wrap(err, "could not fetch indices settings")
}

var resp map[string]struct {
Settings struct {
Index struct {
Hidden boolStr `json:"hidden"`
} `json:"index"`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not Hidden bool, or Index IndexSettings json:"index"?

Copy link
Contributor Author

@ycombinator ycombinator Jul 7, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It had to do with the type of index.hidden in the Elasticsearch API's JSON response being a string, e.g. "true", instead of being a JSON boolean, e.g. true. Let me make a commit showing what I tried before this, implementing json.Unmarshaler, then we can decide which implementation we like better.

Copy link
Contributor Author

@ycombinator ycombinator Jul 7, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@urso I've made an alternate implementation (the one I tried originally) in 87b73dd. Let me know if you prefer that or the one right before that. Thanks!

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uff, I see. If we would use something like boolstr more often we might consider to provide a package/library with decoding primitives. But here as a standalone single use case within the package I think I prefer to decode into string first. A hybrid solution could be Index struct { Hidden boolstr json:"index" }, but keep IndexSetting{ Hidden bool }. Then you might be able to cast from Index into IndexSetting directly. Using boolstr we see decoding errors directly returned from the JSON decoder if something changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, let me try the hybrid solution next.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@urso Implemented the hybrid approach in eb621b5. Let me know what you think. Thanks!

} `json:"settings"`
}

err = json.Unmarshal(content, &resp)
if err != nil {
return nil, errors.Wrap(err, "could not parse indices settings response")
}

ret := make(map[string]IndexSettings, len(resp))
for index, settings := range resp {
ret[index] = IndexSettings{
Hidden: bool(settings.Settings.Index.Hidden),
}
}

return ret, nil
}

// IsMLockAllEnabled returns if the given Elasticsearch node has mlockall enabled
func IsMLockAllEnabled(http *helper.HTTP, resetURI, nodeID string) (bool, error) {
content, err := fetchPath(http, resetURI, "_nodes/"+nodeID, "filter_path=nodes.*.process.mlockall")
Expand Down
3 changes: 2 additions & 1 deletion metricbeat/module/elasticsearch/elasticsearch_integration_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,8 +226,10 @@ func TestGetAllIndices(t *testing.T) {
switch idx.Index {
case indexVisible:
idxVisibleExists = true
require.False(t, idx.Hidden)
case indexHidden:
idxHiddenExists = true
require.True(t, idx.Hidden)
}
}

Expand Down Expand Up @@ -592,7 +594,6 @@ func ingestAndEnrichDoc(host string) error {

func countIndices(elasticsearchHostPort string) (int, error) {
return countCatItems(elasticsearchHostPort, "indices", "&expand_wildcards=open,hidden")

}

func countShards(elasticsearchHostPort string) (int, error) {
Expand Down
11 changes: 11 additions & 0 deletions metricbeat/module/elasticsearch/index/data_xpack.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ type Index struct {
Index string `json:"index"`
Created int64 `json:"created"`
Status string `json:"status"`
Hidden bool `json:"hidden"`
Shards shardStats `json:"shards"`
}

Expand Down Expand Up @@ -141,11 +142,21 @@ func eventsMappingXPack(r mb.ReporterV2, m *MetricSet, info elasticsearch.Info,
return errors.Wrap(err, "failure parsing Indices Stats Elasticsearch API response")
}

indicesSettings, err := elasticsearch.GetIndicesSettings(m.HTTP, m.HTTP.GetURI())
if err != nil {
return errors.Wrap(err, "failure retrieving indices settings from Elasticsearch")
}

var errs multierror.Errors
for name, idx := range indicesStats.Indices {
event := mb.Event{}
idx.Index = name

settings, exists := indicesSettings[name]
if exists {
idx.Hidden = settings.Hidden
}

err = addClusterStateFields(&idx, clusterState)
if err != nil {
errs = append(errs, errors.Wrap(err, "failure adding cluster state fields"))
Expand Down