Fix data-stream name resolution for wild-cards

[sandeshkr419]

Signed-off-by: Sandesh Kumar [email protected]

Description

[Describe what this change achieves]

• Category: Bug fix

• Security plugin is not able to invalidate non-permitted data-stream (get/delete/stats) requests sent using wild-card expressions.
  This is because the list of 'allIndices' in 'ResolvedIndicesProvider' class does not has the resolved names of data streams. If the 'allIndices' variable when resolved, is empty -> leads the authorization to succeed as there are no eligible index patterns to block the request.
  In this change, we add the the names of resolved data streams to 'allIndices' so data stream names also can get resolved.

• What is the old behavior before changes and new behavior after changes?

Suppose the indices are as follows in a OS cluster.

kusandes@a483e789b681 ~ % curl -XGET 'https://localhost:9200/_cat/indices?v' --insecure -u 'admin:admin'
health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
kusandes@a483e789b681 ~ % curl -XGET 'https://localhost:9200/_cat/indices?v' --insecure -u 'admin:admin'
health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .ds-logs-nginx1-000001       3V1VOJ4NQ0KEqtrJ51el8g   1   1          0            0       208b           208b
yellow open   logs2                        AwHqtjeITBGumadYv5GNuQ   1   1          0            0       208b           208b
yellow open   .ds-logs-nginx11-000001      lnMe0uUPToKbQKJCrMUoSw   1   1          0            0       208b           208b
yellow open   logs1                        oR5D0gWFT_u5lbpvSi-SOA   1   1          0            0       208b           208b
yellow open   security-auditlog-2022.03.28 YknIcv-GQLGtdocWUA6-UA   1   1         31            0     94.8kb         94.8kb
yellow open   .ds-logs-nginx3-000001       E2fpfxHSSMarY84WMIjhhQ   1   1          0            0       208b           208b
green  open   .opendistro_security         h6L4aW-QR5KMrCui6BL12g   1   0          9            0     87.9kb         87.9kb

Assume the user sandesh1 to have the following data-streams related permissions:

kusandes@a483e789b681 ~ % curl -XPUT https://localhost:9200/_plugins/_security/api/roles/ds_all -u 'admin:admin' --insecure -H 'Content-Type: application/json' -d '
{
  "cluster_permissions": [ ],
  "index_permissions": [{
    "index_patterns": [
      "logs-nginx1*"
    ],
    "dls": "",
    "fls": [],
    "masked_fields": [],
    "allowed_actions": [
      "indices:admin/data_stream/get",
      "indices:admin/data_stream/delete",
      "indices:monitor/data_stream/stats"
    ]
  }],
  "tenant_permissions": [{
    "tenant_patterns": [
      "human_resources"
    ],
    "allowed_actions": [
    ]
  }]
}
'

The ideal (expected) behavior is that user sandesh should not be able to access data-streams other than logs-nginx1 & logs-nginx11.

Old Behaviour:

kusandes@a483e789b681 ~ % curl -XGET 'https://localhost:9200/_data_stream/logs-nginx*?pretty' --insecure -u 'sandesh:kumar'
{
  "data_streams" : [
    {
      "name" : "logs-nginx1",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-logs-nginx1-000001",
          "index_uuid" : "3V1VOJ4NQ0KEqtrJ51el8g"
        }
      ],
      "generation" : 1,
      "status" : "YELLOW",
      "template" : "logs-template"
    },
    {
      "name" : "logs-nginx11",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-logs-nginx11-000001",
          "index_uuid" : "lnMe0uUPToKbQKJCrMUoSw"
        }
      ],
      "generation" : 1,
      "status" : "YELLOW",
      "template" : "logs-template"
    },
    {
      "name" : "logs-nginx3",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-logs-nginx3-000001",
          "index_uuid" : "E2fpfxHSSMarY84WMIjhhQ"
        }
      ],
      "generation" : 1,
      "status" : "YELLOW",
      "template" : "logs-template"
    }
  ]
}

New Behaviour:

kusandes@a483e789b681 ~ % curl -XGET 'https://localhost:9200/_data_stream/logs-nginx*?pretty' --insecure -u 'sandesh1:kumar'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "no permissions for [indices:admin/data_stream/get] and User [name=sandesh, backend_roles=[], requestedTenant=null]"
      }
    ],
    "type" : "security_exception",
    "reason" : "no permissions for [indices:admin/data_stream/get] and User [name=sandesh, backend_roles=[], requestedTenant=null]"
  },
  "status" : 403
}

kusandes@a483e789b681 ~ % curl -XGET 'https://localhost:9200/_data_stream/logs-nginx1*?pretty' --insecure -u 'sandesh1:kumar'
{
  "data_streams" : [
    {
      "name" : "logs-nginx1",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-logs-nginx1-000001",
          "index_uuid" : "3V1VOJ4NQ0KEqtrJ51el8g"
        }
      ],
      "generation" : 1,
      "status" : "YELLOW",
      "template" : "logs-template"
    },
    {
      "name" : "logs-nginx11",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-logs-nginx11-000001",
          "index_uuid" : "lnMe0uUPToKbQKJCrMUoSw"
        }
      ],
      "generation" : 1,
      "status" : "YELLOW",
      "template" : "logs-template"
    }
  ]
}

{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "no permissions for [indices:admin/data_stream/get] and User [name=sandesh1, backend_roles=[], requestedTenant=null]"
      }
    ],
    "type" : "security_exception",
    "reason" : "no permissions for [indices:admin/data_stream/get] and User [name=sandesh1, backend_roles=[], requestedTenant=null]"
  },
  "status" : 403
}

kusandes@a483e789b681 ~ % curl -XGET 'https://localhost:9200/_data_stream/logs-nginx1,logs-nginx11?pretty' --insecure -u 'sandesh1:kumar'
{
  "data_streams" : [
    {
      "name" : "logs-nginx1",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-logs-nginx1-000001",
          "index_uuid" : "3V1VOJ4NQ0KEqtrJ51el8g"
        }
      ],
      "generation" : 1,
      "status" : "YELLOW",
      "template" : "logs-template"
    },
    {
      "name" : "logs-nginx11",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-logs-nginx11-000001",
          "index_uuid" : "lnMe0uUPToKbQKJCrMUoSw"
        }
      ],
      "generation" : 1,
      "status" : "YELLOW",
      "template" : "logs-template"
    }
  ]
}

Issues Resolved

#1498

Is this a backport? If so, please add backport PR # and/or commits #

Testing

• Tested the above behaviour/fix manually with get/delete/stats data stream api.
• Added integration tests for the same to verify.

Check List

• [Y] New functionality includes testing
• [Y] New functionality has been documented
• [Y] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #1716 (6749cd6) into 1.3 (09b3b54) will decrease coverage by 0.00%.
The diff coverage is 60.00%.

@@             Coverage Diff              @@
##                1.3    #1716      +/-   ##
============================================
- Coverage     64.62%   64.62%   -0.01%     
- Complexity     3216     3219       +3     
============================================
  Files           247      247              
  Lines         17351    17363      +12     
  Branches       3082     3086       +4     
============================================
+ Hits          11213    11220       +7     
- Misses         4591     4594       +3     
- Partials       1547     1549       +2     

Impacted Files	Coverage Δ
...earch/security/resolver/IndexResolverReplacer.java	64.16% <60.00%> (+0.52%)	⬆️
...urity/ssl/transport/SecuritySSLNettyTransport.java	69.14% <0.00%> (-4.26%)	⬇️
...earch/security/ssl/util/SSLConnectionTestUtil.java	93.18% <0.00%> (-2.28%)	⬇️
...search/security/configuration/DlsFlsValveImpl.java	71.73% <0.00%> (+1.51%)	⬆️
...security/privileges/TermsAggregationEvaluator.java	61.29% <0.00%> (+3.22%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 09b3b54...6749cd6. Read the comment docs.

[cliu123]

LGTM!

[zengyan-amazon]

I have a questions which is not specific to this change, just wondering if this use case is supported:

one data stream is backed by a list of indices, e.g. data stream my-data-stream is backed by indices .ds_my-data-steram_1 to .ds_my-data-steram_{n} and one user has permission to access data stream my-data-stream, can the user query using the concrete index name .ds_my-data-steram_1, e.g. POST /.ds_my-data-steram_1/_search will it be allowed or denied?

[sandeshkr419]

I have a questions which is not specific to this change, just wondering if this use case is supported:

one data stream is backed by a list of indices, e.g. data stream my-data-stream is backed by indices .ds_my-data-steram_1 to .ds_my-data-steram_{n} and one user has permission to access data stream my-data-stream, can the user query using the concrete index name .ds_my-data-steram_1, e.g. POST /.ds_my-data-steram_1/_search will it be allowed or denied?

They will be denied.
For backing indices, the user will have to specify the appropriate index patterns if they want to access specific indices of a data stream (for eg: something like *my-data-stream_1* for search specific actions in this case).

[peternied]

This was our mistake this change should not be released with 1.3.1, I've reverted it and this can be reopened against main

[cliu123]

Hi @sandeshkr419, we had to revert your commit because 1.3.1 release is being built on 1.3 branch today.
Can you create a PR against main branch with the changes? We have auto backport tool on Github that will automatically backport your changes to the old branches. So you won't have to worry about backport.

[sandeshkr419]

Thanks @peternied & @cliu123

Raised the PR against main branch:
#1723

Please review & merge the above PR and help me backport them to older branches as well.

[sandeshkr419]

@zengyan-amazon To clarify the doubts about permissions regarding backing indices of data stream - I have added relevant test cases as well in the new PR - #1723