[Ingest Manager] Add namespace validation #75381

jen-huang · 2020-08-18T22:58:20Z

Summary

Resolves #74667. This PR adds additional validation for namespace field on agent and package policies. As we use the namespace field in the last part of our indexing strategy, the validation method partially implements ES index name rules, omitting the irrelevant ones.

The validation is applied on our API routes using a custom validation schema, as well as UI:

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2020-08-18T23:43:48Z

Pinging @elastic/ingest-management (Team:Ingest Management)

nchaulet

🚀

jen-huang · 2020-08-20T20:02:18Z

@elasticmachine merge upstream

kibanamachine · 2020-08-20T22:19:12Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: dd62085

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
ingestManager	518	+1	517

async chunks size

id	value	diff	baseline
ingestManager	1.1MB	+582.0B	1.1MB

page load bundle size

id	value	diff	baseline
ingestManager	466.4KB	+507.0B	465.9KB

History

💚 Build #69161 succeeded 40cecb4
💚 Build #68945 succeeded 00d5d90
💔 Build #68932 failed d236d4a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Add namespace validation on APIs and UI * Add test coverage * Fix imports * Fix schema * Rename to policy * Fix typo

* Add namespace validation on APIs and UI * Add test coverage * Fix imports * Fix schema * Rename to policy * Fix typo # Conflicts: # x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_policy/components/agent_policy_form.tsx # x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_policy/create_package_policy_page/services/validate_package_policy.ts # x-pack/plugins/ingest_manager/public/applications/ingest_manager/services/index.ts # x-pack/plugins/ingest_manager/server/types/models/agent_policy.ts # x-pack/plugins/ingest_manager/server/types/models/package_policy.ts # x-pack/test/ingest_manager_api_integration/apis/agent_policy/agent_policy.ts # x-pack/test/ingest_manager_api_integration/apis/package_policy/create.ts

* Add namespace validation on APIs and UI * Add test coverage * Fix imports * Fix schema * Rename to policy * Fix typo

* Add namespace validation on APIs and UI * Add test coverage * Fix imports * Fix schema * Rename to policy * Fix typo # Conflicts: # x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_policy/components/agent_policy_form.tsx # x-pack/plugins/ingest_manager/public/applications/ingest_manager/sections/agent_policy/create_package_policy_page/services/validate_package_policy.ts # x-pack/plugins/ingest_manager/public/applications/ingest_manager/services/index.ts # x-pack/plugins/ingest_manager/server/types/models/agent_policy.ts # x-pack/plugins/ingest_manager/server/types/models/package_policy.ts # x-pack/test/ingest_manager_api_integration/apis/agent_policy/agent_policy.ts # x-pack/test/ingest_manager_api_integration/apis/package_policy/create.ts

* Add namespace validation on APIs and UI * Add test coverage * Fix imports * Fix schema * Rename to policy * Fix typo

michalpristas · 2020-08-25T07:17:29Z

x-pack/plugins/ingest_manager/common/services/is_valid_namespace.ts

+  return (
+    typeof namespace === 'string' &&
+    // Lowercase only
+    namespace === namespace.toLowerCase() &&


just a few cases missing which agent validates

namespace is allowed to contain . but is not allowed to be equal to . or ..

namespace cannot be longer than 255 chars

namespace cannot start with -, _ or +

@michalpristas I think the validation is incorrect,

Cannot be longer than 255 bytes (note it is bytes, so multi-byte characters will count towards the 255 limit faster)

This mean, type + dataset + namespace <= 255 bytes.

what we do in agent is that we put 255 constraint on dataset, namespace and a product of concatenation, so eventaully we will fail when index > 255.
putting some not communicated constraint felt weird (such as 255/3 or some other math) so i used 255 on each part and result as well.
as namespace is created before configuration and can lead to different indexes based on dataset and type it would make sense to agree on some constraint which can be applied at the time of specifying namespace.

cc @jen-huang

I followed up in #75846 (comment).

ghost · 2020-08-31T07:04:07Z

Hi @jen-huang /@EricDavisX

We have validated this ticket on latest 7.9.1-SNAPSHOT Kibana cloud environment and found it fixed.

Provide namespace value starting with capital letter in agent configuration or integration while adding and editing them.

Observations:

Error message "Namespace contains invalid characters" is displayed on providing namespace starting with capital letter while creating or editing agent configuration.
Error message "Namespace contains invalid characters" is displayed on providing namespace starting with capital letter while adding or editing integration present in agent configuration.

Screenshot:

While creating or editing agent configuration :-
While creating or editing integration in agent configuration :-

We have created and executed 04 testcases under Add namespace validation TestRun.

Testcases created:
https://elastic.testrail.io/index.php?/cases/view/27064
https://elastic.testrail.io/index.php?/cases/view/27066
https://elastic.testrail.io/index.php?/cases/view/27063
https://elastic.testrail.io/index.php?/cases/view/27065

Hence, we are closing the ticket.

EricDavisX · 2020-08-31T13:51:25Z

Thanks Rahul

jen-huang added 2 commits August 18, 2020 15:46

Add namespace validation on APIs and UI

3c09ab8

Add test coverage

d236d4a

jen-huang added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.10.0 Team:Fleet Team label for Observability Data Collection Fleet team v7.9.1 labels Aug 18, 2020

jen-huang self-assigned this Aug 18, 2020

jen-huang added 2 commits August 18, 2020 16:06

Fix imports

4aa1f79

Fix schema

00d5d90

jen-huang marked this pull request as ready for review August 18, 2020 23:43

jen-huang requested a review from a team August 18, 2020 23:43

jen-huang mentioned this pull request Aug 18, 2020

[Ingest Manager] [BUG] Using namespace with capital letters causes agent to not install beats/endpoint #74667

Closed

ph mentioned this pull request Aug 19, 2020

[Elastic Agent] Enforce validation on namespace elastic/beats#20693

Closed

jen-huang added 3 commits August 19, 2020 10:53

Merge remote-tracking branch 'upstream/master' into ingest/bug/74667

40cecb4

Merge remote-tracking branch 'upstream/master' into ingest/bug/74667

1e50244

Rename to policy

5deb2d5

nchaulet approved these changes Aug 20, 2020

View reviewed changes

jen-huang added 2 commits August 20, 2020 10:26

Fix typo

848ced9

Merge remote-tracking branch 'upstream/master' into ingest/bug/74667

3210347

Merge remote-tracking branch 'upstream/master' into ingest/bug/74667

dd62085

jen-huang merged commit 3201efe into elastic:master Aug 20, 2020

jen-huang deleted the ingest/bug/74667 branch August 20, 2020 22:30

jen-huang mentioned this pull request Aug 20, 2020

[7.x] [Ingest Manager] Add namespace validation (#75381) #75611

Merged

jen-huang added a commit to jen-huang/kibana that referenced this pull request Aug 20, 2020

[Ingest Manager] Add namespace validation (elastic#75381)

414ea20

* Add namespace validation on APIs and UI * Add test coverage * Fix imports * Fix schema * Rename to policy * Fix typo

jen-huang mentioned this pull request Aug 20, 2020

[7.9] [Ingest Manager] Add namespace validation (#75381) #75612

Merged

jen-huang added a commit that referenced this pull request Aug 21, 2020

[Ingest Manager] Add namespace validation (#75381) (#75611)

67a914d

* Add namespace validation on APIs and UI * Add test coverage * Fix imports * Fix schema * Rename to policy * Fix typo

thomasneirynck pushed a commit to thomasneirynck/kibana that referenced this pull request Aug 21, 2020

[Ingest Manager] Add namespace validation (elastic#75381)

31f1a83

* Add namespace validation on APIs and UI * Add test coverage * Fix imports * Fix schema * Rename to policy * Fix typo

michalpristas reviewed Aug 25, 2020

View reviewed changes

jen-huang mentioned this pull request Aug 25, 2020

[Ingest Manager] Align namespace validation rules #75846

Closed

jen-huang mentioned this pull request Sep 25, 2020

[Ingest Manager] Add namespace max length limit #78522

Merged

jen-huang mentioned this pull request Oct 7, 2020

Error "Namespace contains invalid characters" is displayed while provide name in "Default namespace" with special characters #(hash) or *(asterisk) under policies page. #79818

Closed

jen-huang mentioned this pull request Mar 22, 2021

[Fleet] Namespace section in Fleet Agent Policies should disallow special characters #95075

Closed

jsoriano mentioned this pull request Sep 20, 2022

Validate dataset and namespace elastic/package-spec#57

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Add namespace validation #75381

[Ingest Manager] Add namespace validation #75381

jen-huang commented Aug 18, 2020 •

edited

Loading

elasticmachine commented Aug 18, 2020

nchaulet left a comment

jen-huang commented Aug 20, 2020

kibanamachine commented Aug 20, 2020

michalpristas Aug 25, 2020

ph Aug 25, 2020

michalpristas Aug 25, 2020

michalpristas Aug 25, 2020

jen-huang Aug 25, 2020

ghost commented Aug 31, 2020

EricDavisX commented Aug 31, 2020

[Ingest Manager] Add namespace validation #75381

[Ingest Manager] Add namespace validation #75381

Conversation

jen-huang commented Aug 18, 2020 • edited Loading

Summary

Checklist

elasticmachine commented Aug 18, 2020

nchaulet left a comment

Choose a reason for hiding this comment

jen-huang commented Aug 20, 2020

kibanamachine commented Aug 20, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

async chunks size

page load bundle size

History

michalpristas Aug 25, 2020

Choose a reason for hiding this comment

ph Aug 25, 2020

Choose a reason for hiding this comment

michalpristas Aug 25, 2020

Choose a reason for hiding this comment

michalpristas Aug 25, 2020

Choose a reason for hiding this comment

jen-huang Aug 25, 2020

Choose a reason for hiding this comment

ghost commented Aug 31, 2020

EricDavisX commented Aug 31, 2020

jen-huang commented Aug 18, 2020 •

edited

Loading