Updated README with app level index routing #292

hvaghani221 · 2021-09-24T11:09:45Z

No description provided.

luckyj5

Lets also add that "ADD_APP_INFO" (cache info) is required to be enabled for per app index routing feature.
If Add-app-info is not enabled, nozzle will not have app's metadata which will include SPLUNK_INDEX value.

luckyj5 · 2021-09-24T21:14:47Z

README.md

+
+You can also update the env on the fly using cf-cli command:
+```
+cf set-env <APP_NAME> SPLUNK_INDEX <ENV_VAR_VALUE>


I think for this, if you set-env via CLI, you still need to restage the application for those updated envs to take effect. Please cross check. Thanks!

I think for this, if you set-env via CLI, you still need to restage the application for those updated envs to take effect. Please cross check. Thanks!

In my testing, I was able to route data-gen events to the index provided using cf command. I tested following scenarios:

deployed data_gen without setting SPLUNK_INDEX env variable and set env after it is deployed

deployed data_gen with setting SPLUNK_INDEX env variable and update env with new index value

In both cases, event were ingested in newly specified index.

I think, if the app is using the environment variable then we are required to restage the application. For our use case, app doesn't rely on SPLUNK_INDEX, so no need to restage. Cloud controller endpoint will always returns latest env data. So as long as connector is invalidating app cache, it should be fine

hvaghani221 · 2021-09-27T05:48:03Z

Lets also add that "ADD_APP_INFO" (cache info) is required to be enabled for per app index routing feature.
If Add-app-info is not enabled, nozzle will not have app's metadata which will include SPLUNK_INDEX value.

It doen't depend on ADD_APP_INFO config. As you can see in router, as long as event has app_id, it will call event.AnnotateWithAppData method.
In event.AnnotateWithAppData method, it gets app information from cache from cf_app_id. If it find's SPLUNK_INDEX in app env, it'll add info_splunk_index field in the event

luckyj5 · 2021-09-28T17:33:02Z

Lets also add that "ADD_APP_INFO" (cache info) is required to be enabled for per app index routing feature.
If Add-app-info is not enabled, nozzle will not have app's metadata which will include SPLUNK_INDEX value.

It doen't depend on ADD_APP_INFO config. As you can see in router, as long as event has app_id, it will call event.AnnotateWithAppData method. In event.AnnotateWithAppData method, it gets app information from cache from cf_app_id. If it find's SPLUNK_INDEX in app env, it'll add info_splunk_index field in the event

ADD_APP_INFO config triggers nozzle to fetch cache info for apps. If it's not selected, then nozzle won't have cache info, which means it won't have env variables either. Let me know if it's otherwise.

hvaghani221 · 2021-09-29T05:05:37Z

Lets also add that "ADD_APP_INFO" (cache info) is required to be enabled for per app index routing feature.
If Add-app-info is not enabled, nozzle will not have app's metadata which will include SPLUNK_INDEX value.

It doen't depend on ADD_APP_INFO config. As you can see in router, as long as event has app_id, it will call event.AnnotateWithAppData method. In event.AnnotateWithAppData method, it gets app information from cache from cf_app_id. If it find's SPLUNK_INDEX in app env, it'll add info_splunk_index field in the event

ADD_APP_INFO config triggers nozzle to fetch cache info for apps. If it's not selected, then nozzle won't have cache info, which means it won't have env variables either. Let me know if it's otherwise.

On further analysis, found that boltdb cache is initialised only if ADD_APP_INFO is enabled otherwise NoCache is initialised. And app info will never be fetched from remote.
I'll make changes according to that.
Thanks.

luckyj5 · 2021-09-29T17:57:31Z

README.md

 Index routing is a feature that can be used to send different Cloud Foundry logs to different indexes for better ACL and data retention control in Splunk.

-#### Per application index routing via application manifest
+### Per application index routing via application manifest
+To enable per app index routing, you are required to set environmt variable `SPLUNK_INDEX`in your application and enable ADD_APP_INFO(Select at least one metadata) to enable app info caching. Make sure that given HEC token can ingest events to the SPLUNK_INDEX.


Please update as below:

To enable per application index routing:

Please set environment variable SPLUNK_INDEX in your application's manifest (example below - link to the example)

Make sure Splunk nozzle is configured with ADD_APP_INFO (Select at least one of AppName, OrgName/Id, SpaceName/id) to enable app info caching

Make sure SPLUNK_INDEX specified in app's manifest exist in Splunk and can receive data for the configured Splunk HEC token.

luckyj5 · 2021-09-30T17:52:48Z

Thanks for the update @harshit-splunk.

luckyj5

Just noticed this - note for SPLUNK_VERSION
Please remove this as well.

luckyj5

LGTM

* update docs (#276) * Update Go version and migrate to Go modules (#287) * Adding go.mod and go.sum to migrate to Go modules from Glide * Removed old Vendor folder after migrating to Go modules * Adding new Vendor folder after migrating to Go mudules * Removed Glide files, references in readme and updated CI/CD * Migrated from bolt to bbolt for cache module to resolve race issue * Added tags from the envelope to the events * Updated tile.yml with new config * Updated README * Updated migration script to prevent unnecessary migration (#288) * Updated cache.GetApp logic to retrieve app info from boltdb database as well * Bugfix/disconnect due to slow consumer event drop (#289) * dropping events when the queue is full * Corrected the newly added test case with data race condition * minor change in logging * added event drop warning threshold in config * minor update in the logging * Updated comments and log info * Trimmed white space for ApiEndPoint and SplunkHost * Updated docs with required field & optional value * Added default value in README * Updated events doc * Removed SPLUNK_VERSION config as not needed anymore * Removed SplunkVersion from eventsink.splunk_test * Updated README with app level index routing (#292) * Updated README with app level index routing * Removed unnecessary changes * added required config to enable index routing * updated index routing doc * added warning in index routing docs * upgraded module version * upgraded gogo/protobuf version * executed go mod tidy * upgraded indirect dependency * updated version in a build example in readme * Corrected index routing docs * Added more details (#299) * Updated docs around - Add Tags: Performance impact of adding tags in events - How to detect slow downstream related issues and event drops * added 'drop_warn_threshold' config in tile (#300) Co-authored-by: Matthew Heidemann <[email protected]> Co-authored-by: Shubham Jain <[email protected]> Co-authored-by: harshit-splunk <[email protected]> Co-authored-by: harshit-splunk <[email protected]>

hvaghani221 requested review from luckyj5 and kashyap-splunk September 24, 2021 11:09

luckyj5 requested changes Sep 24, 2021

View reviewed changes

hvaghani221 requested a review from luckyj5 September 27, 2021 05:56

harshit-splunk added 3 commits September 29, 2021 14:58

Updated README with app level index routing

2fa57e1

Removed unnecessary changes

bb15592

added required config to enable index routing

1a00c2d

hvaghani221 force-pushed the readme-docs-update branch from 39d3254 to 1a00c2d Compare September 29, 2021 09:56

removed unnecessary changes

e620ce3

kashyap-splunk approved these changes Sep 29, 2021

View reviewed changes

luckyj5 reviewed Sep 29, 2021

View reviewed changes

updated index routing doc

663d94d

luckyj5 self-requested a review September 30, 2021 17:53

luckyj5 mentioned this pull request Sep 30, 2021

"[Docs update]" Docs update - update docs to reflect required config for app level index routing to work #281

Closed

luckyj5 requested changes Sep 30, 2021

View reviewed changes

added warning in index routing docs

83b07d2

hvaghani221 mentioned this pull request Oct 1, 2021

[BUG] Events of other apps get lost when a single app is using Index Routing with invalid index #295

Closed

luckyj5 approved these changes Oct 1, 2021

View reviewed changes

luckyj5 merged commit 2bebecf into develop Oct 1, 2021

luckyj5 deleted the readme-docs-update branch October 1, 2021 21:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated README with app level index routing #292

Updated README with app level index routing #292

hvaghani221 commented Sep 24, 2021

luckyj5 left a comment

luckyj5 Sep 24, 2021

hvaghani221 Sep 27, 2021 •

edited

Loading

hvaghani221 commented Sep 27, 2021 •

edited

Loading

luckyj5 commented Sep 28, 2021

hvaghani221 commented Sep 29, 2021 •

edited

Loading

luckyj5 Sep 29, 2021

luckyj5 commented Sep 30, 2021 •

edited

Loading

luckyj5 left a comment •

edited

Loading

luckyj5 left a comment

Updated README with app level index routing #292

Updated README with app level index routing #292

Conversation

hvaghani221 commented Sep 24, 2021

luckyj5 left a comment

Choose a reason for hiding this comment

luckyj5 Sep 24, 2021

Choose a reason for hiding this comment

hvaghani221 Sep 27, 2021 • edited Loading

Choose a reason for hiding this comment

hvaghani221 commented Sep 27, 2021 • edited Loading

luckyj5 commented Sep 28, 2021

hvaghani221 commented Sep 29, 2021 • edited Loading

luckyj5 Sep 29, 2021

Choose a reason for hiding this comment

luckyj5 commented Sep 30, 2021 • edited Loading

luckyj5 left a comment • edited Loading

Choose a reason for hiding this comment

luckyj5 left a comment

Choose a reason for hiding this comment

hvaghani221 Sep 27, 2021 •

edited

Loading

hvaghani221 commented Sep 27, 2021 •

edited

Loading

hvaghani221 commented Sep 29, 2021 •

edited

Loading

luckyj5 commented Sep 30, 2021 •

edited

Loading

luckyj5 left a comment •

edited

Loading