Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Allow Transform job to target alias index #656

Open
dragosrosculete opened this issue Jan 10, 2023 · 3 comments
Open

[FEATURE] Allow Transform job to target alias index #656

dragosrosculete opened this issue Jan 10, 2023 · 3 comments
Assignees
Labels

Comments

@dragosrosculete
Copy link

Right now Transform can only target an existing index or create a new one. It does not work if an alias index is provided.
This makes it impossible to automate the process and integrate with rollover jobs .

More details:
Lets take the following indexes as example, created hourly

elb-metrics-2022-12-19-01;
elb-metrics-2022-12-19-02;
elb-metrics-2022-12-19-03 ; etc ..

From transform job we can select as a source a wildcard of indexes, for example elb-metrics-* ( this will include present and future indexes). But on the target index I only have the option to select an existing index, for example transformed-metrics or it will create this one if it doesn’t already exist .
Now this transform jobs runs continuously, so once it is set up, it will write from source (wildcard indexes) to this transformed-metrics index.
The problem is that this index transformed-metrics will eventually have months to years of data but we only want to keep it for a few months.
And there is no way to rotate or create transformed-metrics-2 because the continuous job is set to write into transformed-metrics.
I want to be able to write in a target alias if possible so I can then rotate these indexes and keep only some.

I tried to have Transform to a write into an alias index but this doesn’t work, it does not support to target an alias
Error: Failed to index the documents

@bowenlan-amzn
Copy link
Member

Similar request for rollup
#61
#445

@bowenlan-amzn
Copy link
Member

#226 is related and best to be resolved along with this one

@n-dohrmann
Copy link
Contributor

Could I be assigned to this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: In Progress
Development

No branches or pull requests

4 participants