Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML-DataFrame] Combine task_state and indexer_state in stats #45201

Closed
droberts195 opened this issue Aug 5, 2019 · 1 comment · Fixed by #45276
Closed

[ML-DataFrame] Combine task_state and indexer_state in stats #45201

droberts195 opened this issue Aug 5, 2019 · 1 comment · Fixed by #45276
Assignees
@droberts195
Labels
:ml/Transform Transform v7.4.0

Comments

@droberts195
Copy link
Contributor

#43767 moved indexer_state into checkpointing.next, and it has been pointed out that this means it is only available when a checkpoint is in progress.

From an end user perspective the difference between task_state and indexer_state is an internal implementation detail. But for debugging purposes we might want to see it even when there isn't a checkpoint in progress. If we move it to the top level then as an end user I'm back to wondering which of task_state and indexer_state I should be taking notice of, and why there are two states in the first place. A better alternative is to have just one top level state that combines the two, like anomaly detection jobs and datafeeds have. It can be defined as:

  • failed if what's currently reported as task_state is failed
  • stopped if there is no persistent task
  • Otherwise what's currently reported as indexer_state

To avoid multiple breaking changes to the stats format in consecutive versions and complex BWC this change should be made for 7.4.
@droberts195 droberts195 self-assigned this Aug 5, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core

droberts195 added a commit to droberts195/elasticsearch that referenced this issue Aug 7, 2019
@droberts195
[ML-DataFrame] Combine task_state and indexer_state in _stats
f5324b4 
This commit replaces task_state and indexer_state in the
data frame _stats output with a single top level state
that combines the two. It is defined as:

- failed if what's currently reported as task_state is failed
- stopped if there is no persistent task
- Otherwise what's currently reported as indexer_state

Closes elastic#45201
@droberts195 droberts195 mentioned this issue Aug 7, 2019
Merged
droberts195 added a commit that referenced this issue Aug 7, 2019
@droberts195
[ML-DataFrame] Combine task_state and indexer_state in _stats (#45276)
65b5020 
This commit replaces task_state and indexer_state in the
data frame _stats output with a single top level state
that combines the two. It is defined as:

- failed if what's currently reported as task_state is failed
- stopped if there is no persistent task
- Otherwise what's currently reported as indexer_state

Closes #45201
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@droberts195 droberts195

Labels
:ml/Transform Transform v7.4.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML-DataFrame] Combine task_state and indexer_state in _stats droberts195/elasticsearch
2 participants
@droberts195 @elasticmachine