Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clo: Add orig_file_id and log_event_ix to search results. #424

Merged
merged 9 commits into from
Jun 12, 2024

Conversation

haiqi96
Copy link
Contributor

@haiqi96 haiqi96 commented Jun 4, 2024

References

Description

Current CLP's search results contain the original message, timestamp and original file path. However, the original file path is not sufficient to identify the file if multiple file shares the same path (if they are from different file system, or they are both rotated log with the same file name).

In this PR, we add file's original id and the log event index to every search result, which allows us to locate the specific file split that contains the matching result.

The change can be viewed as two smaller change:

  1. We add a new member variable "log_event_ix" in the Message.hpp that tracks the index in the original file. The original "message_number" is renamed to "ix_in_split" to improve clarity.
  2. We updated the add_results methods in the output handler to add log_event_ix and original_file_id to the search results output by clo executable.

Note that the new fields are not used in the code yet.

Validation performed

Built the package and compress a log file locally, Verified that

  1. The results can be returned as normal by running search.sh
  2. There's no issue when performing a search from webui.
  3. Verifiy that extra fields are properly inserted into the results cache

@haiqi96 haiqi96 changed the title Draft: Add log_event core: Add original_file_id and log_event_ix to clp search results Jun 4, 2024
@haiqi96 haiqi96 changed the title core: Add original_file_id and log_event_ix to clp search results clp: Add original_file_id and log_event_ix to clp search results Jun 10, 2024
components/core/src/clp/clo/OutputHandler.hpp Outdated Show resolved Hide resolved
components/core/src/clp/clo/OutputHandler.hpp Outdated Show resolved Hide resolved
components/core/src/clp/clo/OutputHandler.hpp Outdated Show resolved Hide resolved
components/core/src/clp/clo/OutputHandler.cpp Show resolved Hide resolved
components/core/src/clp/clo/OutputHandler.cpp Outdated Show resolved Hide resolved
@haiqi96 haiqi96 requested a review from kirkrodrigues June 12, 2024 18:38
Copy link
Member

@kirkrodrigues kirkrodrigues left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the PR title, how about:

clo: Add orig_file_id and log_event_ix to search results.

@haiqi96 haiqi96 changed the title clp: Add original_file_id and log_event_ix to clp search results clo: Add orig_file_id and log_event_ix to search results. Jun 12, 2024
@haiqi96 haiqi96 merged commit 7116bdf into y-scope:main Jun 12, 2024
11 checks passed
@haiqi96 haiqi96 deleted the ExtraFields branch June 28, 2024 14:43
jackluo923 pushed a commit to jackluo923/clp that referenced this pull request Dec 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants