Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add new Drain tokenizer that splits on most punctuation #13143

Merged
merged 6 commits into from
Jun 7, 2024
Merged

feat: Add new Drain tokenizer that splits on most punctuation #13143

merged 6 commits into from
Jun 7, 2024

Commits on Jun 5, 2024

  1. Use similar drain params to compare 

    Optimize output drain
    @benclive
    benclive committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    c05f6be View commit details
    Browse the repository at this point in the history

  2. Clean up & add placeholder deduplication step

    @benclive
    benclive committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    8279ac1 View commit details
    Browse the repository at this point in the history

  3. Remove unused regex

    @benclive
    benclive committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    58ca640 View commit details
    Browse the repository at this point in the history

  4. Update tests

    @benclive
    benclive committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    6c618bb View commit details
    Browse the repository at this point in the history

  5. Clean up & tests

    @benclive
    benclive committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    9314e8f View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. golangci-lint fixes

    @benclive
    benclive committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    b15a41e View commit details
    Browse the repository at this point in the history