fix(remap): Preserve type defs when assigning fields #7118
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, remap would overwrite the type def of `.` whenever a new field was assigned. That is: ``` .foo = 5 .bar = 6 ``` Would result in the compiler only having type info for `.bar`. This change causes it to merge the typedefs whenever a list of path segments appears. Signed-off-by: Jesse Szwedko <[email protected]>
jszwedko
requested review from
StephenWakely,
JeanMertz,
a team and
pablosichert
and removed request for
a team
StephenWakely
approved these changes
Apr 14, 2021
There was a problem hiding this comment.
This all looks sound according to our investigations, but I would really like to understand why the code was originally only merging for indexed paths. There could well be some angle we are still missing. Perhaps @JeanMertz has some insight?
Signed-off-by: Jesse Szwedko <[email protected]>
JeanMertz
approved these changes
Apr 16, 2021
There was a problem hiding this comment.
Interesting find. This seems like a genuine oversight, but it's remarkable that none of our tests actually caught this (although looking at the changed Cue doc, it does make sense that split would have to be infallible, since the regex will always set .message to a string field).
Good catch, and good fix 👍
jszwedko
added a commit
that referenced
this pull request
Apr 16, 2021
I broke them in #7118 Signed-off-by: Jesse Szwedko <[email protected]>
jszwedko
added a commit
that referenced
this pull request
Apr 17, 2021
* chore(performance): Fix remap benches I broke them in #7118 Signed-off-by: Jesse Szwedko <[email protected]>
langecode
added a commit
to neticdk/vector
that referenced
this pull request
May 31, 2021
* chore(deps): bump tokio-util from 0.6.5 to 0.6.6 (#7103) Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.6.5 to 0.6.6. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.5...tokio-util-0.6.6) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(ci): bump actions/cache from v2.1.4 to v2.1.5 (#7104) Bumps [actions/cache](https://github.com/actions/cache) from v2.1.4 to v2.1.5. - [Release notes](https://github.com/actions/cache/releases) - [Commits](https://github.com/actions/cache/compare/v2.1.4...1a9e2138d905efd099035b49d8b7a3888c653ca8) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(datadog service): Document region parameter (#7093) * fix(datadog service): Document region parameter Closes #7079 This was added to the shared Datadog documentation in https://github.com/timberio/vector/pull/4174 but never added specifically to the sinks. Signed-off-by: Jesse Szwedko <[email protected]> * Add missing cue config for region Signed-off-by: Jesse Szwedko <[email protected]> * fix(socket source): Remove socket file on source shutdown (#7047) * Delete socket Signed-off-by: ktf <[email protected]> * Add tests Signed-off-by: ktf <[email protected]> * Remove spaces Signed-off-by: ktf <[email protected]> * chore(deps): bump reqwest from 0.11.2 to 0.11.3 (#7100) Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.2 to 0.11.3. - [Release notes](https://github.com/seanmonstar/reqwest/releases) - [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md) - [Commits](https://github.com/seanmonstar/reqwest/compare/v0.11.2...v0.11.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(clickhouse sink): Fix clickhouse sink incorrectly encodes arrays (#7081) * fix bug:clickhouse sink incorrectly encodes arrays Signed-off-by: shumin <[email protected]> * Add test for clickhouse arrays Signed-off-by: Jesse Szwedko <[email protected]> Co-authored-by: shumin <[email protected]> Co-authored-by: Jesse Szwedko <[email protected]> * Added collect_subtypes. (#7092) Signed-off-by: Stephen Wakely <[email protected]> * fix(docker_logs source): Docker logs missing when container has a tty (#7111) * fix(docker_logs source): Docker logs missing when container has a tty Enum bollard::container::LogOutput is Console when docker started with a tty. LogOutput::Console was missing in the new_event of a stream value received https://docs.rs/bollard/0.10.1/bollard/container/enum.LogOutput.html Fixes #5903 Signed-off-by: Jean Prat <[email protected]> * chore(ci): Remove unneeded advisory from deny.toml (#7114) This advisory seems to no longer be valid. Closes: #6223 Signed-off-by: Jesse Szwedko <[email protected]> * fix(external docs): Fix link to unit testing documentation (#7122) Fixes #7121 Signed-off-by: Jesse Szwedko <[email protected]> * chore(deps): Upgrade cue to 0.3.2 (#7128) Signed-off-by: Jesse Szwedko <[email protected]> * fix(external docs): Document additional encoding fields (#7127) Ensure `except_fields`, `only_fields`, and `timestamp_format` appear on all sinks supporting `encoding`. Previously they only appeared on sinks that had healthchecks. Fixes: #6949 Signed-off-by: Jesse Szwedko <[email protected]> * chore(ci): Add advisory RUSTSEC-2021-0013 back to deny.toml (#7133) * chore(ci): Add advisory RUSTSEC-2021-0013 back to deny.toml I neglected to enable all features. The `wasm` dependencies still depend on an affected version of rust-cpuid. This reverts commit f30d4c6ed43838b36769753f2a591c2378162e04. Signed-off-by: Jesse Szwedko <[email protected]> * Run cargo deny in CI if deny.toml changes Signed-off-by: Jesse Szwedko <[email protected]> * chore(deps): bump async-trait from 0.1.48 to 0.1.49 (#7131) Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.48 to 0.1.49. - [Release notes](https://github.com/dtolnay/async-trait/releases) - [Commits](https://github.com/dtolnay/async-trait/compare/0.1.48...0.1.49) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(ci): Ensure all benchmark artifacts are included (#7113) * fix(ci): Ensure all benchmark artaficts are included CRITERION_HOME having "s was causing it to put it in the wrong place (locally it created a directory called "). Also just upload `./target/criterion` since Github does the zipping for us. We did need to zip when previously downloading the artifact in the same workflow as this was not zipped yet and so we were hitting limits trying to individually pull a very large number of files. Signed-off-by: Jesse Szwedko <[email protected]> * chore(deps): bump tokio from 1.4.0 to 1.5.0 (#7102) Bumps [tokio](https://github.com/tokio-rs/tokio) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/tokio-rs/tokio/releases) - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-1.4.0...tokio-1.5.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump notify from 4.0.15 to 4.0.16 (#7142) Bumps [notify](https://github.com/notify-rs/notify) from 4.0.15 to 4.0.16. - [Release notes](https://github.com/notify-rs/notify/releases) - [Changelog](https://github.com/notify-rs/notify/blob/v4.0.16/CHANGELOG.md) - [Commits](https://github.com/notify-rs/notify/compare/v4.0.15...v4.0.16) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(remap): Preserve type defs when assigning fields (#7118) * fix(remap): Preserve type defs when assigning fields Previously, remap would overwrite the type def of `.` whenever a new field was assigned. That is: ``` .foo = 5 .bar = 6 ``` Would result in the compiler only having type info for `.bar`. This change causes it to merge the typedefs whenever a list of path segments appears. Signed-off-by: Jesse Szwedko <[email protected]> * chore(deps): bump pretty_assertions from 0.7.1 to 0.7.2 (#7143) Bumps [pretty_assertions](https://github.com/colin-kiegel/rust-pretty-assertions) from 0.7.1 to 0.7.2. - [Release notes](https://github.com/colin-kiegel/rust-pretty-assertions/releases) - [Changelog](https://github.com/colin-kiegel/rust-pretty-assertions/blob/main/CHANGELOG.md) - [Commits](https://github.com/colin-kiegel/rust-pretty-assertions/compare/v0.7.1...v0.7.2) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(kubernetes_logs source): Always use file checkpoints if they exist (#7140) * fix(kubernetes_logs source): Always use file checkpoints if they exist The `kubernetes_logs` source exposes a `PathProvider` that breaks one of the `FileServer`s assumptions that all available files will be listed at Vector startup time. Instead, the files are only returned once the k8s metadata is available to the `kubernetes_logs` source. This caused the `FileServer` to ignore any checkpoints that existed for these files. As a short-term fix, we just always use the checkpoint, if available, for any new files that are seen. This fixes the case for the `kubernetes_logs` source where they are seen as "new" after start-up. https://github.com/timberio/vector/pull/6564 exists to test this behavior, but it seems to pass even without this change, so that test will need to be updated. Signed-off-by: Jesse Szwedko <[email protected]> * fix(remap): Fix case_sensitive defaults for string matching functions (#7091) Fixes #7044 The docs and examples indicated that it should default to case sensitive; however, `contains`, `starts_with`, and `ends_with`, defaulted to case insensitive matching. Signed-off-by: Jesse Szwedko <[email protected]> * fix(compression): Switch to MultiGzDecoder instead of GzDecoder (#7138) * fix(compression): Switch to MultiGzDecoder instead ef GzDecoder Fixes #7061 It appears that AWS's ALB logging gzip's multi-part files which we were only reading the first part of. I tested that `MultiGzDecoder` works on simple gzip files so I figured we should switch to it everywhere. It was already being used by the `file` source. Signed-off-by: Jesse Szwedko <[email protected]> * Add test for multi-part zst files Signed-off-by: Jesse Szwedko <[email protected]> * chore(performance): Fix remap benches (#7155) * chore(performance): Fix remap benches I broke them in https://github.com/timberio/vector/pull/7118 Signed-off-by: Jesse Szwedko <[email protected]> * chore(deps): `cargo upgrade` (#7154) I backed out a few changes: ``` Upgrading grok v~1.0.1 -> v1.2.0 Upgrading async-graphql-warp v=2.6.4 -> v2.8.2 Upgrading once_cell v1.3 -> v1.7.2 Upgrading async-graphql v=2.6.4 -> v2.8.2 ``` Due to incompatibilities in package dependencies. I backed out ``` Upgrading db-key v0.0.5 -> v0.1.0 ``` Because it requires code changes. Signed-off-by: Jesse Szwedko <[email protected]> * chore(deps): bump rust_decimal from 1.10.3 to 1.11.0 (#7160) Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.10.3 to 1.11.0. - [Release notes](https://github.com/paupino/rust-decimal/releases) - [Commits](https://github.com/paupino/rust-decimal/compare/1.10.3...1.11.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(docker_logs source): ensure that docker labels are flattened (#7152) Fixes: #5716 Previously labels on docker containers were inserted in such a way that dots in their names would end up creating a nested structure in the log event due to the semantics of `LogEvent.insert`. For example: ```json { "container_created_at": "2021-04-16T18:53:19.946155600Z", "container_id": "d6bd69d4bc64bef20b4e992dcc23113741067e8268762694f92899504ae14319", "container_name": "docker_echo_1", "host": "COMP-C02DV25MML87", "image": "hashicorp/http-echo:latest", "label": { "com": { "docker": { "compose": { "config-hash": "e7e5ba19811180f27a7af36667652d0cd686599e6184cb023d9b71d791ff6a1e", "container-number": "1", "oneoff": "False", "project": { "config_files": "docker-compose.yml", "working_dir": "/private/tmp/docker" }, "service": "echo", "version": "1.27.4" } } } }, "message": "2021/04/16 19:14:10 localhost:5678 172.29.0.1:61824 \"GET / HTTP/1.1\" 200 6 \"curl/7.64.1\" 35.6µs", "source_type": "docker", "stream": "stdout", "timestamp": "2021-04-16T19:14:10.400790400Z" } ``` This change ensures that labels are inserted as-is as keys: ```json { "container_created_at": "2021-04-16T18:53:19.946155600Z", "container_id": "d6bd69d4bc64bef20b4e992dcc23113741067e8268762694f92899504ae14319", "container_name": "docker_echo_1", "host": "COMP-C02DV25MML87", "image": "hashicorp/http-echo:latest", "label": { "com.docker.compose.config-hash": "e7e5ba19811180f27a7af36667652d0cd686599e6184cb023d9b71d791ff6a1e", "com.docker.compose.container-number": "1", "com.docker.compose.oneoff": "False", "com.docker.compose.project": "docker", "com.docker.compose.project.config_files": "docker-compose.yml", "com.docker.compose.project.working_dir": "/private/tmp/docker", "com.docker.compose.service": "echo", "com.docker.compose.version": "1.27.4" }, "message": "2021/04/16 19:12:01 localhost:5678 172.29.0.1:61820 \"GET / HTTP/1.1\" 200 6 \"curl/7.64.1\" 18.1µs", "source_type": "docker", "stream": "stdout", "timestamp": "2021-04-16T19:12:01.622769500Z" } ``` Signed-off-by: Jesse Szwedko <[email protected]> * chore(deps): bump gouth from 0.2.0 to 0.2.1 (#7162) Bumps [gouth](https://github.com/mechiru/gouth) from 0.2.0 to 0.2.1. - [Release notes](https://github.com/mechiru/gouth/releases) - [Commits](https://github.com/mechiru/gouth/compare/v0.2.0...v0.2.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump async-trait from 0.1.49 to 0.1.50 (#7164) Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.49 to 0.1.50. - [Release notes](https://github.com/dtolnay/async-trait/releases) - [Commits](https://github.com/dtolnay/async-trait/compare/0.1.49...0.1.50) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): Upgrade curl-sys (#7174) Resolves CVE-2021-22876 CVE-2021-22890 Signed-off-by: Jesse Szwedko <[email protected]> * chore: Extract Core RFC (#7135) This commit introduces a new RFC for extracting the core of vector. Closes #7027. Signed-off-by: Brian L. Troutwine <[email protected]> * chore: Create skeleton `lib/core` (#7178) * Create skeleton `lib/core` Per RFC 7027 we intend to create an extracted vector core. This is the first step in that RFC's plan of action, to create a skeleton package "core" to migrate code into. There's very little going on here, just a new library -- with aggressive clippy settings that we might back out later -- and a correctness and benchmark test. Closes #7147 Signed-off-by: Brian L. Troutwine <[email protected]> * Add newline to the end of Cargo.toml Signed-off-by: Brian L. Troutwine <[email protected]> * Remove unused benchmark Signed-off-by: Brian L. Troutwine <[email protected]> * chore(deps): bump rust_decimal from 1.11.0 to 1.11.1 (#7182) Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.11.0 to 1.11.1. - [Release notes](https://github.com/paupino/rust-decimal/releases) - [Commits](https://github.com/paupino/rust-decimal/compare/1.11.0...1.11.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump httpmock from 0.5.7 to 0.5.8 (#7181) Bumps [httpmock](https://github.com/alexliesenfeld/httpmock) from 0.5.7 to 0.5.8. - [Release notes](https://github.com/alexliesenfeld/httpmock/releases) - [Changelog](https://github.com/alexliesenfeld/httpmock/blob/master/CHANGELOG.md) - [Commits](https://github.com/alexliesenfeld/httpmock/compare/v0.5.7...v0.5.8) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(kubernetes_logs source): Refactor stream and invocation errors to support recoverable error types (#6816) * Refactor stream and invocation errors to support recoverable error types Signed-off-by: Ian Henry <[email protected]> * feat(remap): add to_regex function (#7074) * add to_regex function Signed-off-by: Jake He <[email protected]> Signed-off-by: Jesse Szwedko <[email protected]> Co-authored-by: Jesse Szwedko <[email protected]> * chore(host_metrics source): Document symlink warning workaround (#7179) * chore(host_metrics source): Document symlink warning workaround Signed-off-by: Bruce Guenter <[email protected]> * Fix spacing Signed-off-by: Bruce Guenter <[email protected]> * Add adjustments to the kubernetes helm chart to match documentation Signed-off-by: Bruce Guenter <[email protected]> * enhancement(remap): Support identifiers with leading numeric characters as fields in path (#7045) * Support integers in path Signed-off-by: ktf <[email protected]> * fmt Signed-off-by: ktf <[email protected]> * Refactor Signed-off-by: ktf <[email protected]> * New way Signed-off-by: ktf <[email protected]> * Remove support for .0 Signed-off-by: ktf <[email protected]> * Add docs. Signed-off-by: ktf <[email protected]> * Update name Signed-off-by: ktf <[email protected]> * Rename `core` to `vector-core` (#7195) Rust has its own crate `core` and I didn't anticipate that our own core would collide with it. In retrospect it surely does. REF #7148 Signed-off-by: Brian L. Troutwine <[email protected]> * chore(releasing): Merge down of 0.13.0 release (#7194) * chore(releasing): Prep 0.13.0 release Signed-off-by: Jesse Szwedko <[email protected]> * chore(dev): Bump to 0.14.0 (#7205) Now that 0.13.0 is released. Signed-off-by: Jesse Szwedko <[email protected]> * fix(remap): Escape quotes and backslashes when printing VRL strings (#6739) * Escape quotes when displaying VRL strings Signed-off-by: Pablo Sichert <[email protected]> * Escape backslashes when displaying VRL strings Signed-off-by: Pablo Sichert <[email protected]> * Move shared library to dev dependencies Signed-off-by: Pablo Sichert <[email protected]> * Escape newlines when displaying VRL strings Signed-off-by: Pablo Sichert <[email protected]> Co-authored-by: Jesse Szwedko <[email protected]> * chore(deps): bump cidr-utils from 0.5.1 to 0.5.3 (#7211) Bumps [cidr-utils](https://github.com/magiclen/cidr-utils) from 0.5.1 to 0.5.3. - [Release notes](https://github.com/magiclen/cidr-utils/releases) - [Commits](https://github.com/magiclen/cidr-utils/compare/v0.5.1...v0.5.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump async-stream from 0.3.0 to 0.3.1 (#7209) Bumps [async-stream](https://github.com/tokio-rs/async-stream) from 0.3.0 to 0.3.1. - [Release notes](https://github.com/tokio-rs/async-stream/releases) - [Commits](https://github.com/tokio-rs/async-stream/compare/v0.3.0...v0.3.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump syn from 1.0.69 to 1.0.70 (#7210) Bumps [syn](https://github.com/dtolnay/syn) from 1.0.69 to 1.0.70. - [Release notes](https://github.com/dtolnay/syn/releases) - [Commits](https://github.com/dtolnay/syn/compare/1.0.69...1.0.70) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump hyper from 0.14.5 to 0.14.6 (#7212) Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.5 to 0.14.6. - [Release notes](https://github.com/hyperium/hyper/releases) - [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md) - [Commits](https://github.com/hyperium/hyper/compare/v0.14.5...v0.14.6) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Move `Result` and `Error` into vector-core (#7207) These two types are required for the `Event` interface. I have re-exported them back in the top-level package so no code changes are necessary at that level. Signed-off-by: Brian L. Troutwine <[email protected]> * Remove RUSTSEC-2020-0053 from ignore list since `dirs` is maintained again (#7208) Signed-off-by: Pablo Sichert <[email protected]> * chore(deps): Remove RUSTSEC-2020-0146 from deny.toml (#7216) Appears to no longer be relevant. Signed-off-by: Jesse Szwedko <[email protected]> * chore(performance): Consider dropping jemalloc (#6781) I was originally going to table this after some discussion with Luke about the poor performance of musl's liballoc in multi-threaded environment, however, I had some more discussion with Brian (thank you!) that convinced me that defaulting to the system allocator may still be a better default. An argument follows: The primary motivator for using musl, and thus greatly benefiting from jemalloc in multi-threaded environments, is with alpine docker images. Users typically choose alpine images for size reasons; however, we also publish distroless images that include libc that are only 5MB larger (which, coincidentally is also bump in size that results from including jemalloc; https://github.com/timberio/vector/pull/310). 0.12.1 images: ``` timberio/vector 0.12.1-distroless-libc a01c8409bacb 4 days ago 93MB timberio/vector 0.12.1-distroless-static b61e0880d16c 4 days ago 82.4MB timberio/vector 0.12.1-alpine ef029e56b5ee 4 days ago 88MB timberio/vector 0.12.1-debian 2ccea0b5ffdb 4 days ago 164MB ``` Given this, it seems like we can encourage users to install the `0.12.1-distroless-libc` image as the image of choice for size conscious installs. Distroless also has the security advantage by providing less in the container itself . Additionally, jemalloc is historically preferred for certain workloads but it doesn't seem like we have an understanding of vector's allocation profile yet and which strategies will result in the best performance. Until we have a very strong reason to, just defaulting to the system allocator seems like a good default. Thus the proposal is to: * Default to the system allocator. * Make it easy to swap the allocator to test out different allocators in the future. * Strongly highlight this change with the 0.13 release and encourage anyone using a musl-based build to consider whether the glibc build would be usable for them. * Document this performance difference between glibc and musl's liballoc. Signed-off-by: Jesse Szwedko <[email protected]> * chore(ci): bump actions/github-script from v3.1 to v4.0.1 (#7213) Bumps [actions/github-script](https://github.com/actions/github-script) from v3.1 to v4.0.1. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v3.1...85e88a66eaa831097093a3d278536947f2984d20) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add a cargo-hack workflow (#7224) In #7200 I ran into #7222 and figured we might do well to make cargo-hack use official. These runs will take _a long_ time considering the feature surface of this project so I've made it strictly a nightly run. Signed-off-by: Brian L. Troutwine <[email protected]> * chore: Adjust import of `crate::Event` to `crate::event::Event` (#7200) * Adjust import of `crate::Event` to `crate::event::Event` My end goal here is to move the `Event` and related types into `vector-core`. This is possible but trickier if we maintain a top-level `Event` import in vector and so here I've adjusted all imports of the relevant type to `crate::event::Event`, removing the top-level `Event`. This is all mechanical work and I have intentionally not changed any logic, in deference to the size of this PR. REF #7148 Signed-off-by: Brian L. Troutwine <[email protected]> * Correct benchmark compilation failure Signed-off-by: Brian L. Troutwine <[email protected]> * More build fixes Signed-off-by: Brian L. Troutwine <[email protected]> * More build fixes, wasm Signed-off-by: Brian L. Troutwine <[email protected]> * another bench import fix Signed-off-by: Brian L. Troutwine <[email protected]> * chore(deps): bump hyper from 0.14.6 to 0.14.7 (#7232) Bumps [hyper](https://github.com/hyperium/hyper) from 0.14.6 to 0.14.7. - [Release notes](https://github.com/hyperium/hyper/releases) - [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md) - [Commits](https://github.com/hyperium/hyper/compare/v0.14.6...v0.14.7) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump regex from 1.4.5 to 1.4.6 (#7233) Bumps [regex](https://github.com/rust-lang/regex) from 1.4.5 to 1.4.6. - [Release notes](https://github.com/rust-lang/regex/releases) - [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md) - [Commits](https://github.com/rust-lang/regex/compare/1.4.5...1.4.6) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(external docs): Correct highlight dates for 0.13.0 Signed-off-by: Jesse Szwedko <[email protected]> * chore(deps): bump async-nats from 0.9.12 to 0.9.14 (#7234) Bumps [async-nats](https://github.com/nats-io/nats.rs) from 0.9.12 to 0.9.14. - [Release notes](https://github.com/nats-io/nats.rs/releases) - [Changelog](https://github.com/nats-io/nats.rs/blob/master/CHANGELOG.md) - [Commits](https://github.com/nats-io/nats.rs/compare/0.9.12...0.9.14) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Run cargo-hack on self-hosted machines (#7227) * Run cargo-hack on self-hosted machines We found in the previous run of cargo-hack that it was OOM'ed. While we were expecting a build failure -- see #7222 -- we did not expect _that_ failure. Run on self-hosted hardware to get more memory for the build. Signed-off-by: Brian L. Troutwine <[email protected]> * Update .github/workflows/cargo_hack.yml Signed-off-by: Brian L. Troutwine <[email protected]> Co-authored-by: Jesse Szwedko <[email protected]> * don't consume all memory Signed-off-by: Brian L. Troutwine <[email protected]> * don't clean-per-run Signed-off-by: Brian L. Troutwine <[email protected]> * remove PR run directive Signed-off-by: Brian L. Troutwine <[email protected]> Co-authored-by: Jesse Szwedko <[email protected]> * chore(buffers): Upgrade DiskBuffer to futures 0.3 (#7165) * Update writer Signed-off-by: ktf <[email protected]> * Update reader Signed-off-by: ktf <[email protected]> * Update tests Signed-off-by: ktf <[email protected]> * Cleanup Signed-off-by: ktf <[email protected]> * Update benches Signed-off-by: ktf <[email protected]> * Fix benches Signed-off-by: ktf <[email protected]> * Use start_send Signed-off-by: ktf <[email protected]> * Bump Signed-off-by: ktf <[email protected]> * Fmt Signed-off-by: ktf <[email protected]> * Add comment Signed-off-by: ktf <[email protected]> * chore(splunk_hec source): Update to `futures` 0.3 (#7225) * Update source Signed-off-by: ktf <[email protected]> * Use forward Signed-off-by: ktf <[email protected]> * enhancement(graphql api): Expose `events_in` & `events_out` metrics (#6888) * Add in/out to API Signed-off-by: ktf <[email protected]> * Update client Signed-off-by: ktf <[email protected]> * Update tests Signed-off-by: ktf <[email protected]> * Add new line Signed-off-by: ktf <[email protected]> * Apply suggestions Co-authored-by: Lee Benson <[email protected]> Signed-off-by: ktf <[email protected]> * Update schema Signed-off-by: ktf <[email protected]> Co-authored-by: Lee Benson <[email protected]> * chore(deps): bump rust_decimal from 1.11.1 to 1.12.2 (#7243) Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.11.1 to 1.12.2. - [Release notes](https://github.com/paupino/rust-decimal/releases) - [Commits](https://github.com/paupino/rust-decimal/compare/1.11.1...1.12.2) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump tokio-postgres from 0.7.1 to 0.7.2 (#7244) Bumps [tokio-postgres](https://github.com/sfackler/rust-postgres) from 0.7.1 to 0.7.2. - [Release notes](https://github.com/sfackler/rust-postgres/releases) - [Commits](https://github.com/sfackler/rust-postgres/compare/tokio-postgres-v0.7.1...tokio-postgres-v0.7.2) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(ci): bump actions/github-script from v4.0.1 to v4.0.2 (#7245) Bumps [actions/github-script](https://github.com/actions/github-script) from v4.0.1 to v4.0.2. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v4.0.1...a3e7071a34d7e1f219a8a4de9a5e0a34d1ee1293) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump libc from 0.2.93 to 0.2.94 (#7254) Bumps [libc](https://github.com/rust-lang/libc) from 0.2.93 to 0.2.94. - [Release notes](https://github.com/rust-lang/libc/releases) - [Commits](https://github.com/rust-lang/libc/compare/0.2.93...0.2.94) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(external docs): Document VRL |= operator (#7252) Closes #6900 Signed-off-by: Jesse Szwedko <[email protected]> * enhancement(kubernetes_logs source): Allow configuration of max_line_bytes (#7226) * enhancement(kubernetes_logs source): Allow configuration of max_line_bytes Fixes: #6966 This is specifically to allow a user to workaround the too-low hardcoded value that was being used. We have https://github.com/timberio/vector/issues/6967 to address that. I lost the logs that the user provided that ran over that limit but I believe it was due to having a very large number of labels in the metadata for each message which is not included in the normal 16 KB docker limit. Signed-off-by: Jesse Szwedko <[email protected]> * chore(deps): bump tower from 0.4.6 to 0.4.7 (#7260) Bumps [tower](https://github.com/tower-rs/tower) from 0.4.6 to 0.4.7. - [Release notes](https://github.com/tower-rs/tower/releases) - [Commits](https://github.com/tower-rs/tower/compare/tower-0.4.6...tower-0.4.7) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump syn from 1.0.70 to 1.0.71 (#7261) Bumps [syn](https://github.com/dtolnay/syn) from 1.0.70 to 1.0.71. - [Release notes](https://github.com/dtolnay/syn/releases) - [Commits](https://github.com/dtolnay/syn/compare/1.0.70...1.0.71) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(kafka source): Fix runaway memory usage of kafka source (#7266) * fix(kafka source): Fix runaway memory usage of kafka source Fixes: #7258 Previously, we were cloning the `Pipeline` for each message which, due to the behavior of futures::sync::mpsc::channel resulted in the effective channel buffer growing by one for each message rather than any back pressure. Signed-off-by: Jesse Szwedko <[email protected]> * fix(external docs): typo (#7268) * chore(deps): bump openssl from 0.10.33 to 0.10.34 (#7270) Bumps [openssl](https://github.com/sfackler/rust-openssl) from 0.10.33 to 0.10.34. - [Release notes](https://github.com/sfackler/rust-openssl/releases) - [Commits](https://github.com/sfackler/rust-openssl/compare/openssl-v0.10.33...openssl-v0.10.34) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump pulsar from 2.0.0 to 2.0.1 (#7271) Bumps [pulsar](https://github.com/wyyerd/pulsar-rs) from 2.0.0 to 2.0.1. - [Release notes](https://github.com/wyyerd/pulsar-rs/releases) - [Commits](https://github.com/wyyerd/pulsar-rs/compare/v2.0.0...v2.0.1) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(releasing): Update install script to 0.13.1 (#7275) Signed-off-by: Jesse Szwedko <[email protected]> * feat(new source): Initial `exec` source (#6876) * feat(exec source): new `exec` source (#992) (#7) Added exec source with streaming and scheduled capability. Generates log events from stdout and stderr of a command execution. Signed-off-by: Stuart Broad <[email protected]> * enhancement: created new lookup library (#7087) * Created new lookup library Signed-off-by: Stephen Wakely <[email protected]> * Rudimentary form of lookup working with Vrl. Signed-off-by: Stephen Wakely <[email protected]> * Lookup can parse paths itself. Signed-off-by: Stephen Wakely <[email protected]> * Restored VRL coalesce Signed-off-by: Stephen Wakely <[email protected]> * Updated property tests. Signed-off-by: Stephen Wakely <[email protected]> * Fixed issue with inserting a path that changes the type. Plus a number of other tidy ups. Signed-off-by: Stephen Wakely <[email protected]> * Clippy and formatting. Signed-off-by: Stephen Wakely <[email protected]> * Removed unneeded file Signed-off-by: Stephen Wakely <[email protected]> * Removed commented out Path. Signed-off-by: Stephen Wakely <[email protected]> * Formatting Signed-off-by: Stephen Wakely <[email protected]> * Vrl needs to include the . when displaying paths. Signed-off-by: Stephen Wakely <[email protected]> * Feedback from blt. Signed-off-by: Stephen Wakely <[email protected]> * Fixes following feedback from Jean. Signed-off-by: Stephen Wakely <[email protected]> * Fixes following feedback from Bruce. Signed-off-by: Stephen Wakely <[email protected]> * Clippy Signed-off-by: Stephen Wakely <[email protected]> * More clippy fixes Signed-off-by: Stephen Wakely <[email protected]> * Renamed clone_lookup to be to_lookup Signed-off-by: Stephen Wakely <[email protected]> * Enable negative indexing Signed-off-by: Stephen Wakely <[email protected]> * Added quickcheck tests for lookups Signed-off-by: Stephen Wakely <[email protected]> * Fixed setting zero index Signed-off-by: Stephen Wakely <[email protected]> * Add limit to alternative component path Signed-off-by: Stephen Wakely <[email protected]> * Removed irrelevant tests Signed-off-by: Stephen Wakely <[email protected]> * Clippy Signed-off-by: Stephen Wakely <[email protected]> * Move rerun-if-changed instruction Signed-off-by: Stephen Wakely <[email protected]> * Tidied some comments. Signed-off-by: Stephen Wakely <[email protected]> * Don't serialize metadata Signed-off-by: Stephen Wakely <[email protected]> * Update types when array index is changed. Signed-off-by: Stephen Wakely <[email protected]> * Removed insert_null. Signed-off-by: Stephen Wakely <[email protected]> * Clippy Signed-off-by: Stephen Wakely <[email protected]> * Minor comment fixes Signed-off-by: Stephen Wakely <[email protected]> * Remove serialize test as it is incorrect Signed-off-by: Stephen Wakely <[email protected]> * Feedback from Bruce. Signed-off-by: Stephen Wakely <[email protected]> * Don't wipe out metadata on vrl remove Signed-off-by: Stephen Wakely <[email protected]> * chore(deps): bump rust_decimal from 1.12.2 to 1.12.3 (#7280) Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.12.2 to 1.12.3. - [Release notes](https://github.com/paupino/rust-decimal/releases) - [Commits](https://github.com/paupino/rust-decimal/compare/1.12.2...1.12.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump libz-sys from 1.1.2 to 1.1.3 (#7278) Bumps [libz-sys](https://github.com/rust-lang/libz-sys) from 1.1.2 to 1.1.3. - [Release notes](https://github.com/rust-lang/libz-sys/releases) - [Commits](https://github.com/rust-lang/libz-sys/compare/1.1.2...1.1.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump async-nats from 0.9.14 to 0.9.16 (#7279) Bumps [async-nats](https://github.com/nats-io/nats.rs) from 0.9.14 to 0.9.16. - [Release notes](https://github.com/nats-io/nats.rs/releases) - [Changelog](https://github.com/nats-io/nats.rs/blob/master/CHANGELOG.md) - [Commits](https://github.com/nats-io/nats.rs/compare/0.9.14...0.9.16) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Update `top` with `events_in_total` and `events_out_total` metrics (#7286) * Use in/out Signed-off-by: ktf <[email protected]> * Remove processed_events Signed-off-by: ktf <[email protected]> * Remove empty line Signed-off-by: ktf <[email protected]> * Events header Signed-off-by: ktf <[email protected]> * Update column width Signed-off-by: ktf <[email protected]> * Pluralize Signed-off-by: ktf <[email protected]> * Fixed doc tests from lookup (#7288) Signed-off-by: Stephen Wakely <[email protected]> * chore(ci): Fix action-discord version for test workflow (#7285) Accidentally replaced with 0.3.2 when upgrading cue due to an eager find/replace. Signed-off-by: Jesse Szwedko <[email protected]> * fix(graphql api): `--no-default-features --features api` build failure (#7284) * host metrics Signed-off-by: Lee Benson <[email protected]> * -pub Signed-off-by: Lee Benson <[email protected]> * fix(external docs): Document scrape_interval_secs for `internal_metrics` source (#7289) Signed-off-by: Jesse Szwedko <[email protected]> * chore(tests): Simplify `cargo test` usage (#7287) We currently use `--all-targets` to include benches (`--bench`), but `--all-targets` does not include `--doc` apparently (https://github.com/rust-lang/cargo/issues/6669). Instead of using `--all-targets` we just mark benches with `test = true` so that they will be tested along with the default targets (https://doc.rust-lang.org/cargo/commands/cargo-test.html#target-selection) Also: * Avoid setting `SCOPE` by default in the `Makefile` as this causes examples to be skipped for some reason. * Use `make test` for Windows to take advantage of existing make targets. I tried `make test-components` there too, but it failed to build krb5. Signed-off-by: Jesse Szwedko <[email protected]> * chore(deps): bump tracing-core from 0.1.17 to 0.1.18 (#7294) Bumps [tracing-core](https://github.com/tokio-rs/tracing) from 0.1.17 to 0.1.18. - [Release notes](https://github.com/tokio-rs/tracing/releases) - [Commits](https://github.com/tokio-rs/tracing/compare/tracing-core-0.1.17...tracing-core-0.1.18) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump tui from 0.14.0 to 0.15.0 (#7295) Bumps [tui](https://github.com/fdehau/tui-rs) from 0.14.0 to 0.15.0. - [Release notes](https://github.com/fdehau/tui-rs/releases) - [Changelog](https://github.com/fdehau/tui-rs/blob/master/CHANGELOG.md) - [Commits](https://github.com/fdehau/tui-rs/compare/v0.14.0...v0.15.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump built from 0.4.4 to 0.5.0 (#7297) Bumps [built](https://github.com/lukaslueg/built) from 0.4.4 to 0.5.0. - [Release notes](https://github.com/lukaslueg/built/releases) - [Commits](https://github.com/lukaslueg/built/compare/0.4.4...0.5.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * enhancement(kafka source, kafka sink): Emit internal metrics provided by rdkafka (#5874) * Emit internal metrics for rdkafka Signed-off-by: Pablo Sichert <[email protected]> * Rename KafkaStatistics -> KafkaStatisticsReceived Signed-off-by: Pablo Sichert <[email protected]> * Emit KafkaStatisticsReceived as counter with JSON annotation Signed-off-by: Pablo Sichert <[email protected]> * Fix missing fields in KafkaSinkConfig Signed-off-by: Pablo Sichert <[email protected]> * Flatten into statistics_interval_ms Signed-off-by: Pablo Sichert <[email protected]> * Document statistics_interval_ms option for kafka source/sink Signed-off-by: Pablo Sichert <[email protected]> * Create producer in kafka sink with KafkaStatisticsContext Signed-off-by: Pablo Sichert <[email protected]> * Add component docs for kafka_statistics_received_total telemetry Signed-off-by: Pablo Sichert <[email protected]> * Emit selected metrics from kafka statistics Signed-off-by: Pablo Sichert <[email protected]> * Remove trivial constructor and take statistics by reference Signed-off-by: Pablo Sichert <[email protected]> * Remove statistics_interval_ms config options and set kafka statistics interval to 1s by default Signed-off-by: Pablo Sichert <[email protected]> * Change kafka metrics from gauge to counter where appropriate Signed-off-by: Pablo Sichert <[email protected]> * Fix unused import Signed-off-by: Pablo Sichert <[email protected]> * Rerun checks for benchmark results Signed-off-by: Pablo Sichert <[email protected]> * Fix punctuation Co-authored-by: Jesse Szwedko <[email protected]> Signed-off-by: Pablo Sichert <[email protected]> * Calculate counter delta locally instead of in global recorder/layer Signed-off-by: Pablo Sichert <[email protected]> * Emit delta before dropping mutex guard / document behavior Signed-off-by: Pablo Sichert <[email protected]> * Warn and emit 0 instead of panicking on non-monotonic values Signed-off-by: Pablo Sichert <[email protected]> * Fix non-capitalized log message Signed-off-by: Pablo Sichert <[email protected]> * Fix typo Signed-off-by: Pablo Sichert <[email protected]> * Use atomic instead of mutex in update_counter! Signed-off-by: Pablo Sichert <[email protected]> * Use const initializer instead of lazy_static! Signed-off-by: Pablo Sichert <[email protected]> Co-authored-by: Jesse Szwedko <[email protected]> * chore(deps): bump rust_decimal from 1.12.3 to 1.12.4 (#7301) Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.12.3 to 1.12.4. - [Release notes](https://github.com/paupino/rust-decimal/releases) - [Commits](https://github.com/paupino/rust-decimal/commits) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump bstr from 0.2.15 to 0.2.16 (#7302) Bumps [bstr](https://github.com/BurntSushi/bstr) from 0.2.15 to 0.2.16. - [Release notes](https://github.com/BurntSushi/bstr/releases) - [Commits](https://github.com/BurntSushi/bstr/compare/0.2.15...0.2.16) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(external docs): Typo (#7304) Signed-off-by: Jesse Szwedko <[email protected]> * chore: Make CI tests run a little more quiet (#7269) * Separate bench and test, uh, tests This commit re-introduces the `--quiet` flag -- this will still output for failing tests -- and splits up the notion of the benchmark sanity tests versus the correctness, correctness tests. Signed-off-by: Brian L. Troutwine <[email protected]> * Split benches debug tests into their own workflow Signed-off-by: Brian L. Troutwine <[email protected]> * correct 'bench unit' name Signed-off-by: Brian L. Troutwine <[email protected]> * feedback updates Signed-off-by: Brian L. Troutwine <[email protected]> * more elaborate caching Signed-off-by: Brian L. Troutwine <[email protected]> * Revert "more elaborate caching" This reverts commit 9ed4fdd0b4e7ae32d59531262ba63a474668efdd. Signed-off-by: Brian L. Troutwine <[email protected]> * warning fix Signed-off-by: Brian L. Troutwine <[email protected]> * Remove unused target Signed-off-by: Brian L. Troutwine <[email protected]> * enhancement(remap): Wdd `whitespace` option to `parse_key_value` function (#7292) * Empty value Signed-off-by: ktf <[email protected]> * Whitespace option Signed-off-by: ktf <[email protected]> * Whitespace before Signed-off-by: ktf <[email protected]> * Docs Signed-off-by: ktf <[email protected]> * Replace bool with Whitespace Signed-off-by: ktf <[email protected]> * chore(deps): bump tracing-subscriber from 0.2.17 to 0.2.18 (#7298) Bumps [tracing-subscriber](https://github.com/tokio-rs/tracing) from 0.2.17 to 0.2.18. - [Release notes](https://github.com/tokio-rs/tracing/releases) - [Commits](https://github.com/tokio-rs/tracing/compare/tracing-subscriber-0.2.17...tracing-subscriber-0.2.18) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump snap from 1.0.4 to 1.0.5 (#7300) Bumps [snap](https://github.com/BurntSushi/rust-snappy) from 1.0.4 to 1.0.5. - [Release notes](https://github.com/BurntSushi/rust-snappy/releases) - [Commits](https://github.com/BurntSushi/rust-snappy/compare/1.0.4...1.0.5) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: Run cargo-flake on the project (#7307) * Run cargo-flake on the project I have struggled in #7240 to get reliable test results. This has inspired both PR #7269 and #7306 to reduce both output and turn-around time for tests. In this PR I introduce a tool that is meant to repeatedly run our tests and identify any that are flakey. Signed-off-by: Brian L. Troutwine <[email protected]> * adjustments Signed-off-by: Brian L. Troutwine <[email protected]> * `cargo flake` not `cargo-flake` Signed-off-by: Brian L. Troutwine <[email protected]> * remove temporary PR run Signed-off-by: Brian L. Troutwine <[email protected]> * chore: Introduce more aggressive caching (#7306) * Introduce more aggressive caching This commit introduces caching of `target/debug` into our jobs as well as the use of `restore-keys` to hit cache even when `Cargo.lock` has changed. This commit is further work from #7269 Signed-off-by: Brian L. Troutwine <[email protected]> * `target/debug` not `~/target/debug` Signed-off-by: Brian L. Troutwine <[email protected]> * Back off caching `target/debug` Signed-off-by: Brian L. Troutwine <[email protected]> * faster macOS dev builds (#7317) Signed-off-by: Lee Benson <[email protected]> * chore: Extract `Event` into vector-core (#7240) * chore: Extract `Event` into vector-core This PR extracts `Event`, `metrics` and `mapping` module into vector-core. I had hoped to do these separately but they proved to be tied together in a way I couldn't separate. I've intentionally not made many structural changes to this code. As much as possible it's a simple move operation. Closes #7151 #7148 Signed-off-by: Brian L. Troutwine <[email protected]> * address clippy warnings Signed-off-by: Brian L. Troutwine <[email protected]> * address failing vector-core tests Signed-off-by: Brian L. Troutwine <[email protected]> * clippy fixes Signed-off-by: Brian L. Troutwine <[email protected]> * correct log_schema Signed-off-by: Brian L. Troutwine <[email protected]> * correct panic! without formatting string Signed-off-by: Brian L. Troutwine <[email protected]> * address build issue after rebase Signed-off-by: Brian L. Troutwine <[email protected]> * chore(deps): bump rust_decimal from 1.12.4 to 1.13.0 (#7320) Bumps [rust_decimal](https://github.com/paupino/rust-decimal) from 1.12.4 to 1.13.0. - [Release notes](https://github.com/paupino/rust-decimal/releases) - [Commits](https://github.com/paupino/rust-decimal/commits/1.13.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump ordered-float from 2.1.1 to 2.2.0 (#7321) Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 2.1.1 to 2.2.0. - [Release notes](https://github.com/reem/rust-ordered-float/releases) - [Commits](https://github.com/reem/rust-ordered-float/compare/v2.1.1...v2.2.0) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Move `benches/lookup` into vector-core (#7331) While the fixture data for this benchmark was moved into `vector-core` as a part of #7240 the benchmark itself was not. This broke the build. Resolves #7329 Signed-off-by: Brian L. Troutwine <[email protected]> * fix(ci): Flag benchmark runs that fail to actually execute (#7330) I had included `continue-on-error` originally since we didn't want to flag regressions yet, but this also suppresses other errors (like the benches couldn't run). The `|| true` in `scripts/check-criterion-output.sh` should stop it from flagging regressions as failed CI checks still. Signed-off-by: Jesse Szwedko <[email protected]> * fix(splunk_hec source): Default `host` to remote address (#7293) * Use remote address Signed-off-by: ktf <[email protected]> * Add test Signed-off-by: ktf <[email protected]> * Add issue link Signed-off-by: ktf <[email protected]> * fix(cli): Enable use of delimiters (#7316) Signed-off-by: ktf <[email protected]> * enhancement(datadog sinks): Generic site support (#7281) Add a `site` option for Datadog sinks, it aims to ultimately replace the `region` option. Signed-off-by: prognant <[email protected]> * chore: Reference RFC template in RFC readme (#7344) * chore: Reference RFC template in RFC readme Signed-off-by: Pablo Sichert <[email protected]> * Use relative link Co-authored-by: Jesse Szwedko <[email protected]> Signed-off-by: Pablo Sichert <[email protected]> Co-authored-by: Jesse Szwedko <[email protected]> * Tidy up iterator logic (#7338) This commit tidies up the iterator logic suggested by clippy to be more direct. Changed suggested by @bruceg in #7240. Partially resolves #7327 Signed-off-by: Brian L. Troutwine <[email protected]> * Make vrl optional in vector-core (#7335) While VRL is essential to vector it is not a part of core. Allowing core to be built without linking to VRL was originally a feature of #7240 but was removed as it increased the testing surface overmuch. By default, now, vector-core will not link with VRL. Partially addresses #7327 Signed-off-by: Brian L. Troutwine <[email protected]> * chore: Use 'cargo hack' instead of check-component-features (#7311) * Use 'cargo hack' instead of check-component-features This commit replaces our check-component-features with 'cargo hack' to the same purpose. We also remove the nightly check as it appears that it is recently fixed. Signed-off-by: Brian L. Troutwine <[email protected]> * Correct makefile typo Signed-off-by: Brian L. Troutwine <[email protected]> * Address build failure Signed-off-by: Brian L. Troutwine <[email protected]> * kafka build issue resolved, all-metrics feature flag Signed-off-by: Brian L. Troutwine <[email protected]> * re-introduce 'check-component-features' target Signed-off-by: Brian L. Troutwine <[email protected]> * Exclude 'utils' features from hack check Turns out we have features that are only for utility purposes, perfectly valid but if we check them alone we'll end up with export warnings. Signed-off-by: Brian L. Troutwine <[email protected]> * chore(deps): bump syn from 1.0.71 to 1.0.72 (#7334) Bumps [syn](https://github.com/dtolnay/syn) from 1.0.71 to 1.0.72. - [Release notes](https://github.com/dtolnay/syn/releases) - [Commits](https://github.com/dtolnay/syn/compare/1.0.71...1.0.72) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump regex from 1.4.6 to 1.5.3 (#7319) Bumps [regex](https://github.com/rust-lang/regex) from 1.4.6 to 1.5.3. - [Release notes](https://github.com/rust-lang/regex/releases) - [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md) - [Commits](https://github.com/rust-lang/regex/compare/1.4.6...1.5.3) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jesse Szwedko <[email protected]> * chore(deps): bump tracing from 0.1.25 to 0.1.26 (#7296) Bumps [tracing](https://github.com/tokio-rs/tracing) from 0.1.25 to 0.1.26. - [Release notes](https://github.com/tokio-rs/tracing/releases) - [Commits](https://github.com/tokio-rs/tracing/compare/tracing-0.1.25...tracing-0.1.26) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jesse Szwedko <[email protected]> * fix(ci): Run tests for all targets on all platforms (#7323) Previous benches job was actually running unit tests too. This reverts to running all targets in one job. Signed-off-by: Jesse Szwedko <[email protected]> * Derive `Default` for `MakeLogEvent` (#7339) As noted by @bruceg in #7240 we could avoid the explicit call to `LogEvent::default` by deriving `Default` for `MakeLogEvent`. This is now done and I've tidied up the relevant call-site. Partial resolves #7327 Signed-off-by: Brian L. Troutwine <[email protected]> * chore(sinks): Drop unnecessary `Request` parameter from types (#7251) Signed-off-by: Bruce Guenter <[email protected]> * chore: Drop `futures01` (#7343) Signed-off-by: ktf <[email protected]> * chore(deps): bump sha-1 from 0.9.4 to 0.9.5 (#7357) Bumps [sha-1](https://github.com/RustCrypto/hashes) from 0.9.4 to 0.9.5. - [Release notes](https://github.com/RustCrypto/hashes/releases) - [Commits](https://github.com/RustCrypto/hashes/compare/sha-1-v0.9.4...sha-1-v0.9.5) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix the postinst conditionnal for adding vector to the systemd-journal group (#7359) Signed-off-by: Johann Queuniet <[email protected]> * chore(deps): bump metrics from 0.14.2 to 0.15.1 (#7313) * chore(deps): bump metrics from 0.14.2 to 0.15.1 Bumps [metrics](https://github.com/metrics-rs/metrics) from 0.14.2 to 0.15.1. - [Release notes](https://github.com/metrics-rs/metrics/releases) - [Changelog](https://github.com/metrics-rs/metrics/blob/main/release.toml) - [Commits](https://github.com/metrics-rs/metrics/compare/metrics-v0.14.2...metrics-v0.15.1) Signed-off-by: dependabot[bot] <[email protected]> * Upgrade other metrics dependencies and fix conflicts Signed-off-by: Jesse Szwedko <[email protected]> * Update metrics versions in vector core too Signed-off-by: Jesse Szwedko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jesse Szwedko <[email protected]> * chore(deps): Update grok and dependencies (#7176) * chore(deps): Update grok and dependencies This updates grok to 1.1 and its dependencies. I couldn't upgrade to 1.2 due to a conflict with lucet on `userfaultfd`: ``` error: failed to select a version for `userfaultfd`. ... required by package `lucet-runtime-internals v0.7.0-dev (https://github.com/bytecodealliance/lucet.git?rev=b1863dacc8c92c11e5434fc8815d9b9a26cfe3db#b1863dac)` ... which is depended on by `lucet-runtime v0.7.0-dev (https://github.com/bytecodealliance/lucet.git?rev=b1863dacc8c92c11e5434fc8815d9b9a26cfe3db#b1863dac)` ... which is depended on by `lucet-wasi v0.7.0-dev (https://github.com/bytecodealliance/lucet.git?rev=b1863dacc8c92c11e5434fc8815d9b9a26cfe3db#b1863dac)` ... which is depended on by `vector v0.13.0 (/Users/jesse.szwedko/workspace/vector)` versions that meet the requirements `^0.2.0` are: 0.2.1, 0.2.0 ``` This required adding additional build dependencies to the cross and Windows builds: `llvm` and `clang`. Signed-off-by: Jesse Szwedko <[email protected]> * chore: Drop redundant copy of built info (#7365) Signed-off-by: Bruce Guenter <[email protected]> * chore: RFC for emitting multiple log events from remap (#7038) * chore: RFC for emitting multiple log events from remap Signed-off-by: Jesse Szwedko <[email protected]> * chore: Enable stricter clippy lints in vector-core (#7341) * Enable stricter clippy lints in vector-core In the early drafts of #7240 we had strict clippy lints on. While these were desirable they ended up bloating the PR. We now re-introduce those strict lints. Partially resolves #7327 Signed-off-by: Brian L. Troutwine <[email protected]> * A small host of clippy::pedantic suggestions Signed-off-by: Brian L. Troutwine <[email protected]> * more clippy::pedentic suggestions Signed-off-by: Brian L. Troutwine <[email protected]> * more clippy::pedantic fixes Signed-off-by: Brian L. Troutwine <[email protected]> * fix reference issue Signed-off-by: Brian L. Troutwine <[email protected]> * Back off mandatory `must_use` Signed-off-by: Brian L. Troutwine <[email protected]> * Allow `clippy::match_wildcard_for_single_variants` Signed-off-by: Brian L. Troutwine <[email protected]> * Remove site grant for `redundant_closure_for_method_calls` Signed-off-by: Brian L. Troutwine <[email protected]> * Slightly adjust Number parse Signed-off-by: Brian L. Troutwine <[email protected]> * Site grant for `needless_pass_by_value` Signed-off-by: Brian L. Troutwine <[email protected]> * Remove unused import Signed-off-by: Brian L. Troutwine <[email protected]> * Remove allow for `clippy::doc_markdown` Signed-off-by: Brian L. Troutwine <[email protected]> * Remove global grant for allow_too_many_lines Signed-off-by: Brian L. Troutwine <[email protected]> * Remove global grant for enum_glob_use Signed-off-by: Brian L. Troutwine <[email protected]> * Disallow global grant for cast_possible_truncation Signed-off-by: Brian L. Troutwine <[email protected]> * Remove global grant for `missing_error_docs` Signed-off-by: Brian L. Troutwine <[email protected]> * Remove global grant for `missing_panic_docs` Signed-off-by: Brian L. Troutwine <[email protected]> * clippy dings in lua-town Signed-off-by: Brian L. Troutwine <[email protected]> * add documentation Signed-off-by: Brian L. Troutwine <[email protected]> * clippy dings in test code Signed-off-by: Brian L. Troutwine <[email protected]> * reverse non_ascii_literal Signed-off-by: Brian L. Troutwine <[email protected]> * documentation, 'cargo hack' detected test failures Signed-off-by: Brian L. Troutwine <[email protected]> * Document clippy lint Signed-off-by: Brian L. Troutwine <[email protected]> * more documentation Signed-off-by: Brian L. Troutwine <[email protected]> * Update lib/vector-core/src/config/log_schema.rs Signed-off-by: Brian L. Troutwine <[email protected]> Co-authored-by: Bruce Guenter <[email protected]> * Don't use approximate constants Signed-off-by: Brian L. Troutwine <[email protected]> * Value -> Self Signed-off-by: Brian L. Troutwine <[email protected]> * address clippy ding Signed-off-by: Brian L. Troutwine <[email protected]> Co-authored-by: Bruce Guenter <[email protected]> * chore(metrics): Remove emits & alias `processed_events_total` metric (#7345) * Add alias Signed-off-by: ktf <[email protected]> * Remove emit Signed-off-by: ktf <[email protected]> * Update docs Signed-off-by: ktf <[email protected]> * Format cue Signed-off-by: ktf <[email protected]> * chore(deps): bump regex from 1.5.3 to 1.5.4 (#7374) Bumps [regex](https://github.com/rust-lang/regex) from 1.5.3 to 1.5.4. - [Release notes](https://github.com/rust-lang/regex/releases) - [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md) - [Commits](https://github.com/rust-lang/regex/compare/1.5.3...1.5.4) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(ci): Install llvm on Windows nightly and release bulids too (#7377) Installed in test.yml as part of https://github.com/timberio/vector/pull/7176 Signed-off-by: Jesse Szwedko <[email protected]> * chore(metrics): Fix alias for processed_events_total (#7378) Signed-off-by: Jesse Szwedko <[email protected]> * fix(observability)!: Correct all `*_nanoseconds` metrics to `_seconds` (#7373) * fix(observability): Correct all `*_nanoseconds` metrics to `_seconds` All the referenced metrics labelled as "nanoseconds" are passed a `Duration` as the value, which the `metrics` crate automatically converts into (fractional) seconds internally. As such, all these names are incorrect and need correction. * Fix some more named `*_ns` but were really seconds too Signed-off-by: Bruce Guenter <[email protected]> * chore(data model): Add metadata support for end-to-end acknowledgements (#7141) * Drop unreferenced struct from RFC * chore(data model): Add metadata for event finalization * Add the event finalizers to the top-level metadata * Move cloning into registration functions This avoids extraneous use of `Arc::clone` when the item is already in the table. * Add a "counting" DashMap to provide serialization semantics * Back out support for serialzing finalization status In discussion with Brian and Luke, we decided to punt for now on supporting end-to-end acknowledgements after an event has been buffered to disk. This greatly simplifies the initial requirements, but we will come back to this later. * Drop remaining `impl From<LogEvent>` that silently drop metadata * Quiet a clippy warning In `enum BufferInputCloner`, the `Disk` variant is quite a bit larger than `Memory` with the increased size of `Event` (233 vs 25 bytes). Clippy recommends boxing this, but that is beyond the scope of this PR. * Add `metadata_mut` accessors to event types * Rename `EventStatus::NoOp` to `Recorded` Signed-off-by: Bruce Guenter <[email protected]> * enhancement(http sink): Add support for end-to-end acknowledgements (#7265) * Create `EventRef` wrapper for references to event inner types * Rework MetricsBuffer to use the Metric type internally * Merge some common bits in http sink tests * Add tests for retries and failures to http sink * Add Batch wrapper to store metadata * Use metadata batch for batch sinks * Update event metadata in `ServiceSink` * Add metadata output to `trait HttpSink::encode_event` * Handle metadata in `BatchSink` wrapper * Modify http sink to pass along metadata * Add batch status asserts to all HTTP tests Signed-off-by: Bruce Guenter <[email protected]> * chore(ci): Fix deb verify steps (#7385) They verify installation in a container without systemd running so the `systemctl daemon-reload` step fails. Signed-off-by: Jesse Szwedko <[email protected]> * feat(remap transform): Emitting multiple events (#7267) Add support to the `remap` transform to emit multiple events by setting `.` to an array. Signed-off-by: Jesse Szwedko <[email protected]> * enhancement(remap): Add parse_klog function (#7349) Closes: #6793 This is very similar to parse_glog but I think a separate function is still prudent in-case the continue to diverge. The biggeste differences are just spacing and the fact that timestamps don't have a year in the…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, remap would overwrite the type def of
.whenever a newfield was assigned. That is:
Would result in the compiler only having type info for
.bar.This change causes it to merge the typedefs whenever a list of path
segments appears.
Big thangs to @FungusHumungus for talking this one through with me.
This addresses the example that appears here: #6792 (comment) . It may also resolve some of the other comments on that issue, but I'm not 100% sure until I get responses from the commenters since I couldn't reproduce the other cases.
Signed-off-by: Jesse Szwedko [email protected]