Decoding codecs cause Vector to stop on an error

[StephenWakely]

Given the following config:

[sources.in]
type = "stdin"
decoding.codec = "json"

[sinks.out]
type = "console"
inputs = ["in" ]
target = "stdout"
encoding.codec = "json"

If I run Vector and enter invalid json, it causes Vector to stop:

dodgyjson
2022-02-08T11:23:00.171560Z  WARN source{component_kind="source" component_id=in component_type=stdin component_name=in}: vector::internal_events::decoder: Failed deserializing frame. error=Error parsing JSON: Error("expected value", line: 1, column: 1) internal_log_rate_secs=10
2022-02-08T11:23:00.172573Z  INFO source{component_kind="source" component_id=in component_type=stdin component_name=in}: vector::sources::stdin: Finished sending.
2022-02-08T11:23:00.172773Z  INFO vector::shutdown: All sources have finished.
2022-02-08T11:23:00.172935Z  INFO vector: Vector has stopped.

Ideally, Vector should report the error and carry on accepting input.

It may be worth noting that with #11244 entering an invalid syslog message causes the same behaviour.

[pablosichert]

I was initially suspecting the logic in can_continue to be faulty, but it looks good. Couldn't see any relevant changes otherwise. Did a git bisect and it pointed me to d6ace26 / #9938:

vector $ bisect good
d6ace26a214f61f33b5063f45333e9765befd2f3 is the first bad commit
commit d6ace26a214f61f33b5063f45333e9765befd2f3
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sat Dec 4 00:20:03 2021 +0000

    chore(deps): bump tokio-util from 0.6.8 to 0.6.9 (#9938)
    
    Bumps [tokio-util](https://github.com/tokio-rs/tokio) from 0.6.8 to 0.6.9.
    - [Release notes](https://github.com/tokio-rs/tokio/releases)
    - [Commits](https://github.com/tokio-rs/tokio/compare/tokio-util-0.6.8...tokio-util-0.6.9)
    
    ---
    updated-dependencies:
    - dependency-name: tokio-util
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...
    
    Signed-off-by: dependabot[bot] <[email protected]>
    
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

 Cargo.lock               | 4 ++--
 src/expiring_hash_map.rs | 1 -
 2 files changed, 2 insertions(+), 3 deletions(-)

That is rather nasty... The diff suggests that the state machine in FramedRead changed regarding error handling. Looking into that.

[JeanMertz]

I'm re-opening this. Seems like reverting the tokio-util upgrade is a stop-gap solution until @pablosichert finds the actual cause of this problem.

[jszwedko]

I'd opened #11257 to track upgrading again. This particular issue is resolved, in my opinion, we are just blocked on upgrading.