Skip to content

Commit

Permalink
client: shard OffsetFetchRequest, redux group sharded requests
Browse files Browse the repository at this point in the history
OffsetFetchRequest must now be sharded. We do this in a forward
compatible way such that anything defining the request for versions v0
thru v7 are forward compatible with v8 with no problems.

We avoid being backwards compatible: if a person defines multiple groups
in the request, we do not try to split and reissue if we detect the
response is pre v8.

To be forward compatible, if the user defines the top level group /
topics, we also add that top level to the Groups slice itself, and on
response, if the response contains one element in the Groups slice, we
translate that back to the top level. From a usage perspective, a user
will notice nothing (except the groups slice will have an element that
they are not looking at).

This change changes retries for every other sharded request. Previously,
we never used cached coordinators for sharded requests, so if any shard
failed with a retriable error, we would just return that retriable error
to the client. We wouldn't expect retriable errors because by not using
a cached coordinator, we would load fresh data immediately before
issuing our request. This is the same logic we now use in topic /
partitions based requests. Well, now we are flipping this for group
requests.

Now, we do use cached coordinators, but if a shard fails with a
retriable request, we delete the cached coordinator and re-issue the
shard. This is the same behavior that a "simple" non-sharded request
goes through.

When sharding a request, we also now allow individual FindCoordinator
requests to fail. Previously, if one request failed, we would just fail
the entire sharding process, and we would not retry. Now, individual
group failure lookups will create failed shards for those failed groups,
but successful lookups will proceed to be issued. We try to group errors
as much as we can, but we cannot group every error, so the functions get
a bit ugly.
  • Loading branch information
twmb committed Jul 9, 2021
1 parent e5e37fc commit 3a3cc06
Show file tree
Hide file tree
Showing 2 changed files with 420 additions and 100 deletions.
Loading

0 comments on commit 3a3cc06

Please sign in to comment.