Fix missing eof when writer cancelled

[Dreamsorcerer]

Fixes #5220.

I believe this is a better fix than #5238. That PR detects that we didn't finish sending a chunked response and then closes the connection. This PR ensures that we simply complete the chunked response by sending the EOF bytes, allowing the connection to remain open and be reused normally.

[codecov]

Codecov Report

Merging #7764 (2aa7d8b) into master (7a19191) will increase coverage by 0.02%.
Report is 6 commits behind head on master.
The diff coverage is 97.46%.

@@            Coverage Diff             @@
##           master    #7764      +/-   ##
==========================================
+ Coverage   97.36%   97.38%   +0.02%     
==========================================
  Files         106      106              
  Lines       31722    31863     +141     
  Branches     3633     3648      +15     
==========================================
+ Hits        30885    31029     +144     
+ Misses        633      632       -1     
+ Partials      204      202       -2     

Flag	Coverage Δ
CI-GHA	97.30% <97.46%> (+0.02%)	⬆️
OS-Linux	96.97% <97.46%> (+0.02%)	⬆️
OS-Windows	95.45% <97.43%> (+0.03%)	⬆️
OS-macOS	96.63% <97.46%> (+0.02%)	⬆️
Py-3.10.11	95.38% <97.43%> (+0.03%)	⬆️
Py-3.10.13	96.83% <97.46%> (+0.02%)	⬆️
Py-3.11.6	96.51% <97.46%> (+0.02%)	⬆️
Py-3.8.10	95.35% <97.43%> (+0.03%)	⬆️
Py-3.8.18	96.76% <97.46%> (+0.02%)	⬆️
Py-3.9.13	95.34% <97.43%> (+0.03%)	⬆️
Py-3.9.18	96.78% <97.46%> (+0.02%)	⬆️
Py-pypy7.3.11	96.21% <97.46%> (+0.04%)	⬆️
VM-macos	96.63% <97.46%> (+0.02%)	⬆️
VM-ubuntu	96.97% <97.46%> (+0.02%)	⬆️
VM-windows	95.45% <97.43%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
aiohttp/client.py	94.46% <100.00%> (+0.01%)	⬆️
aiohttp/client_reqrep.py	98.56% <100.00%> (+0.99%)	⬆️
tests/test_client_response.py	99.12% <100.00%> (+<0.01%)	⬆️
tests/test_proxy_functional.py	64.96% <100.00%> (+0.07%)	⬆️
tests/test_web_functional.py	98.04% <ø> (-0.01%)	⬇️
tests/test_client_functional.py	98.38% <95.00%> (-0.07%)	⬇️

... and 4 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[Dreamsorcerer]

@webknjaz Would be good if you can check this one over. The changes became more invasive than I originally imagined.

Basically, to fix the reported issue, I've changed the code to send the end bytes for a chunked request when the writer is cancelled. But, the method that does the cancellation is not async, so it can't wait for the writer to complete. To accommodate that, I've had to make all the exit/release code wait on the writer, to ensure it's finished writing before the connection can be reused.

I've had to make several small changes throughout the code in order to get the tests passing and maintain compatibility. The only behaviour change that can be seen in the tests, is that after a simple await sess.get(...) with a small response from the server, you'll find the response is not closed and connection not released immediately on the next line of code (as there's nothing to hook into to wait on the release). I don't believe anything in our docs has ever guaranteed that a connection would be released at this point, so I think the change is fine (the couple of affected tests I've just changed to use async with). We can still guarantee that the connection is released immediately after async with (our most recommended approach), after .close(), and after .read() (which I think our docs do say will release the connection).

[Dreamsorcerer]

I'm going to merge this one too, would be great to get a review in before the final 3.9 release though.

[patchback]

Backport to 3.9: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply 9c07121 on top of patchback/backports/3.9/9c071217f6594cb7da7133e575bea038b488ae80/pr-7764

Backporting merged PR #7764 into master

1. Ensure you have a local repo clone of your fork. Unless you cloned it
   from the upstream, this would be your origin remote.
2. Make sure you have an upstream repo added as a remote too. In these
   instructions you'll refer to it by the name upstream. If you don't
   have it, here's how you can add it:

   $ git remote add upstream https://github.com/aio-libs/aiohttp.git

3. Ensure you have the latest copy of upstream and prepare a branch
   that will hold the backported code:

   $ git fetch upstream
   $ git checkout -b patchback/backports/3.9/9c071217f6594cb7da7133e575bea038b488ae80/pr-7764 upstream/3.9

4. Now, cherry-pick PR Fix missing eof when writer cancelled #7764 contents into that branch:

   $ git cherry-pick -x 9c071217f6594cb7da7133e575bea038b488ae80

   If it'll yell at you with something like fatal: Commit 9c071217f6594cb7da7133e575bea038b488ae80 is a merge but no -m option was given., add -m 1 as follows instead:

   $ git cherry-pick -m1 -x 9c071217f6594cb7da7133e575bea038b488ae80

5. At this point, you'll probably encounter some merge conflicts. You must
   resolve them in to preserve the patch from PR Fix missing eof when writer cancelled #7764 as close to the
   original as possible.
6. Push this branch to your fork on GitHub:

   $ git push origin patchback/backports/3.9/9c071217f6594cb7da7133e575bea038b488ae80/pr-7764

7. Create a PR, ensure that the CI is green. If it's not — update it so that
   the tests and any other checks pass. This is it!
   Now relax and wait for the maintainers to process your pull request
   when they have some cycles to do reviews. Don't worry — they'll tell you if
   any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.