-
Notifications
You must be signed in to change notification settings - Fork 62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Always close the response_stream
after HTTP.request
#467
Merged
bors
merged 8 commits into
JuliaCloud:master
from
nickrobinson251:npr/466-close-response-stream
Sep 28, 2021
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
631dbb8
Always close the `response_stream`
nickrobinson251 65da857
Only try to close an actual stream
nickrobinson251 4dcecc6
Add test
nickrobinson251 02687e6
Don't keep creating and deleting bucekts in `issues` tests
nickrobinson251 27dfaee
Wrap tests in try/finally so `delete_bucket` always called
nickrobinson251 f0d6e11
try to make formatter happy
nickrobinson251 817ba0c
try again to make formatter happy
nickrobinson251 8dde0b0
Update test/issues.jl
nickrobinson251 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,83 +1,102 @@ | ||
@service S3 | ||
|
||
bucket_name = "aws-jl-test-issues---" * _now_formatted() | ||
S3.create_bucket(bucket_name) | ||
|
||
@testset "issue 223" begin | ||
# https://github.com/JuliaCloud/AWS.jl/issues/223 | ||
body = "Hello World!" | ||
file_name = "contains spaces" | ||
|
||
try | ||
S3.put_object(bucket_name, file_name, Dict("body" => body)) | ||
resp = S3.get_object(bucket_name, file_name) | ||
|
||
@test String(resp) == body | ||
finally | ||
S3.delete_object(bucket_name, file_name) | ||
S3.delete_bucket(bucket_name) | ||
BUCKET_NAME = "aws-jl-test-issues---" * _now_formatted() | ||
|
||
try | ||
S3.create_bucket(BUCKET_NAME) | ||
|
||
@testset "issue 223" begin | ||
# https://github.com/JuliaCloud/AWS.jl/issues/223 | ||
body = "Hello World!" | ||
file_name = "contains spaces" | ||
|
||
try | ||
S3.put_object(BUCKET_NAME, file_name, Dict("body" => body)) | ||
resp = S3.get_object(BUCKET_NAME, file_name) | ||
|
||
@test String(resp) == body | ||
finally | ||
S3.delete_object(BUCKET_NAME, file_name) | ||
end | ||
end | ||
end | ||
|
||
@testset "issue 227" begin | ||
@testset "s3 public bucket" begin | ||
# https://github.com/JuliaCloud/AWS.jl/issues/227 | ||
config = AWSConfig(; creds=nothing) | ||
resp = S3.get_object("www.invenia.ca", "index.html"; aws_config=config) | ||
@testset "issue 227" begin | ||
@testset "s3 public bucket" begin | ||
# https://github.com/JuliaCloud/AWS.jl/issues/227 | ||
config = AWSConfig(; creds=nothing) | ||
resp = S3.get_object("www.invenia.ca", "index.html"; aws_config=config) | ||
|
||
@test !isempty(resp) | ||
end | ||
|
||
@testset "s3 private bucket" begin | ||
bucket_name = "aws-jl-test-issues---" * _now_formatted() | ||
file_name = "hello_world" | ||
|
||
try | ||
S3.create_bucket(bucket_name) | ||
S3.put_object(bucket_name, file_name) | ||
|
||
@test !isempty(resp) | ||
@test_throws AWSException S3.get_object( | ||
bucket_name, file_name; aws_config=AWSConfig(; creds=nothing) | ||
) | ||
finally | ||
S3.delete_object(bucket_name, file_name) | ||
S3.delete_bucket(bucket_name) | ||
end | ||
end | ||
|
||
@testset "lambda" begin | ||
@service Lambda | ||
|
||
@test_throws NoCredentials Lambda.list_functions(; | ||
aws_config=AWSConfig(; creds=nothing) | ||
) | ||
end | ||
end | ||
|
||
@testset "s3 private bucket" begin | ||
bucket_name = "aws-jl-test-issues---" * _now_formatted() | ||
file_name = "hello_world" | ||
@testset "issue 324" begin | ||
body = "Hello World!" | ||
file_name = "streaming.bin" | ||
|
||
try | ||
S3.create_bucket(bucket_name) | ||
S3.put_object(bucket_name, file_name) | ||
S3.put_object(BUCKET_NAME, file_name, Dict("body" => body)) | ||
resp = S3.get_object(BUCKET_NAME, file_name) | ||
@test String(resp) == body | ||
|
||
@test_throws AWSException S3.get_object( | ||
bucket_name, file_name; aws_config=AWSConfig(; creds=nothing) | ||
# ERROR: MethodError: no method matching iterate(::Base.BufferStream) | ||
# => BUG: header `response_stream` is pushed into the query... | ||
io = Base.BufferStream() | ||
S3.get_object( | ||
BUCKET_NAME, | ||
file_name, | ||
Dict("response_stream" => io, "return_stream" => true), | ||
) | ||
if bytesavailable(io) > 0 | ||
@test String(readavailable(io)) == body | ||
else | ||
@test "no body data was available" == body | ||
end | ||
|
||
finally | ||
S3.delete_object(bucket_name, file_name) | ||
S3.delete_bucket(bucket_name) | ||
S3.delete_object(BUCKET_NAME, file_name) | ||
end | ||
end | ||
|
||
@testset "lambda" begin | ||
@service Lambda | ||
|
||
@test_throws NoCredentials Lambda.list_functions(; | ||
aws_config=AWSConfig(; creds=nothing) | ||
) | ||
end | ||
end | ||
@testset "issue 466" begin | ||
file_name = "hang.txt" | ||
|
||
@testset "issue 324" begin | ||
body = "Hello World!" | ||
file_name = "streaming.bin" | ||
|
||
try | ||
S3.create_bucket(bucket_name) | ||
S3.put_object(bucket_name, file_name, Dict("body" => body)) | ||
resp = S3.get_object(bucket_name, file_name) | ||
@test String(resp) == body | ||
|
||
# ERROR: MethodError: no method matching iterate(::Base.BufferStream) | ||
# => BUG: header `response_stream` is pushed into the query... | ||
io = Base.BufferStream() | ||
S3.get_object( | ||
bucket_name, file_name, Dict("response_stream" => io, "return_stream" => true) | ||
) | ||
if bytesavailable(io) > 0 | ||
@test String(readavailable(io)) == body | ||
else | ||
@test "no body data was available" == body | ||
try | ||
S3.put_object(BUCKET_NAME, file_name) | ||
stream = S3.get_object(BUCKET_NAME, file_name, Dict("return_stream" => true)) | ||
println("test #466") # So we know if this is the reason for tests hanging. | ||
@test eof(stream) # This will hang if #466 not fixed and using HTTP.jl v0.9.15+ | ||
println("#466 fixed") | ||
finally | ||
S3.delete_object(BUCKET_NAME, file_name) | ||
end | ||
|
||
finally | ||
S3.delete_object(bucket_name, file_name) | ||
S3.delete_bucket(bucket_name) | ||
end | ||
|
||
finally | ||
S3.delete_bucket(BUCKET_NAME) | ||
end |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If an end-user were to pass in their own
response_stream
, what would happen here if it closes? Is that data still available in it, or does it go away?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was gonna suggest something like
earlier. E.g. only close if it was opened internally.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the behaviour is exactly as before (i.e. as we had with older HTTP.jl versions). In older HTTP versions,
close
was already called inside theHTTP.request
above https://github.com/JuliaWeb/HTTP.jl/pull/752/files#diff-2877f0a59db9c7dda95204097d3c2010021eb8f6c59c98d56d3f7d16e38b0acbL149There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the idea is that this maybe wasn't correct from HTTP.jl which is why it got changed there. #396 (comment) and #433 have lots of discussion on this (which I forgot completely about until just now)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wouldn't this be different behaviour to what we had when using older HTTP.jl versions?
i.e. didn't
HTTP.request
just callclose
on theresponse_stream
(if it wasn'tnothing
)?(i don't know, since i'm not too familiar with this stuff)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have not been paying too much attention to the HTTP.jl changes. Are there use cases anyone can think of to keeping the streams open and having users manually close them?
Personally I think closing the HTTP requests automatically makes sense. It gives a loose contract of, this is the data which AWS has provided you from your request.
Maybe I'm missing something obvious though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, maybe it should be reverted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW, i have no opinion on how best to fix this. But i am keen to fix it as fast as possible, as not being able get files from S3 is quite debilitating for some workflows at Invenia 😊
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, I would say we should merge this PR which basically brings back the behavior of HTTP 0.9.14 and then continue this discussion in an issue.
I would really like to see this merged today.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#468