Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DownloadsBackend inconsistently closes streams #471

Closed
omus opened this issue Sep 29, 2021 · 3 comments
Closed

DownloadsBackend inconsistently closes streams #471

omus opened this issue Sep 29, 2021 · 3 comments

Comments

@omus
Copy link
Member

omus commented Sep 29, 2021

The following test case is from #470:

@testset "issue 466" begin
    file_name = "hang.txt"

    try
        S3.put_object("anewbucket", file_name)

        # Note: Using `eof` for these tests can hang when using an unclosed `Base.BufferStream`

        stream = S3.get_object("anewbucket", file_name, Dict("return_stream" => true))
        if AWS.DEFAULT_BACKEND[] isa AWS.HTTPBackend
            @test !isopen(stream)
        else
            @test isopen(stream)
        end

        stream = Base.BufferStream()
        S3.get_object("anewbucket", file_name, Dict("response_stream" => stream))
        if AWS.DEFAULT_BACKEND[] isa AWS.HTTPBackend
            @test !isopen(stream)
        else
            @test_broken isopen(stream)
        end

        stream = Base.BufferStream()
        S3.get_object(
            "anewbucket",
            file_name,
            Dict("response_stream" => stream, "return_stream" => true),
        )
        if AWS.DEFAULT_BACKEND[] isa AWS.HTTPBackend
            @test !isopen(stream)
        else
            @test isopen(stream)
        end
    finally
        S3.delete_object("anewbucket", file_name)
    end
end

Note how when using a non-IOBuffer and not using the "return_stream" parameter that the I/O stream is closed but in all other cases it leaves the stream open. This inconsistency should not be present.

Additionally, most likely the behaviour of the HTTPBackend and the DownloadsBackend should made to be consistent.

@omus omus changed the title DownloadsBackend inconsitently closes streams DownloadsBackend inconsistently closes streams Sep 29, 2021
@omus
Copy link
Member Author

omus commented Sep 29, 2021

Related to #433

@ericphanson
Copy link
Member

Just to add another ref, I think this is case 2 from #396 (comment)

bors bot added a commit that referenced this issue Sep 29, 2021
384: Use `AWS.Response` to handle streaming/raw/parsing r=omus a=omus

The idea is to use a struct in AWS.jl that can be used to handle the automatic parsing that is currently used to turn XML/JSON into a dict while also giving the option of accessing the raw output as a string or stream without all the keywords currently needed to be specified.

Depends on:
- #457

Related:
- #346
- #348
- #438

Closes:
- #224
- #433
- #468 (when using `use_response_type` the passed in I/O is not closed)
- #471
- #433

Update: The tests in this PR run using the deprecated behaviour. Mainly I did that here to prove this change is non-breaking. For the updated tests see #458 

Co-authored-by: Curtis Vogt <[email protected]>
@omus
Copy link
Member Author

omus commented Sep 29, 2021

As of AWS.jl 1.63.0 (#384) streams are no longer closed when using the feature use_response_type (e.g. @service S3 use_response_type=true). If you enable this feature and pass in a Base.BufferStream you're responsible for closing it yourself.

When not enabling the use_response_type feature the current behaviour as of AWS 1.62.1 remains unchanged.

@omus omus closed this as completed Sep 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants