Add debug logging messages on retry #549

ericphanson · 2022-05-11T18:16:57Z

Closes #514

Based on #548

The logging messages are structured so the string is a human-readable summary of the message, and then the fields are more structured for later analysis, e.g. retry::Bool, reason::String, etc. Additionally, the id of the log message will be useful for aggregation.

ericphanson · 2022-05-11T18:17:07Z

Let's see if CI likes it:

bors try

src/utilities/request.jl

bors · 2022-05-11T18:23:39Z

try

Build succeeded:

christopher-dG

LGTM!

ericphanson · 2022-05-13T15:08:51Z

ok great! I think we should probably first review/merge #548, then merge this on top.

test/issues.jl

mattBrzezinski

bors r+

bors · 2023-03-20T20:33:13Z

Merge conflict.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

mattBrzezinski

bors r+

549: Add debug logging messages on retry r=mattBrzezinski a=ericphanson Closes #514 Based on #548 The logging messages are structured so the string is a human-readable summary of the message, and then the fields are more structured for later analysis, e.g. `retry::Bool`, `reason::String`, etc. Additionally, the id of the log message will be useful for aggregation. Co-authored-by: Eric Hanson <[email protected]> Co-authored-by: Matt Brzezinski <[email protected]>

bors · 2023-03-20T20:49:45Z

Build failed:

Julia 1 - ubuntu-latest - x64

mattBrzezinski · 2023-03-20T20:57:29Z

bors r+

549: Add debug logging messages on retry r=mattBrzezinski a=ericphanson Closes #514 Based on #548 The logging messages are structured so the string is a human-readable summary of the message, and then the fields are more structured for later analysis, e.g. `retry::Bool`, `reason::String`, etc. Additionally, the id of the log message will be useful for aggregation. Co-authored-by: Eric Hanson <[email protected]> Co-authored-by: Matt Brzezinski <[email protected]>

github-actions · 2023-03-20T20:58:53Z

src/utilities/downloads_backend.jl

@@ -110,6 +110,19 @@ function _http_request(backend::DownloadsBackend, request::Request, response_str
        return nothing
    end

+    check = (s, e) -> begin


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

check = (s, e) -> begin

check =

(s, e) -> begin

github-actions · 2023-03-20T20:58:53Z

src/utilities/downloads_backend.jl

@@ -110,6 +110,19 @@ function _http_request(backend::DownloadsBackend, request::Request, response_str
        return nothing
    end

+    check = (s, e) -> begin
+            if isa(e, HTTP.StatusError) && AWS._http_status(e) >= 500
+                @debug "AWS.jl Downloads inner retry for status >= 500" retry=true reason="status >= 500" status=_http_status(e) exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl Downloads inner retry for status >= 500" retry=true reason="status >= 500" status=_http_status(e) exception=e

@debug "AWS.jl Downloads inner retry for status >= 500" retry = true reason = "status >= 500" status = _http_status(

e

) exception = e

github-actions · 2023-03-20T20:58:53Z

src/utilities/downloads_backend.jl

+                @debug "AWS.jl Downloads inner retry for status >= 500" retry=true reason="status >= 500" status=_http_status(e) exception=e
+                return true
+            elseif isa(e, Downloads.RequestError)
+                @debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry=true reason="Downloads.RequestError" exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry=true reason="Downloads.RequestError" exception=e

@debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry =

true reason = "Downloads.RequestError" exception = e

github-actions · 2023-03-20T20:58:53Z

src/utilities/downloads_backend.jl

+                @debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry=true reason="Downloads.RequestError" exception=e
+                return true
+            else
+                @debug "AWS.jl Downloads inner retry declined" retry=false reason="Exception passed onwards" exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl Downloads inner retry declined" retry=false reason="Exception passed onwards" exception=e

@debug "AWS.jl Downloads inner retry declined" retry = false reason = "Exception passed onwards" exception =

e

github-actions · 2023-03-20T20:58:54Z

src/utilities/request.jl

+    check = (s, e) -> begin
        # Pass on non-AWS exceptions.
        if !(e isa AWSException)
+            @debug "AWS.jl declined to retry non-AWSException" retry=false reason="Non-AWSException" exception=e
            return false
        end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

check = (s, e) -> begin

# Pass on non-AWS exceptions.

if !(e isa AWSException)

@debug "AWS.jl declined to retry non-AWSException" retry=false reason="Non-AWSException" exception=e

return false

end

check =

(s, e) -> begin

# Pass on non-AWS exceptions.

if !(e isa AWSException)

@debug "AWS.jl declined to retry non-AWSException" retry = false reason = "Non-AWSException" exception =

e

return false

end

github-actions · 2023-03-20T20:58:56Z

src/utilities/request.jl

+            if isa(e, error)
+                @debug "AWS.jl HTTP inner retry for $error" retry = true reason = "$error" exception =
+                    e
+                return true


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

return true

) exception = e

return true

github-actions · 2023-03-20T20:58:56Z

src/utilities/request.jl

+        end
+        if (isa(e, HTTP.StatusError) && _http_status(e) >= 500)
+            @debug "AWS.jl HTTP inner retry for status >= 500" retry = true reason = "status >= 500" status = AWS._http_status(


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

end

if (isa(e, HTTP.StatusError) && _http_status(e) >= 500)

@debug "AWS.jl HTTP inner retry for status >= 500" retry = true reason = "status >= 500" status = AWS._http_status(

@debug "AWS.jl HTTP inner retry declined" retry = false reason = "Exception passed onwards" exception =

github-actions · 2023-03-20T20:58:56Z

src/utilities/request.jl

+            ) exception = e
+            return true


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

) exception = e

return true

return false

github-actions · 2023-03-20T20:58:56Z

src/utilities/request.jl

+        @debug "AWS.jl HTTP inner retry declined" retry = false reason = "Exception passed onwards" exception =
+            e
+        return false
+    end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl HTTP inner retry declined" retry = false reason = "Exception passed onwards" exception =

e

return false

end

github-actions · 2023-03-20T20:58:56Z

test/issues.jl

+        record.level == Logging.Debug &&
+        get(record.kwargs, :retry, nothing) == successful


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

record.level == Logging.Debug &&

get(record.kwargs, :retry, nothing) == successful

record.level == Logging.Debug && get(record.kwargs, :retry, nothing) == successful

bors · 2023-03-20T21:02:37Z

Build failed:

Julia 1 - ubuntu-latest - x64

mattBrzezinski · 2023-03-20T21:17:53Z

bors try

bors · 2023-03-20T21:22:29Z

try

Build failed:

Julia 1 - ubuntu-latest - x64

github-actions · 2023-03-20T21:26:29Z

src/utilities/request.jl

@@ -147,52 +147,71 @@ function submit_request(aws::AbstractAWSConfig, request::Request; return_headers
            if e isa HTTP.StatusError
                e = AWSException(e, stream)
                rethrow(e)
+            elseif !(e isa AWSException)
+                @debug "AWS.jl declined to retry non-AWSException" retry=false reason="Non-AWSException" exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl declined to retry non-AWSException" retry=false reason="Non-AWSException" exception=e

@debug "AWS.jl declined to retry non-AWSException" retry = false reason = "Non-AWSException" exception =

e

mattBrzezinski · 2023-03-20T21:27:22Z

bors try

bors · 2023-03-20T21:31:57Z

try

Build failed:

Julia 1.6 - ubuntu-latest - x64

mattBrzezinski · 2023-03-20T21:40:07Z

bors try

github-actions · 2023-03-20T21:41:40Z

src/utilities/downloads_backend.jl

@@ -110,6 +110,19 @@ function _http_request(backend::DownloadsBackend, request::Request, response_str
        return nothing
    end

+    check = (s, e) -> begin
+            if isa(e, HTTP.StatusError) && AWS._http_status(e) >= 500
+                @debug "AWS.jl Downloads inner retry for status >= 500" retry=true reason="status >= 500" status=AWS._http_status(e) exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

@debug "AWS.jl Downloads inner retry for status >= 500" retry=true reason="status >= 500" status=AWS._http_status(e) exception=e

@debug "AWS.jl Downloads inner retry for status >= 500" retry = true reason = "status >= 500" status = AWS._http_status(

e

) exception = e

github-actions · 2023-03-20T21:41:40Z

src/utilities/request.jl

+    check = (s, e) -> begin
+        errors = (Sockets.DNSError, HTTP.ParseError, Base.IOError)
+        for error in errors
+            if isa(e, error)
+                @debug "AWS.jl HTTP inner retry for $error" retry=true reason="$error" exception=e


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

check = (s, e) -> begin

errors = (Sockets.DNSError, HTTP.ParseError, Base.IOError)

for error in errors

if isa(e, error)

@debug "AWS.jl HTTP inner retry for $error" retry=true reason="$error" exception=e

check =

(s, e) -> begin

errors = (Sockets.DNSError, HTTP.ParseError, Base.IOError)

for error in errors

if isa(e, error)

@debug "AWS.jl HTTP inner retry for $error" retry = true reason = "$error" exception =

e

return true

end

end

if (isa(e, HTTP.StatusError) && _http_status(e) >= 500)

@debug "AWS.jl HTTP inner retry for status >= 500" retry = true reason = "status >= 500" status = _http_status(

e

) exception = e

github-actions · 2023-03-20T21:41:40Z

src/utilities/request.jl

+                @debug "AWS.jl HTTP inner retry for $error" retry=true reason="$error" exception=e
+                return true
+            end
+        end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

end

@debug "AWS.jl HTTP inner retry declined" retry = false reason = "Exception passed onwards" exception =

e

return false

end

github-actions · 2023-03-20T21:41:40Z

src/utilities/request.jl

+        if (isa(e, HTTP.StatusError) && _http_status(e) >= 500)
+            @debug "AWS.jl HTTP inner retry for status >= 500" retry=true reason="status >= 500" status=_http_status(e) exception=e
+            return true
+        end
+        @debug "AWS.jl HTTP inner retry declined" retry=false reason="Exception passed onwards" exception=e
+        return false
+    end


[JuliaFormatter] _{reported by reviewdog 🐶}

Suggested change

if (isa(e, HTTP.StatusError) && _http_status(e) >= 500)

@debug "AWS.jl HTTP inner retry for status >= 500" retry=true reason="status >= 500" status=_http_status(e) exception=e

return true

end

@debug "AWS.jl HTTP inner retry declined" retry=false reason="Exception passed onwards" exception=e

return false

end

mattBrzezinski · 2023-03-20T21:43:15Z

bors try

bors · 2023-03-20T21:43:16Z

try

Already running a review

bors · 2023-03-20T21:45:17Z

try

Build failed:

Julia 1.6 - ubuntu-latest - x64

mattBrzezinski · 2023-03-21T14:29:55Z

bors try

bors · 2023-03-21T14:34:38Z

try

Build failed:

Julia 1 - ubuntu-latest - x64

mattBrzezinski · 2023-03-21T16:31:41Z

Discussed offline, but we'll be revising how to properly introduce debugging here, closing this PR for now.

bors bot added a commit that referenced this pull request May 11, 2022

Try #549:

921ada6

github-actions bot reviewed May 11, 2022

View reviewed changes

src/utilities/request.jl Outdated Show resolved Hide resolved

christopher-dG approved these changes May 12, 2022

View reviewed changes

ericphanson mentioned this pull request May 19, 2022

Increase some retries from 4 to 10 #550

Closed

omus reviewed May 19, 2022

View reviewed changes

test/issues.jl Outdated Show resolved Hide resolved

test/issues.jl Outdated Show resolved Hide resolved

Base automatically changed from eph/base-retry to master May 25, 2022 15:08

mattBrzezinski approved these changes Mar 20, 2023

View reviewed changes

ericphanson and others added 6 commits March 20, 2023 15:45

wip

5a94611

add more logging

5d69160

wip tests

ba7a65a

add tests

c4741ba

Update src/utilities/request.jl

f960817

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Longer line lengths

f5c8cc7

mattBrzezinski force-pushed the eph/log-retries branch from 952895b to f5c8cc7 Compare March 20, 2023 20:46

Rebase fix

95ccc75

mattBrzezinski approved these changes Mar 20, 2023

View reviewed changes

Rebase fixes x2

d8a00dd

github-actions bot reviewed Mar 20, 2023

View reviewed changes

bors bot added a commit that referenced this pull request Mar 20, 2023

Try #549:

fc02ce8

Fix?

5b62567

github-actions bot reviewed Mar 20, 2023

View reviewed changes

bors bot added a commit that referenced this pull request Mar 20, 2023

Try #549:

b47f367

one more pass

5b6c716

bors bot added a commit that referenced this pull request Mar 20, 2023

Try #549:

e307bcb

github-actions bot reviewed Mar 20, 2023

View reviewed changes

debug statements

fcc394b

bors bot added a commit that referenced this pull request Mar 21, 2023

Try #549:

48e59f1

mattBrzezinski closed this Mar 21, 2023

	@debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry=true reason="Downloads.RequestError" exception=e
	@debug "AWS.jl Downloads inner retry for Downloads.RequestError" retry =
	true reason = "Downloads.RequestError" exception = e

	@debug "AWS.jl Downloads inner retry declined" retry=false reason="Exception passed onwards" exception=e
	@debug "AWS.jl Downloads inner retry declined" retry = false reason = "Exception passed onwards" exception =
	e

		record.level == Logging.Debug &&
		get(record.kwargs, :retry, nothing) == successful

	record.level == Logging.Debug &&
	get(record.kwargs, :retry, nothing) == successful
	record.level == Logging.Debug && get(record.kwargs, :retry, nothing) == successful

	@debug "AWS.jl declined to retry non-AWSException" retry=false reason="Non-AWSException" exception=e
	@debug "AWS.jl declined to retry non-AWSException" retry = false reason = "Non-AWSException" exception =
	e

Add debug logging messages on retry #549

Add debug logging messages on retry #549

Conversation

ericphanson commented May 11, 2022

ericphanson commented May 11, 2022

bors bot commented May 11, 2022

try

christopher-dG left a comment

Choose a reason for hiding this comment

ericphanson commented May 13, 2022

mattBrzezinski left a comment

Choose a reason for hiding this comment

bors bot commented Mar 20, 2023

mattBrzezinski left a comment

Choose a reason for hiding this comment

bors bot commented Mar 20, 2023

mattBrzezinski commented Mar 20, 2023

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

bors bot commented Mar 20, 2023

mattBrzezinski commented Mar 20, 2023

bors bot commented Mar 20, 2023

try

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

mattBrzezinski commented Mar 20, 2023

bors bot commented Mar 20, 2023

try

mattBrzezinski commented Mar 20, 2023

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

github-actions bot Mar 20, 2023

Choose a reason for hiding this comment

mattBrzezinski commented Mar 20, 2023

bors bot commented Mar 20, 2023

try

bors bot commented Mar 20, 2023

try

mattBrzezinski commented Mar 21, 2023

bors bot commented Mar 21, 2023

try

mattBrzezinski commented Mar 21, 2023