Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve exception handling in .NET isolated #3034

Open
wants to merge 5 commits into
base: dev
Choose a base branch
from

Conversation

andystaples
Copy link
Contributor

@andystaples andystaples commented Feb 6, 2025

Improves exception handling in .NET isolated by serializing the exception details in the worker middleware, then deserializing them back in host middleware.
This may introduce breaking changes to the way that the Functions Host logs exceptions from Durable activities.
Also adds testing for error handling, activity retry, and activity timeout scenarios in dotnet-isolated

Resolves #2711

Pull request checklist

  • My changes do not require documentation changes
    • Otherwise: Documentation PR is ready to merge and referenced in pending_docs.md
  • My changes should not be added to the release notes for the next release
    • Otherwise: I've added my notes to release_notes.md
  • My changes do not need to be backported to a previous version
    • Otherwise: Backport tracked by issue/PR #issue_or_pr
  • I have added all required tests (Unit tests, E2E tests)
  • My changes do not require any extra work to be leveraged by OutOfProc SDKs
    • Otherwise: That work is being tracked here: #issue_or_pr_in_each_sdk
  • My changes do not change the version of the WebJobs.Extensions.DurableTask package
    • Otherwise: major or minor version updates are reflected in /src/Worker.Extensions.DurableTask/AssemblyInfo.cs
  • My changes do not add EventIds to our EventSource logs
    • Otherwise: Ensure the EventIds are within the supported range in our existing Windows infrastructure. You may validate this with a deployed app's telemetry. You may also extend the range by completing a PR such as this one.
  • My changes should be added to v2.x branch.
    • Otherwise: This change applies exclusively to WebJobs.Extensions.DurableTask v3.x. It will be retained only in the dev and main branches and will not be merged into the v2.x branch.

- Serialize exception details in worker middleware
- Attempt to deserialize in host middleware
- Ensure inner exceptions + complex messages are preserved
@andystaples andystaples changed the title Add error handling and timeout tests Improve exception handling in .NET isolated Feb 6, 2025
test/e2e/Tests/Tests/ErrorHandlingTests.cs Outdated Show resolved Hide resolved

var orchestrationDetails = await DurableHelpers.GetRunningOrchestrationDetailsAsync(statusQueryGetUri);
Assert.StartsWith("Microsoft.DurableTask.TaskFailedException", orchestrationDetails.Output);
Assert.Contains("This activity failed", orchestrationDetails.Output);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there anything more we can validate here? For example, the type of the inner exception?

test/e2e/Tests/Tests/ErrorHandlingTests.cs Outdated Show resolved Hide resolved
test/e2e/Tests/Tests/TimeoutTests.cs Show resolved Hide resolved
src/WebJobs.Extensions.DurableTask/OutOfProcMiddleware.cs Outdated Show resolved Hide resolved

public override string? StackTrace => this.FromException.StackTrace;

private static TaskFailureDetails? ExceptionToTaskFailureDetailsRecursive(Exception? fromException)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a shared version of this we can use? Or is it in some inaccessible place, so we have to copy it? I'm a tiny bit worried about keeping multiple copies up-to-date if we decide to add richer information to TaskFailureDetails (which is very likely).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to a new TaskFailureDetailsConverter

Comment on lines +130 to +133
catch (Exception ex)
{
throw new DurableSerializationException(ex);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be beneficial to catch and do this for orchestration and entity triggers as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[out-of-proc] RetryContext.LastFailure incorrectly captures thrown exception
2 participants