Support Failure Recovery

[arhimondr]

This issue is a parent issue for failure recovery support in Trino

High level design document (open for commenting): https://docs.google.com/document/d/18QQy8dkZQDNiBPEe3bsyt3RhImCKye4fUJB8bvoeLBI

Full Query Retries

• Introduce task attempt concept: Implement full query retries #9361
• Implement de-duplication buffer: Implement full query retries #9361
• Refactor SqlStageExecution to support task re-scheduling: Implement full query retries #9361
• Refactor SqlQueryScheduler to support full query retries: Implement full query retries #9361
• Implement failure injection based integration tests for failure recovery: Implement full query retries #9361
• Remove de-duplication buffer capacity limitations to support failure recovery for queries with large output data set: Deduplication buffer spooling #10507
• Support dynamic filtering for full query retries #9934
• Original failure cause sometimes lost with query retries: Original failure cause sometimes lost with query retries #10395

Task Level Retries

• Introduce external exchange interface Implement task level retries #9818
• Support task level retries in de-duplication buffer Implement task level retries #9818
• Implement fault tolerant scheduler Implement task level retries #9818
• Implement weight based split assignment Handle split weights in SourceDistributionTaskSource #10837
• Implement adaptive task sizing for intermediate stages Join intermediate tasks for small partitions #11023
• Account memory retained by splits on coordinator Limit memory used by fault tolerant scheduler on coordinator #10877
• Support dynamic filtering for fault tolerant execution #9935
• Optimize Page serialization for external exchange (Remove SerializedPage #10241)
• Implement DFS based external exchange #9936
• Implement Parallel Read from S3 for external exchange #11050
• Run memory intensive tasks on a dedicated node (Allow assigning exclusive node for task execution #10432)
• Allow memory over-commit for fault tolerant execution #11051
• Do not block FaultTolerantStageScheduler loop if node cannot be acquire for single task #10808
• Wait with timeout instead failing immediately if no candidate nodes available in NodeAllocator #11020
• Implement task descriptor storage offloading for fault tolerant execution #12493

Reporting and UI

• Collect and report task failure related statistics in QueryCompletedEvent #10734
• Improve Trino UI to display Tardigrade related information #11959

Connectors support

Ensure DML operations in connectors follow Trino commit semantics and don't produce unwanted side effects

• Ensure query retries work with Hive connector #10252
• Add tests for Iceberg connector correctness with full query retries #10253
• Ensure full query retries work with JDBC connectors based on base-jdbc #10254
• Enable query retries for Delta Lake Connector #11591
• ...

Backlog

• Extend error code with "retriable" field. Mark existing errors accordingly.

[findepi]

Closing in favor of #2909

[arhimondr]

https://trino.io/blog/2022/05/05/tardigrade-launch.html