Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor fault tolerant scheduler components #14069

Merged

Conversation

arhimondr
Copy link
Contributor

Description

Multiple refactors to prepare the components for the upcoming major scheduler refactor

Non-technical explanation

N/A

Release notes

(X) This is not user-visible and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

HostAddress requiredAddress = bucketNodeMap.getAssignedNode(split).getHostAndPort();
int partition = sourcePartitioningScheme.getPartition(split);
Optional<InternalNode> assignedNode = sourcePartitioningScheme.getNodeRequirement(partition);
if (assignedNode.isPresent()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this check now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently the buckets are always pre-assigned to nodes (this is how NodePartitioningManager works). This is a problem for fault tolerant execution as it limits the number of partitions (sets it equal to the number of nodes). In the future we need to resolve this issue. Once the NodePartitioningManager no longer pre-assignes the nodes to buckets there will be no more partition-to-node mapping available. I wanted to make it future proof to avoid another refactoring.

@arhimondr arhimondr force-pushed the fault-tolerant-scheduler-refactors branch from 7577495 to aa990e1 Compare September 12, 2022 17:43
@arhimondr
Copy link
Contributor Author

@losipiuk Updated

@arhimondr arhimondr force-pushed the fault-tolerant-scheduler-refactors branch from aa990e1 to 48ea533 Compare September 14, 2022 18:34
The memory requirement might get readjusted while TaskDescriptor is
supposed to be immutable
@arhimondr arhimondr force-pushed the fault-tolerant-scheduler-refactors branch from 48ea533 to 82be803 Compare September 14, 2022 18:44
@arhimondr
Copy link
Contributor Author

Benchmarks are fine:

+----------------------+-----------------------+----------------------+-----------------------+----------+-----------+
| base_cpu_time_millis | base_wall_time_millis | test_cpu_time_millis | test_wall_time_millis | cpu_diff | wall_diff |
+----------------------+-----------------------+----------------------+-----------------------+----------+-----------+
|          18118319480 |              84493758 |          18230577530 |              84198581 |  1.00620 |   0.99651 |
+----------------------+-----------------------+----------------------+-----------------------+----------+-----------+

+-------------------------------+----------------------+-----------------------+----------------------+-----------------------+----------+-----------+
| suite                         | base_cpu_time_millis | base_wall_time_millis | test_cpu_time_millis | test_wall_time_millis | cpu_diff | wall_diff |
+-------------------------------+----------------------+-----------------------+----------------------+-----------------------+----------+-----------+
| tpcds_sf10000_partitioned     |           2120577324 |              12409639 |           2122791290 |              12255470 |  1.00104 |   0.98758 |
| tpcds_sf10000_partitioned_etl |          12613638112 |              46905557 |          12752260924 |              47210840 |  1.01099 |   1.00651 |
| tpcds_sf100_partitioned       |             25010870 |               3544815 |             24366593 |               3460751 |  0.97424 |   0.97629 |
| tpcds_sf100_partitioned_etl   |            113428024 |               7388384 |            110546905 |               7306338 |  0.97460 |   0.98890 |
| tpch_sf10000_bucketed         |            816035318 |               3238805 |            812939762 |               3221123 |  0.99621 |   0.99454 |
| tpch_sf10000_bucketed_etl     |           2402861533 |               9420345 |           2381666156 |               9232764 |  0.99118 |   0.98009 |
| tpch_sf100_bucketed           |              6338395 |                198803 |              6374424 |                199216 |  1.00568 |   1.00208 |
| tpch_sf100_bucketed_etl       |             20429904 |               1387410 |             19631476 |               1312079 |  0.96092 |   0.94570 |
+-------------------------------+----------------------+-----------------------+----------------------+-----------------------+----------+-----------+

Detailed: https://gist.github.com/arhimondr/407eeaf2174949b0ef3e4b75c1a095b6

@arhimondr arhimondr merged commit e1f688d into trinodb:master Sep 16, 2022
@arhimondr arhimondr deleted the fault-tolerant-scheduler-refactors branch September 16, 2022 02:19
@github-actions github-actions bot added this to the 397 milestone Sep 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

3 participants