Assign all sources/shards, even if this require exceeding the indexer #4363
Conversation
fulmicoton
force-pushed
the
issue/4290-allow-over-assigning
branch
4 times, most recently
from
ecc24a0 to
005380e
Compare
| } | ||
| } | ||
| } | ||
| assert!(load_in_node <= indexer_max_loads.get(node_id).unwrap().cpu_millis()); |
There was a problem hiding this comment.
After this issue, this assert is actually invalid so I removed it.
I added an assert checking that the physical plan matches the solution.
quickwit/quickwit-control-plane/src/indexing_scheduler/scheduling/mod.rs
Show resolved
Hide resolved
fulmicoton
force-pushed
the
issue/4290-allow-over-assigning
branch
7 times, most recently
from
8e00514 to
d213de3
Compare
capacities. We do that simply by scaling the indexer capacities. First we scale the indexer capacities so that the sum of their capacities exceeds the total load of the source to assign. If the packing algorithm fails, we inflates the capacities again by 10% and retry. Added assert on convert solution to physical solution. Closes #4290
fulmicoton
force-pushed
the
issue/4290-allow-over-assigning
branch
from
d213de3 to
0374250
Compare
| @@ -124,7 +125,21 @@ fn convert_physical_plan_to_solution( | |||
| source_id: indexing_task.source_id.clone(), | |||
| }; | |||
| if let Some(source_ord) = id_to_ord_map.source_ord(&source_uid) { | |||
| indexer_assignment.add_shards(source_ord, indexing_task.shard_ids.len() as u32); | |||
| let source_type = &source_ord_map.get(&source_ord).unwrap().source_type; | |||
There was a problem hiding this comment.
I'm always afraid of those unwraps, I'm unsure if a comment is worth it on the function or maybe replace unwrap by expect("")
| // we are certain that even on our first attempt, the total capacity of the indexer exceeds 120% | ||
| // of the partial solution. | ||
| // | ||
| // 1.2^30 > 240. |
|
So we have 2 layers here: we rewrite the problem in a space where all shards have the same type, all source have the same type, and source and indexer are identified by a small indexable id. So the logic goes
The logical part is proptested already. So far the bugs have been related to edge case on the conversion stuff. |
capacities.
We do that simply by scaling the indexer capacities. First we scale the indexer capacities so that the sum of their capacities exceeds the total load of the source to assign.
If the packing algorithm fails, we inflates the capacities again by 10% and retry.
Closes #4290