Inconsistent implementation between turn_indexes_write_buffer and turn_weight_penalties/turn_duration_penalties

[wangyoucao577]

All the index/weight/duration of turns are generated and written into file in void EdgeBasedGraphFactory::GenerateEdgeExpandedEdges. However, their implementation are a little bit different.

Both turn_weight_penalties and turn_duration_penalties are vectors that cross the function, and will be written into file once when the whole processing finished.

osrm-backend/src/extractor/edge_based_graph_factory.cpp

Lines 472 to 474 in 9f80f6d

	// Now, renumber all our maneuver overrides to use edge-based-nodes
	std::vector<StorageManeuverOverride> storage_maneuver_overrides;
	std::vector<NodeID> maneuver_override_sequences;

osrm-backend/src/extractor/edge_based_graph_factory.cpp

Lines 1164 to 1167 in 9f80f6d

	// write weight penalties per turn
	BOOST_ASSERT(turn_weight_penalties.size() == turn_duration_penalties.size());
	files::writeTurnWeightPenalty(turn_weight_penalties_filename, turn_weight_penalties);
	files::writeTurnDurationPenalty(turn_duration_penalties_filename, turn_duration_penalties);

By contrast, turn_indexes_write_buffer will be written to file per 1000. It leads to the write to file action dispersedly in a few places.

osrm-backend/src/extractor/edge_based_graph_factory.cpp

Lines 452 to 454 in 9f80f6d

	storage::tar::FileWriter turn_penalties_index_file(
	turn_penalties_index_filename, storage::tar::FileWriter::GenerateFingerprint);
	turn_penalties_index_file.WriteFrom("/extractor/turn_index", (char *)nullptr, 0);

osrm-backend/src/extractor/edge_based_graph_factory.cpp

Lines 490 to 496 in 9f80f6d

	// Because we write TurnIndexBlock data as we go, we'll
	// buffer them into groups of 1000 to reduce the syscall
	// count by 1000x. This doesn't need much memory, but
	// greatly reduces the syscall overhead of writing lots
	// of small objects
	std::vector<lookup::TurnIndexBlock> turn_indexes_write_buffer;
	turn_indexes_write_buffer.reserve(TURN_INDEX_WRITE_BUFFER_SIZE);

osrm-backend/src/extractor/edge_based_graph_factory.cpp

Lines 1057 to 1064 in 9f80f6d

	// Buffer writes to reduce syscall count
	if (turn_indexes_write_buffer.size() >= TURN_INDEX_WRITE_BUFFER_SIZE)
	{
	turn_penalties_index_file.ContinueFrom("/extractor/turn_index",
	turn_indexes_write_buffer.data(),
	turn_indexes_write_buffer.size());
	turn_indexes_write_buffer.clear();
	}

After go through the codes, I think unify the process of turn_indexes same as turn_weight_penalties/turn_duration_penalties should be ok, right? Any concern?

[akashihi]

(I haven't checked the actual code) It could be saved in batches due to ram limits, which is not a concern in 2020:) so worth trying in my opinion.

[wangyoucao577]

@akashihi I have file a PR, please help to review. thanks!

[akashihi]

@wangyoucao577 Done, sorry for delay :(