File pool does not work as intended

[HanabishiRecca]

I noticed strange behavior of file descriptors. Opened files stale indefinitely, regardless of the file pool size.

Some investigation with TRACE_FILE_VIEW_POOL enabled shows that libtorrent never rotates any descriptors in the pool, except constantly reusing the last one when the pool is filled up.

135805188216512 opening file: (18, 12)
135805188216512 opening file: (18, 6)
135805188216512 opening file: (0, 0)
135805188216512 opening file: (18, 10)
135805188216512 opening file: (18, 8)
135805188216512 opening file: (18, 4)
135805188216512 opening file: (18, 72)
135805188216512 opening file: (18, 73)
135805188216512 opening file: (18, 74)
135805188216512 removing: (18, 74)
135805188216512 opening file: (18, 47)
135805188216512 removing: (18, 47)
135805188216512 opening file: (18, 82)
135805188216512 removing: (18, 82)
135805188216512 opening file: (18, 83)
135805188216512 removing: (18, 83)
135805188216512 opening file: (18, 21)
135805188216512 removing: (18, 21)
135805188216512 opening file: (18, 48)
135805188216512 removing: (18, 48)
135805188216512 opening file: (18, 61)
135805188216512 removing: (18, 61)
135805188216512 opening file: (18, 21)
135805188216512 removing: (18, 21)
135805188216512 opening file: (27, 0)

Obviously, this is not how it intended to work, and actually could lead to severe performance degradation.

I assume this thing does not sort files as expected.

libtorrent/include/libtorrent/aux_/file_view_pool.hpp

Lines 165 to 166 in 6a2c1ee

	// look up files by least recently used
	mi::sequenced<>

Replacing backs with fronts inside the file_view_pool::remove_oldest function seems to mitigate the issue.

--- a/src/file_view_pool.cpp
+++ b/src/file_view_pool.cpp
@@ -351,11 +351,11 @@ namespace libtorrent { namespace aux {
 
 #if TRACE_FILE_VIEW_POOL
 		std::cout << std::this_thread::get_id() << " removing: ("
-			<< lru_view.back().key.first << ", " << lru_view.back().key.second << ")\n";
+			<< lru_view.front().key.first << ", " << lru_view.front().key.second << ")\n";
 #endif
 
-		auto mapping = std::move(lru_view.back().mapping);
-		lru_view.pop_back();
+		auto mapping = std::move(lru_view.front().mapping);
+		lru_view.pop_front();
 
 		// closing a file may be long running operation (mac os x)
 		// let the caller destruct it once it has released the mutex

I don't know how vaild this fix is, but it does make the pool to actually rotate.

125017016645312 opening file: (3, 371)
125017016645312 opening file: (3, 660)
125017016645312 opening file: (3, 129)
125017016645312 opening file: (8, 258)
125017016645312 opening file: (3, 647)
125017016645312 opening file: (22, 0)
125017016645312 opening file: (3, 474)
125017016645312 opening file: (3, 640)
125017016645312 opening file: (3, 304)
125017016645312 removing: (3, 640)
125017016645312 opening file: (3, 534)
125017016645312 removing: (3, 371)
125017016645312 opening file: (3, 556)
125017016645312 removing: (3, 534)
125017016645312 opening file: (3, 495)
125017016645312 removing: (3, 660)
125017016645312 opening file: (3, 603)
125017016645312 removing: (3, 474)
125017016645312 opening file: (3, 343)
125017016645312 removing: (3, 556)
125017016645312 opening file: (3, 639)
125017016645312 removing: (3, 129)
125017016645312 opening file: (3, 544)
125017016645312 removing: (3, 639)
125017016645312 opening file: (3, 375)
125017016645312 removing: (3, 343)
125017016645312 opening file: (3, 77)
125017016645312 removing: (3, 647)

[arvidn]

Thanks for an excellent report. I will investigate!

[arvidn]

addressed here: #7779

[KyleSanderson]

I'm guessing this doesn't impact lt1.2? There's some weird fluttering going on, I haven't traced it yet though

[HanabishiRecca]

No, 1.2 has completely different implementation.

[Toms200]

I get strange behaviour of torrents not uploading after being active for an extended period. I uploaded 500GB from 416 torrents between 24-48 hours, and many torrents became a standstill. I upgraded the version from 5.0.2 to 5.0.3 of qBittorrent and with the fresh start active torrents went from a few to around 15.

FIY: DC++ has a similar problem of uploads becoming a standstill.