CANNOT_READ_ALL_DATA with enabled uncompressed cache and direct io (00024_random_counters)

[CurtizJ]

Describe the bug
A link to the report

How to reproduce
Run test 00024_random_counters.sql with settings --use_uncompressed_cache=1 --min_bytes_to_use_direct_io=1

clickhouse-client -q "system drop uncompressed cache" && cat queries/1_stateful/00024_random_counters.sql | clickhouse-client -n --use_uncompressed_cache=1 --min_bytes_to_use_direct_io=1 > /dev/null
Received exception from server (version 22.7.1):
Code: 33. DB::Exception: Received from localhost:9000. DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 25.: (while reading column Sign): (while reading from part ./data/datasets/visits_v1/20140317_20140323_3_4_1/ from mark 204 with max_rows_to_read = 8192): While executing MergeTreeInOrder. (CANNOT_READ_ALL_DATA)
(query: SELECT uniq(UserID), sum(Sign) FROM datasets.visits_v1 WHERE CounterID = 33436573;)

[tavplubix]

Probably related:

2022.07.21 23:06:01.288886 [ 2880 ] {} <Fatal> BaseDaemon: ########################################
2022.07.21 23:06:01.289194 [ 2880 ] {} <Fatal> BaseDaemon: (version 22.7.1.1, build id: 88CE37A8C0C1997E) (from thread 1013) (query_id: 4b041d0d-c1ee-4edc-a85e-09dda8deb3c6) (query: SELECT uniq(UserID), sum(Sign) FROM test.visits WHERE CounterID = 33436573;) Received signal Aborted (6)
2022.07.21 23:06:01.289399 [ 2880 ] {} <Fatal> BaseDaemon: 
2022.07.21 23:06:01.289616 [ 2880 ] {} <Fatal> BaseDaemon: Stack trace: 0x7f8095cae00b 0x7f8095c8d859 0x7f8095c8d729 0x7f8095c9efd6 0x170f8bae 0x25d5a712 0x2805a470 0x2805a192 0x28059b01 0x280596bd 0x170f8afc 0x170fc613 0x170fc0a5 0x25fdb97a 0x25f88bf2 0x28068e90 0x28068471 0x28b3566c 0x28b35943 0x28b36485 0x28b3aa11 0x28b39900 0x28b217aa 0x28b221d1 0x28b21045 0x284696f0 0x284693eb 0x284a35e3 0x284a32a3 0x284865c1 0x284868f7 0x28485a23 0x28484f98 0x284aa78a 0x284aa660 0x284aa5f5 0x284aa5a1 0x284aa4b2 0x284aa3ba 0x284aa2f5 0x284aa2bd
2022.07.21 23:06:01.289867 [ 2880 ] {} <Fatal> BaseDaemon: 4. gsignal @ 0x7f8095cae00b in ?
2022.07.21 23:06:01.290055 [ 2880 ] {} <Fatal> BaseDaemon: 5. abort @ 0x7f8095c8d859 in ?
2022.07.21 23:06:01.290211 [ 2880 ] {} <Fatal> BaseDaemon: 6. ? @ 0x7f8095c8d729 in ?
2022.07.21 23:06:01.290402 [ 2880 ] {} <Fatal> BaseDaemon: 7. ? @ 0x7f8095c9efd6 in ?
2022.07.21 23:06:01.394509 [ 2880 ] {} <Fatal> BaseDaemon: 8. /build/build_docker/../src/IO/ReadBuffer.h:0: DB::ReadBuffer::next() @ 0x170f8bae in /usr/bin/clickhouse
2022.07.21 23:06:01.469063 [ 2880 ] {} <Fatal> BaseDaemon: 9.1. inlined from /build/build_docker/../src/IO/ReadBuffer.h:98: DB::ReadBuffer::eof()
2022.07.21 23:06:01.469290 [ 2880 ] {} <Fatal> BaseDaemon: 9. ../src/Compression/CompressedReadBufferBase.cpp:114: DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&, bool) @ 0x25d5a712 in /usr/bin/clickhouse
2022.07.21 23:06:01.596266 [ 2880 ] {} <Fatal> BaseDaemon: 10. /build/build_docker/../src/Compression/CachedCompressedReadBuffer.cpp:52: DB::CachedCompressedReadBuffer::nextImpl()::$_0::operator()() const @ 0x2805a470 in /usr/bin/clickhouse
2022.07.21 23:06:01.722075 [ 2880 ] {} <Fatal> BaseDaemon: 11. /build/build_docker/../src/Common/LRUCache.h:123: std::__1::pair<std::__1::shared_ptr<DB::UncompressedCacheCell>, bool> DB::LRUCache<wide::integer<128ul, unsigned int>, DB::UncompressedCacheCell, UInt128TrivialHash, DB::UncompressedSizeWeightFunction>::getOrSet<DB::CachedCompressedReadBuffer::nextImpl()::$_0>(wide::integer<128ul, unsigned int> const&, DB::CachedCompressedReadBuffer::nextImpl()::$_0&&) @ 0x2805a192 in /usr/bin/clickhouse
2022.07.21 23:06:01.845281 [ 2880 ] {} <Fatal> BaseDaemon: 12. /build/build_docker/../src/IO/UncompressedCache.h:66: std::__1::shared_ptr<DB::UncompressedCacheCell> DB::UncompressedCache::getOrSet<DB::CachedCompressedReadBuffer::nextImpl()::$_0>(wide::integer<128ul, unsigned int> const&, DB::CachedCompressedReadBuffer::nextImpl()::$_0&&) @ 0x28059b01 in /usr/bin/clickhouse
2022.07.21 23:06:01.967579 [ 2880 ] {} <Fatal> BaseDaemon: 13. /build/build_docker/../src/Compression/CachedCompressedReadBuffer.cpp:43: DB::CachedCompressedReadBuffer::nextImpl() @ 0x280596bd in /usr/bin/clickhouse
2022.07.21 23:06:02.073178 [ 2880 ] {} <Fatal> BaseDaemon: 14. /build/build_docker/../src/IO/ReadBuffer.h:64: DB::ReadBuffer::next() @ 0x170f8afc in /usr/bin/clickhouse
2022.07.21 23:06:02.181074 [ 2880 ] {} <Fatal> BaseDaemon: 15.1. inlined from /build/build_docker/../src/IO/ReadBuffer.h:98: DB::ReadBuffer::eof()
2022.07.21 23:06:02.181278 [ 2880 ] {} <Fatal> BaseDaemon: 15. ../src/IO/ReadBuffer.h:175: DB::ReadBuffer::read(char*, unsigned long) @ 0x170fc613 in /usr/bin/clickhouse
2022.07.21 23:06:02.288637 [ 2880 ] {} <Fatal> BaseDaemon: 16. /build/build_docker/../src/IO/ReadBuffer.h:202: DB::ReadBuffer::readBig(char*, unsigned long) @ 0x170fc0a5 in /usr/bin/clickhouse
2022.07.21 23:06:02.428265 [ 2880 ] {} <Fatal> BaseDaemon: 17. /build/build_docker/../src/DataTypes/Serializations/SerializationNumber.cpp:154: DB::SerializationNumber<signed char>::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const @ 0x25fdb97a in /usr/bin/clickhouse
2022.07.21 23:06:02.529276 [ 2880 ] {} <Fatal> BaseDaemon: 18. /build/build_docker/../src/DataTypes/Serializations/ISerialization.cpp:132: DB::ISerialization::deserializeBinaryBulkWithMultipleStreams(COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, DB::ISerialization::DeserializeBinaryBulkSettings&, std::__1::shared_ptr<DB::ISerialization::DeserializeBinaryBulkState>&, std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, COW<DB::IColumn>::immutable_ptr<DB::IColumn> > > >*) const @ 0x25f88bf2 in /usr/bin/clickhouse
2022.07.21 23:06:02.684427 [ 2880 ] {} <Fatal> BaseDaemon: 19. /build/build_docker/../src/Storages/MergeTree/MergeTreeReaderWide.cpp:299: DB::MergeTreeReaderWide::readData(DB::NameAndTypePair const&, COW<DB::IColumn>::immutable_ptr<DB::IColumn>&, unsigned long, bool, unsigned long, unsigned long, std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, COW<DB::IColumn>::immutable_ptr<DB::IColumn> > > >&, bool) @ 0x28068e90 in /usr/bin/clickhouse
2022.07.21 23:06:02.848332 [ 2880 ] {} <Fatal> BaseDaemon: 20. /build/build_docker/../src/Storages/MergeTree/MergeTreeReaderWide.cpp:122: DB::MergeTreeReaderWide::readRows(unsigned long, unsigned long, bool, unsigned long, std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn> > >&) @ 0x28068471 in /usr/bin/clickhouse
2022.07.21 23:06:03.048171 [ 2880 ] {} <Fatal> BaseDaemon: 21. /build/build_docker/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:100: DB::MergeTreeRangeReader::DelayedStream::readRows(std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn> > >&, unsigned long) @ 0x28b3566c in /usr/bin/clickhouse
2022.07.21 23:06:03.246992 [ 2880 ] {} <Fatal> BaseDaemon: 22. /build/build_docker/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:174: DB::MergeTreeRangeReader::DelayedStream::finalize(std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn> > >&) @ 0x28b35943 in /usr/bin/clickhouse
2022.07.21 23:06:03.446409 [ 2880 ] {} <Fatal> BaseDaemon: 23. /build/build_docker/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:285: DB::MergeTreeRangeReader::Stream::finalize(std::__1::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn> > >&) @ 0x28b36485 in /usr/bin/clickhouse
2022.07.21 23:06:03.649202 [ 2880 ] {} <Fatal> BaseDaemon: 24. /build/build_docker/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:943: DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::__1::deque<DB::MarkRange, std::__1::allocator<DB::MarkRange> >&) @ 0x28b3aa11 in /usr/bin/clickhouse
2022.07.21 23:06:03.851274 [ 2880 ] {} <Fatal> BaseDaemon: 25. /build/build_docker/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:852: DB::MergeTreeRangeReader::read(unsigned long, std::__1::deque<DB::MarkRange, std::__1::allocator<DB::MarkRange> >&) @ 0x28b39900 in /usr/bin/clickhouse
2022.07.21 23:06:04.162523 [ 2880 ] {} <Fatal> BaseDaemon: 26. /build/build_docker/../src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:310: DB::MergeTreeBaseSelectProcessor::readFromPartImpl() @ 0x28b217aa in /usr/bin/clickhouse
2022.07.21 23:06:04.449223 [ 2880 ] {} <Fatal> BaseDaemon: 27. /build/build_docker/../src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:355: DB::MergeTreeBaseSelectProcessor::readFromPart() @ 0x28b221d1 in /usr/bin/clickhouse
2022.07.21 23:06:04.734244 [ 2880 ] {} <Fatal> BaseDaemon: 28. /build/build_docker/../src/Storages/MergeTree/MergeTreeBaseSelectProcessor.cpp:205: DB::MergeTreeBaseSelectProcessor::generate() @ 0x28b21045 in /usr/bin/clickhouse
2022.07.21 23:06:04.874669 [ 2880 ] {} <Fatal> BaseDaemon: 29. /build/build_docker/../src/Processors/ISource.cpp:124: DB::ISource::tryGenerate() @ 0x284696f0 in /usr/bin/clickhouse
2022.07.21 23:06:05.014618 [ 2880 ] {} <Fatal> BaseDaemon: 30. /build/build_docker/../src/Processors/ISource.cpp:94: DB::ISource::work() @ 0x284693eb in /usr/bin/clickhouse
2022.07.21 23:06:05.118137 [ 2880 ] {} <Fatal> BaseDaemon: 31. /build/build_docker/../src/Processors/Executors/ExecutionThreadContext.cpp:47: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x284a35e3 in /usr/bin/clickhouse
2022.07.21 23:06:05.216311 [ 2880 ] {} <Fatal> BaseDaemon: 32. /build/build_docker/../src/Processors/Executors/ExecutionThreadContext.cpp:92: DB::ExecutionThreadContext::executeTask() @ 0x284a32a3 in /usr/bin/clickhouse
2022.07.21 23:06:05.382137 [ 2880 ] {} <Fatal> BaseDaemon: 33. /build/build_docker/../src/Processors/Executors/PipelineExecutor.cpp:224: DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) @ 0x284865c1 in /usr/bin/clickhouse
2022.07.21 23:06:05.547946 [ 2880 ] {} <Fatal> BaseDaemon: 34. /build/build_docker/../src/Processors/Executors/PipelineExecutor.cpp:189: DB::PipelineExecutor::executeSingleThread(unsigned long) @ 0x284868f7 in /usr/bin/clickhouse
2022.07.21 23:06:05.690586 [ 2880 ] {} <Fatal> BaseDaemon: 35. /build/build_docker/../src/Processors/Executors/PipelineExecutor.cpp:333: DB::PipelineExecutor::executeImpl(unsigned long) @ 0x28485a23 in /usr/bin/clickhouse
2022.07.21 23:06:05.854283 [ 2880 ] {} <Fatal> BaseDaemon: 36. /build/build_docker/../src/Processors/Executors/PipelineExecutor.cpp:90: DB::PipelineExecutor::execute(unsigned long) @ 0x28484f98 in /usr/bin/clickhouse
2022.07.21 23:06:06.013879 [ 2880 ] {} <Fatal> BaseDaemon: 37. /build/build_docker/../src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:79: DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long) @ 0x284aa78a in /usr/bin/clickhouse
2022.07.21 23:06:06.171215 [ 2880 ] {} <Fatal> BaseDaemon: 38. /build/build_docker/../src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:108: DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const @ 0x284aa660 in /usr/bin/clickhouse
2022.07.21 23:06:06.330359 [ 2880 ] {} <Fatal> BaseDaemon: 39. /build/build_docker/../contrib/libcxx/include/type_traits:3648: decltype(static_cast<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&>(fp)()) std::__1::__invoke_constexpr<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) @ 0x284aa5f5 in /usr/bin/clickhouse
2022.07.21 23:06:06.489204 [ 2880 ] {} <Fatal> BaseDaemon: 40. /build/build_docker/../contrib/libcxx/include/tuple:1595: decltype(auto) std::__1::__apply_tuple_impl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) @ 0x284aa5a1 in /usr/bin/clickhouse
2022.07.21 23:06:06.647854 [ 2880 ] {} <Fatal> BaseDaemon: 41. /build/build_docker/../contrib/libcxx/include/tuple:1604: decltype(auto) std::__1::apply<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&) @ 0x284aa4b2 in /usr/bin/clickhouse
2022.07.21 23:06:06.788959 [ 2880 ] {} <Fatal> BaseDaemon: 42. /build/build_docker/../src/Common/ThreadPool.h:187: ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() @ 0x284aa3ba in /usr/bin/clickhouse
2022.07.21 23:06:06.948245 [ 2880 ] {} <Fatal> BaseDaemon: 43. /build/build_docker/../contrib/libcxx/include/type_traits:3640: decltype(static_cast<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(fp)()) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) @ 0x284aa2f5 in /usr/bin/clickhouse
2022.07.21 23:06:07.105611 [ 2880 ] {} <Fatal> BaseDaemon: 44. /build/build_docker/../contrib/libcxx/include/__functional/invoke.h:62: void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) @ 0x284aa2bd in /usr/bin/clickhouse
2022.07.21 23:06:08.415615 [ 2880 ] {} <Fatal> BaseDaemon: Integrity check of the executable skipped because the reference checksum could not be read. (calculated checksum: 1B3C5FC35FA4A1CC62C0D573F6425A89)
2022.07.21 23:06:21.689675 [ 643 ] {} <Fatal> Application: Child process was terminated by signal 6.

https://s3.amazonaws.com/clickhouse-test-reports/39462/821f006a7ada6742ab2f3514b294a83af72f7165/stateful_tests__debug__actions_.html