Intermittent failure in test rocksdb.rocksdb

[jtolmer]

Alexey suggested that I assign this issue to you. We're seeing occasional failures of this test in our CI testing.

rocksdb.rocksdb w9 [ fail ]
Test ended at 2015-09-16 21:23:21

CURRENT_TEST: rocksdb.rocksdb
--- /data/users/jenkins/workspace/github-mysql-nightly/BUILD_TYPE/ASan/CLIENT_MODE/Async/PAGE_SIZE/32/TEST_SET/MixedOtherBig/label/mysql/mysql/mysql-test/suite/rocksdb/r/rocksdb.result 2015-09-17 06:20:07.937508115 +0300
+++ /data/users/jenkins/workspace/github-mysql-nightly/BUILD_TYPE/ASan/CLIENT_MODE/Async/PAGE_SIZE/32/TEST_SET/MixedOtherBig/label/mysql/mysql/_build-5.6-ASan/mysql-test/var/9/log/rocksdb.reject 2015-09-17 07:23:20.339521073 +0300
@@ -1731,7 +1731,7 @@

The following must return true (before the fix, the difference was 70):

select if((@var2 - @var1) < 30, 1, @var2-@var1);
if((@var2 - @var1) < 30, 1, @var2-@var1)
-1
+93
drop table t0,t1;

Issue #33: SELECT ... FROM rocksdb_table ORDER BY primary_key uses sorting

mysqltest: Result content mismatch

[jtolmer]

This test is currently disabled in our CI system because of these failures. It needs to be fixed.

[maykov]

Sergey, any update on this?

[yoshinorim]

I spent some time to look into this issue. rocksdb_number_keys_read was also increased by drop_index_thread. Stack trace was like below. MyRocks's drop_index thread calls Get() to get CF info for all indexes that are ongoing for dropping. The current problem of MyRocks is dropped indexes are not really dropped, so whenever executing DROP TABLE, the target drop index list increases, then drop_index_thread calls Get() many more times. Then rocksdb_number_keys_read is also increased a lot which caused rocksdb.rocksdb test failure.

I think the real fix is making drop table work -- then the drop index list will be kept short enough, so it won't cause noise for rocksdb.rocksdb test.

(gdb) bt
#0  0x0000000000c77290 in rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::string*, bool*) (this=0x7f320174d800, read_options=..., column_family=0x7f32016c5300, key=..., value=0x7f31ff5fdb20, value_found=value_found@entry=0x0) at /home/yoshinori/local/mysql/5.6/rocksdb/db/db_impl.cc:3076
#1  0x0000000000c77bc9 in rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::string*) (this=<optimized out>, read_options=..., column_family=<optimized out>, key=..., value=<optimized out>) at /home/yoshinori/local/mysql/5.6/rocksdb/db/db_impl.cc:3019
#2  0x0000000000d72b46 in rocksdb::StackableDB::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, std::string*) (this=<optimized out>, options=..., column_family=<optimized out>, key=..., value=<optimized out>) at /home/yoshinori/local/mysql/5.6/rocksdb/include/rocksdb/utilities/stackable_db.h:54
#3  0x0000000000c50338 in Dict_manager::Get(rocksdb::Slice const&, std::string*) (this=this@entry=0x1768de0 <dict_manager>, key=..., value=value@entry=0x7f31ff5fdb20) at /home/yoshinori/local/mysql/5.6/storage/rocksdb/rdb_datadic.cc:2288
#4  0x0000000000c50ae1 in Dict_manager::get_cf_flags(unsigned int, unsigned int*) (this=this@entry=0x1768de0 <dict_manager>, cf_id=cf_id@entry=0, cf_flags=cf_flags@entry=0x7f31ff5fdbe0) at /home/yoshinori/local/mysql/5.6/storage/rocksdb/rdb_datadic.cc:2435
#5  0x0000000000c3bf16 in drop_index_thread(void*) () at /home/yoshinori/local/mysql/5.6/storage/rocksdb/ha_rocksdb.cc:5884
#6  0x00007f3204371fa8 in start_thread 

[yoshinorim]

@jtolmer : DROP TABLE fix was committed, and after that I have not seen rocksdb.rocksdb test flakiness anymore (ran mtr --suite=rocksdb rocksdb --repeat=50). Do you think it's ok to close this issue?

[jtolmer]

I've re-enabled the test in our CI system. We can re-open if it hits any problems.