Panicked when create table

[MichaelLeeHZ]

Describe this problem

Here is the stack

2023-01-11 11:01:11.797 ERRO [server/src/http.rs:129] Http service Failed to handle sql, err:Failed to create plan, query:DESCRIBE MMM_2197135594_INFLUENCE_GRAY_SANDBOX_OUTPUT_TABLE, err:Failed to create plan, err:Table not found, table:MMM_2197135594_INFLUENCE_GRAY_SANDBOX_OUTPU
T_TABLE
2023-01-11 11:01:11.798 ERRO [common_util/src/panic.rs:42] thread 'ceres-write' panicked 'assertion failed: success' at "analytic_engine/src/space.rs:147"
   0: common_util::panic::set_panic_hook::{{closure}}
             at /home/chunshao.rcs/github/CeresDB/ceresdb/common_util/src/panic.rs:41:18
   1: std::panicking::rust_panic_with_hook
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/std/src/panicking.rs:702:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/std/src/panicking.rs:586:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/std/src/sys_common/backtrace.rs:138:18
   4: rust_begin_unwind
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/std/src/panicking.rs:584:5
   5: core::panicking::panic_fmt
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/panicking.rs:142:14
   6: core::panicking::panic
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/panicking.rs:48:5
   7: analytic_engine::space::Space::insert_table
             at /home/chunshao.rcs/github/CeresDB/ceresdb/analytic_engine/src/space.rs:147:9
   8: analytic_engine::instance::create::<impl analytic_engine::instance::Instance>::process_create_table_command::{{closure}}
             at /home/chunshao.rcs/github/CeresDB/ceresdb/analytic_engine/src/instance/create.rs:115:9
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/future/mod.rs:91:19
      analytic_engine::instance::write_worker::WriteWorker::handle_create_table::{{closure}}
             at /home/chunshao.rcs/github/CeresDB/ceresdb/analytic_engine/src/instance/write_worker.rs:865:78
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/future/mod.rs:91:19
      analytic_engine::instance::write_worker::WriteWorker::run::{{closure}}
             at /home/chunshao.rcs/github/CeresDB/ceresdb/analytic_engine/src/instance/write_worker.rs:753:50
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/future/mod.rs:91:19
      analytic_engine::instance::write_worker::WriteGroup::new::{{closure}}
             at /home/chunshao.rcs/github/CeresDB/ceresdb/analytic_engine/src/instance/write_worker.rs:616:29
      <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/future/mod.rs:91:19
  10: tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
             at /home/chunshao.rcs/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.20.1/src/runtime/task/core.rs:165:17
      tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/chunshao.rcs/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.20.1/src/loom/std/unsafe_cell.rs:14:9
      tokio::runtime::task::core::CoreStage<T>::poll
             at /home/chunshao.rcs/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.20.1/src/runtime/task/core.rs:155:13
      tokio::runtime::task::harness::poll_future::{{closure}}
             at /home/chunshao.rcs/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.20.1/src/runtime/task/harness.rs:480:19
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/std/src/panicking.rs:492:40
      std::panicking::try
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/std/src/panicking.rs:456:19
  11: std::panic::catch_unwind
             at /rustc/d394408fb38c4de61f765a3ed5189d2731a1da91/library/std/src/panic.rs:137:14
      tokio::runtime::task::harness::poll_future
             at /home/chunshao.rcs/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.20.1/src/runtime/task/harness.rs:468:18
      tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /home/chunshao.rcs/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/tokio-1.20.1/src/runtime/task/harness.rs:104:27
      tokio::runtime::task::harness::Harness<T,S>::poll

Steps to reproduce

I guess the table had been created and dropped for many times, then panicked when creating the table

Expected behavior

Create table successfully

Additional Information

None

[ShiKaiWi]

@MachaelLee Maybe the commit id of the binary helps troubleshoot because the code location is wrong in the latest main branch.

[jiacai2050]

2023-03-14T02:31:10.846583855Z 2023-03-14 02:31:10.846 ERRO [common_util/src/panic.rs:42] thread 'ceres-write' panicked 'assertion failed: success' at "analytic_engine/src/space.rs:159"
2023-03-14T02:31:10.846590798Z    0: common_util::panic::set_panic_hook::{{closure}}
2023-03-14T02:31:10.846598388Z              at ceresdb/common_util/src/panic.rs:41:18
2023-03-14T02:31:10.846603760Z    1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
2023-03-14T02:31:10.846610344Z              at rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/alloc/src/boxed.rs:2002:9
2023-03-14T02:31:10.846638988Z       std::panicking::rust_panic_with_hook
2023-03-14T02:31:10.846646204Z              at rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:692:13
2023-03-14T02:31:10.846651435Z    2: std::panicking::begin_panic_handler::{{closure}}
2023-03-14T02:31:10.846656365Z              at rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:577:13
2023-03-14T02:31:10.846661186Z    3: std::sys_common::backtrace::__rust_end_short_backtrace
2023-03-14T02:31:10.846669922Z              at rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/sys_common/backtrace.rs:137:18
2023-03-14T02:31:10.846675231Z    4: rust_begin_unwind
2023-03-14T02:31:10.846679676Z              at rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/std/src/panicking.rs:575:5
2023-03-14T02:31:10.846683169Z    5: core::panicking::panic_fmt
2023-03-14T02:31:10.846689518Z              at rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/panicking.rs:64:14
2023-03-14T02:31:10.846692993Z    6: core::panicking::panic
2023-03-14T02:31:10.846695905Z              at rustc/11d96b59307b1702fffe871bfc2d0145d070881e/library/core/src/panicking.rs:114:5
2023-03-14T02:31:10.846698880Z    7: analytic_engine::space::Space::insert_table
2023-03-14T02:31:10.846701684Z              at ceresdb/analytic_engine/src/space.rs:159:9
2023-03-14T02:31:10.846704965Z    8: analytic_engine::instance::open::<impl analytic_engine::instance::Instance>::process_recover_table_command::{{closure}}
2023-03-14T02:31:10.846708301Z              at ceresdb/analytic_engine/src/instance/open.rs:200:9
2023-03-14T02:31:10.846711635Z       analytic_engine::instance::write_worker::WriteWorker::handle_recover_table::{{closure}}
2023-03-14T02:31:10.846714672Z              at ceresdb/analytic_engine/src/instance/write_worker.rs:828:13
2023-03-14T02:31:10.846717503Z       analytic_engine::instance::write_worker::WriteWorker::run::{{closure}}
2023-03-14T02:31:10.846720325Z              at ceresdb/analytic_engine/src/instance/write_worker.rs:762:51
2023-03-14T02:31:10.846723184Z    9: analytic_engine::instance::write_worker::WriteGroup::new::{{closure}}
2023-03-14T02:31:10.846725951Z              at ceresdb/analytic_engine/src/instance/write_worker.rs:619:29
2023-03-14T02:31:10.846729072Z   10: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}

A similar failed case, I guess this is the steps cause this issue:

1. Restart server
2. Client insert table A with auto create table

Step 2 happens before server open table, then when step 1 open table A, it will panic at assert

    /// Insert table data into space memory state if the table is
    /// absent. For internal use only
    ///
    /// Panic if the table has already existed.
    pub(crate) fn insert_table(&self, table_data: TableDataRef) {
        let success = self
            .table_datas
            .write()
            .unwrap()
            .insert_if_absent(table_data);
        assert!(success);
    }

A quickfix is to disable write/read request before server open all shards.

ceresdb-server --version
CeresDB Server 
CeresDB version: 1.0.0
Git branch: main
Git commit: 95ea870
Build time: 2023-03-04T20:49:54.88119768Z
Rustc version: 1.69.0-nightly