Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read Doris data through spark error,when date column is null #2138

Closed
zhangy5 opened this issue Nov 6, 2019 · 2 comments
Closed

Read Doris data through spark error,when date column is null #2138

zhangy5 opened this issue Nov 6, 2019 · 2 comments
Labels
kind/fix Categorizes issue or PR as related to a bug.
Milestone

Comments

@zhangy5
Copy link
Contributor

zhangy5 commented Nov 6, 2019

Describe the bug
Read Doris data through spark error,when date column is null. BE crashed.
#0 0x00007fa4392603f7 in raise () from /opt/compiler/gcc-4.8.2/lib64/libc.so.6
#1 0x00007fa4392617d8 in abort () from /opt/compiler/gcc-4.8.2/lib64/libc.so.6
#2 0x0000000002ddc715 in __gnu_cxx::__verbose_terminate_handler () at ../../../../gcc-7.3.0/libstdc++-v3/libsupc++/vterminate.cc:95
#3 0x0000000002d42586 in __cxxabiv1::__terminate (handler=) at ../../../../gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4 0x0000000002d425d1 in std::terminate () at ../../../../gcc-7.3.0/libstdc++-v3/libsupc++/eh_terminate.cc:57
#5 0x0000000002d424d3 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x8bbd640,
tinfo=tinfo@entry=0x2f17108 <typeinfo for boost::exception_detail::clone_impl<boost::exception_detail::error_info_injectorboost::gregorian::bad_day_of_month >>,
dest=dest@entry=0xde3460 <boost::exception_detail::clone_impl<boost::exception_detail::error_info_injectorboost::gregorian::bad_day_of_month >::~clone_impl()>) at ../../../../gcc-7.3.0/libstdc++-v3/libsupc++/eh_throw.cc:93
#6 0x0000000000de582e in boost::throw_exceptionboost::gregorian::bad_day_of_month (e=...) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/throw_exception.hpp:69
#7 0x0000000000de65ab in boost::CV::simple_exception_policy<unsigned short, (unsigned short)1, (unsigned short)31, boost::gregorian::bad_day_of_month>::on_error ()
at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/date_time/constrained_value.hpp:110
#8 0x0000000000f9c299 in boost::CV::constrained_value<boost::CV::simple_exception_policy<unsigned short, (unsigned short)1, (unsigned short)31, boost::gregorian::bad_day_of_month> >::assign (value=0, this=)
at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/date_time/constrained_value.hpp:69
#9 boost::CV::constrained_value<boost::CV::simple_exception_policy<unsigned short, (unsigned short)1, (unsigned short)31, boost::gregorian::bad_day_of_month> >::constrained_value (value=0, this=)
at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/date_time/constrained_value.hpp:48
#10 boost::gregorian::greg_day::greg_day (day_of_month=0, this=) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/date_time/gregorian/greg_day.hpp:45
#11 boost::date_time::parse_dateboost::gregorian::date (s=..., order_spec=order_spec@entry=0) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/date_time/date_parsing.hpp:153
#12 0x0000000000f99aef in boost::date_time::parse_delimited_timeboost::posix_time::ptime (sep=32 ' ', s=...) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/date_time/time_parsing.hpp:177
#13 boost::posix_time::time_from_string (s=...) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/date_time/posix_time/time_parsers.hpp:31
#14 doris::DateTimeValue::unix_timestamp (this=, timestamp=timestamp@entry=0x7fa3e9367880, timezone=...) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/core/be/src/runtime/datetime_value.cpp:1534
#15 0x00000000010b41fd in doris::FromRowBatchConverter::Visit (this=this@entry=0x7fa3e9367a30, type=...) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/core/be/src/util/arrow/row_batch.cpp:273
#16 0x00000000010acdf4 in arrow::VisitTypeInlinedoris::FromRowBatchConverter (visitor=0x7fa3e9367a30, type=...) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/arrow/visitor_inline.h:71
#17 doris::FromRowBatchConverter::convert (out=0x7fa3e9367bb0, this=0x7fa3e9367a30) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/core/be/src/util/arrow/row_batch.cpp:359
#18 doris::convert_to_arrow_batch (batch=..., schema=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<arrow::Schema, std::allocatorarrow::Schema, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<arrow::Schema, std::allocatorarrow::Schema, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 1, weak 0) 0x8ad3b90, pool=, result=result@entry=0x7fa3e9367bb0)
at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/core/be/src/util/arrow/row_batch.cpp:374
#19 0x0000000001640f8a in doris::MemoryScratchSink::send (this=0x8ad2480, state=, batch=0xd737080) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/core/be/src/runtime/memory_scratch_sink.cpp:80
#20 0x0000000001062ca4 in doris::PlanFragmentExecutor::open_internal (this=this@entry=0xa2230f0) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/core/be/src/runtime/plan_fragment_executor.cpp:334
#21 0x0000000001063104 in doris::PlanFragmentExecutor::open (this=this@entry=0xa2230f0) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/core/be/src/runtime/plan_fragment_executor.cpp:284
#22 0x0000000000ff07a7 in doris::FragmentExecState::execute (this=0xa223080) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/core/be/src/runtime/fragment_mgr.cpp:209
#23 0x0000000000ff23f6 in doris::FragmentMgr::exec_actual(std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)>) (this=0x5581c00, exec_state=std::shared_ptr (count 4, weak 0) 0xa223080, cb=...)
at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/core/be/src/runtime/fragment_mgr.cpp:388
#24 0x0000000000ff8854 in boost::_mfi::mf2<void, doris::FragmentMgr, std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)> >::operator()(doris::FragmentMgr*, std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)>) const (a2=..., a1=..., p=, this=)
at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/bind/mem_fn_template.hpp:280
#25 boost::_bi::list3<boost::_bi::valuedoris::FragmentMgr*, boost::_bi::value<std::shared_ptrdoris::FragmentExecState >, boost::_bi::value<std::function<void (doris::PlanFragmentExecutor*)> > >::operator()<boost::_mfi::mf2<void, doris::FragmentMgr, std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)> >, boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf2<void, doris::FragmentMgr, std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)> >&, boost::_bi::list0&, int) (a=..., f=..., this=)
at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/bind/bind.hpp:398
#26 boost::_bi::bind_t<void, boost::_mfi::mf2<void, doris::FragmentMgr, std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)> >, boost::_bi::list3<boost::_bi::valuedoris::FragmentMgr*, boost::_bi::value<std::shared_ptrdoris::FragmentExecState >, boost::_bi::value<std::function<void (doris::PlanFragmentExecutor*)> > > >::operator()() (this=)
at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/bind/bind.hpp:1294
#27 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, doris::FragmentMgr, std::shared_ptrdoris::FragmentExecState, std::function<void (doris::PlanFragmentExecutor*)> >, boost::_bi::list3<boost::_bi::valuedoris::FragmentMgr*, boost::_bi::value<std::shared_ptrdoris::FragmentExecState >, boost::_bi::value<std::function<void (doris::PlanFragmentExecutor*)> > > >, void>::invoke(boost::detail::function::function_buffer&) (function_obj_ptr=...) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/function/function_template.hpp:159
#28 0x0000000000eec9b8 in boost::function0::operator() (this=0x7fa3e9368000) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/thirdparty/installed/include/boost/function/function_template.hpp:759
#29 doris::ThreadPool::work_thread (this=0x5581c80, thread_id=) at /home/palo-ci/CI/workspace/palo-compile/doris-on-es/core/be/src/util/thread_pool.hpp:120
#30 0x0000000001a3093d in thread_proxy ()
#31 0x00007fa4390151c3 in start_thread () from /opt/compiler/gcc-4.8.2/lib64/libpthread.so.0
#32 0x00007fa43931212d in clone () from /opt/compiler/gcc-4.8.2/lib64/libc.so.6

To Reproduce
Execute the sql as follows in spark
select * from baseall where k1 is null;

Expected behavior
expected result is:
mysql> select * from baseall where k1 is null;
+------+------+------+------+------+------+------+------+------+------+------+
| k1 | k2 | k3 | k4 | k5 | k6 | k10 | k11 | k7 | k8 | k9 |
+------+------+------+------+------+------+------+------+------+------+------+
| NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL | NULL |
+------+------+------+------+------+------+------+------+------+------+------+
1 row in set (0.05 sec)

baseall table is :
CREATE TABLE baseall (
k1 tinyint(4) NULL COMMENT "",
k2 smallint(6) NULL COMMENT "",
k3 int(11) NULL COMMENT "",
k4 bigint(20) NULL COMMENT "",
k5 decimal(9, 3) NULL COMMENT "",
k6 char(5) NULL COMMENT "",
k10 date NULL COMMENT "",
k11 datetime NULL COMMENT "",
k7 varchar(20) NULL COMMENT "",
k8 double MAX NULL COMMENT "",
k9 float SUM NULL COMMENT ""
) ENGINE=OLAP
AGGREGATE KEY(k1, k2, k3, k4, k5, k6, k10, k11, k7)
COMMENT "OLAP"
DISTRIBUTED BY HASH(k1) BUCKETS 5
PROPERTIES (
"storage_type" = "COLUMN"
);

@imay imay added the kind/fix Categorizes issue or PR as related to a bug. label Nov 7, 2019
@imay imay added this to the 0.12.0 milestone Nov 7, 2019
@zhangy5
Copy link
Contributor Author

zhangy5 commented Nov 7, 2019

#2106

@zhangy5
Copy link
Contributor Author

zhangy5 commented Nov 13, 2019

Fixed

@zhangy5 zhangy5 closed this as completed Nov 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/fix Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

2 participants