Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

basic support for selection/limit/topn executor in InterpreterDAGRequest #150

Merged
merged 1 commit into from
Aug 2, 2019

Conversation

windtalker
Copy link
Contributor

basic support for selection/limit/topn executor in InterpreterDAGRequest

@windtalker windtalker requested a review from zanmato1984 August 2, 2019 05:56
Copy link
Contributor

@zanmato1984 zanmato1984 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zanmato1984 zanmato1984 merged commit ead9609 into pingcap:cop Aug 2, 2019
@zanmato1984 zanmato1984 deleted the cop branch August 2, 2019 05:58
zanmato1984 added a commit that referenced this pull request Oct 17, 2019
* basic framework for coprocessor support in tiflash

* basic support for InterpreterDagRequestV2

* code refine

* tipb submodule use tipb master branch

* rewrite build flow in InterpreterDagRequest

* rename Dag to DAG

* Update tipb submodule

* basic support for selection/limit/topn executor in InterpreterDAGRequest

* basic support for selection/limit/topn executor in InterpreterDAGRequest (#150)

* merge pingcap/cop branch

* Code reorg

* Format

* merge pingcap/cop

* Refine code

* basic support for dag agg executor

* Code refine

* Refine code

* Another way of getting codec flag

* fix cop test regression (#157)

* fix cop test regression

* address comments

* format code

* fix npe during dag execute (#160)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* Add tipb cpp gen in build script

* Fix build error and adjust some formats

* Fix build error

* Fix build error

* Update flash configs

* Format

*  throw exception when meet error duing cop request handling (#162)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver (#166)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver

* fix bug

* 1. refine code, 2. address comments

* update comments

* columnref index is based on executor output schema (#167)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver

* fix bug

* 1. refine code, 2. address comments

* update comments

* columnref index is based on executor output schema

* Move flash/cop/dag to individual library

* DAG planner fix and mock dag request (#169)

* Enhance dbg invoke and add dag as schemaful function

* Add basic sql parse to dag

* Column id starts from 1

* Fix value to ref

* Add basic dag test

* Fix dag bugs and pass 1st mock test

* Make dag go normal routine and add mock dag

* Add todo

* Add comment

* Fix gcc compile error

* Enhance dag test

* Address comments

* Fix DAG get and lock storage

* handle error in cop request (#171)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver

* fix bug

* 1. refine code, 2. address comments

* update comments

* columnref index is based on executor output schema

* handle error in coprocessor request

* refine code

* use Clear to clear a protobuf message completely

* refine code

*  code refine && several minor bug fix (#174)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver

* fix bug

* 1. refine code, 2. address comments

* update comments

* columnref index is based on executor output schema

* handle error in coprocessor request

* refine code

* use Clear to clear a protobuf message completely

* refine code

* code refine && several minor bug fix

* address comments

* address comments

* Fix region id in mock dag

* support udf in (#175)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver

* fix bug

* 1. refine code, 2. address comments

* update comments

* columnref index is based on executor output schema

* handle error in coprocessor request

* refine code

* use Clear to clear a protobuf message completely

* refine code

* code refine && several minor bug fix

* address comments

* address comments

* support udf in

* refine code

* address comments

* address comments

* 1. fix decode literal expr error, 2. add all scalar function sig in scalar_func_map (#177)

* add all scalar function sig in scalarFunMap

* fix literal expr decode

* enable ltrim && rtrim

* code refine

* use throw instead of rethrow in DAGDriver.cpp

* some bug fix (#179)

* add all scalar function sig in scalarFunMap

* fix literal expr decode

* enable ltrim && rtrim

* code refine

* use throw instead of rethrow in DAGDriver.cpp

* 1. fix decode UInt literal error, 2. support mysqlDecimal type

* format code

* Support all DAG operator types in mock SQL -> DAG parser (#176)

* Enhance dbg invoke and add dag as schemaful function

* Add basic sql parse to dag

* Column id starts from 1

* Fix value to ref

* Add basic dag test

* Fix dag bugs and pass 1st mock test

* Make dag go normal routine and add mock dag

* Add todo

* Add comment

* Fix gcc compile error

* Enhance dag test

* Address comments

* Enhance mock sql -> dag compiler and add project test

* Mock sql dag compiler support more expression types and add filter test

* Add topn and limit test

* Add agg for sql -> dag parser and agg test

* Add dag specific codec

* type

* Update codec accordingly

* Remove cop-test

* filter column must be uint8 in tiflash (#180)

* add all scalar function sig in scalarFunMap

* fix literal expr decode

* enable ltrim && rtrim

* code refine

* use throw instead of rethrow in DAGDriver.cpp

* 1. fix decode UInt literal error, 2. support mysqlDecimal type

* format code

* filter column must be uint8 in tiflash

* address comments

* address comments

* address comments

* remove useless include

* 1. fix encode null error, 2. fix empty field type generated by TiFlash (#182)

* add all scalar function sig in scalarFunMap

* fix literal expr decode

* enable ltrim && rtrim

* code refine

* use throw instead of rethrow in DAGDriver.cpp

* 1. fix decode UInt literal error, 2. support mysqlDecimal type

* format code

* filter column must be uint8 in tiflash

* address comments

* address comments

* address comments

* remove useless include

* 1. fix encode null error, 2. fix empty field type generated by TiFlash

* check validation of dag exprs field type (#183)

* check validation of dag exprs field type

* format code

* address comments

*  add more coprocessor mock tests (#185)

* check validation of dag exprs field type

* format code

* address comments

* add more filter test

* add data type tests

* remove useless comment

* disable decimal test

*  add some log about implicit cast (#188)

* check validation of dag exprs field type

* format code

* address comments

* add more filter test

* add data type tests

* remove useless comment

* disable decimal test

* add some log about implicit cast

* address comment

* Pass DAG tests after merging master (#199)

* Enhance dbg invoke and add dag as schemaful function

* Add basic sql parse to dag

* Column id starts from 1

* Fix value to ref

* Add basic dag test

* Fix dag bugs and pass 1st mock test

* Make dag go normal routine and add mock dag

* Add todo

* Add comment

* Fix gcc compile error

* Enhance dag test

* Address comments

* Enhance mock sql -> dag compiler and add project test

* Mock sql dag compiler support more expression types and add filter test

* Add topn and limit test

* Add agg for sql -> dag parser and agg test

* Add dag specific codec

* type

* Update codec accordingly

* Remove cop-test

* Pass tests after merging master

* Fix date/datetime/bit encode error (#200)

* Enhance dbg invoke and add dag as schemaful function

* Add basic sql parse to dag

* Column id starts from 1

* Fix value to ref

* Add basic dag test

* Fix dag bugs and pass 1st mock test

* Make dag go normal routine and add mock dag

* Add todo

* Add comment

* Fix gcc compile error

* Enhance dag test

* Address comments

* Enhance mock sql -> dag compiler and add project test

* Mock sql dag compiler support more expression types and add filter test

* Add topn and limit test

* Add agg for sql -> dag parser and agg test

* Add dag specific codec

* type

* Update codec accordingly

* Remove cop-test

* Pass tests after merging master

* Copy some changes from xufei

* Enable date/datetime test

* Enable date/datetime test

* Refine code

* Adjust date/datetime tiflash rep to UInt

* Fix datetime to Int

* Typo

* improve dag execution time collection (#202)

* improve dag execution time collection

* address comment

* update comments

* update comment

* update comment

*  column id in table scan operator may be -1 (#205)

* improve dag execution time collection

* address comment

* update comments

* update comment

* update comment

* column id in table scan operator may be -1

* column id in table scan operator may be -1

* quick fix for decimal encode (#210)

* quick fix for decimal encode

* address comments

* update comments

* support udf like with 3 arguments (#212)

* support udf like with 3 arguments

* address comments

* add some comments

* Flash-473 optimize date and datetime comparison (#221)

* support udf like with 3 arguments

* address comments

* add some comments

* Flash-473 optimize date and datetime comparison

* address comments

* FLASH-479 select from empty table throw error in tiflash (#223)

* 1. select from empty table throw error in tiflash, 2. add some logs, 3. disable timestamp literal in DAG request

* revert unrelated change

* Update flash service port

* fix bug in DAGBlockOutputStream (#230)

* FLASH-475: Support BATCH COMMANDS in flash service (#232)

* Initial batch command support

* Add config to control thread pool size

* Address comments

* FLASH-483: Combine raft service and flash service (#235)

* Combine raft service and flash service

* Address comment and fix build error

* Update configs

* Fix build error

* Fix test regression

* Fix null value bug in datum

* FLASH-490: Fix table scan with -1 column ID and no agg (#240)

* Fix table scan with -1 column ID and no agg

* Add break

* Remove useless includes

* Use dag context to store void ft instead of dag query source

* Fix decimal type reverse get

* Change adding smallest column to adding handle column, address comments

* throw error if the cop request is not based on full region scan (#247)

* throw error if the cop request is not based on full region scan

* format code

* FLASH-437 Support time zone in coprocessor (#259)

* do not allow timestamp literal in DAG request

* refine code

* fix cop date type encode error

* support tz info in DAG request

* address comments

* Address comment

* FLASH-489 support key condition for coprocessor query (#261)

* support key condition for coprocessor query

* add tests

* remove useless code

* check validation when build RPNElement for function in/notIn

* address comments

* address comments

* only return execute summaies if requested (#264)

* Refine service init (#265)

* FLASH-554 cop check range should be based on region range (#270)

* only return execute summaies if requested

* cop check range should be based on region range

* address comments

* add tests

* minor improve

* minor improve (#273)

* Fix mutex on timezone retrieval (#276)

* fix mutex contention

* add const ref

* Fix race condition of batch command handling (#277)

* address comment

* address comments

* address comments

* Fix NULL order for dag (#281)

* refine get actions in DAGExpressionAnalyzer, fix bug in dbgFuncCoprocessor (#282)

* remove duplicate agg funcs (#283)

* 1. remove duplicate agg funcs, 2. for column ref expr, change column_id to column_index since the value stored in column ref expr is not column id

* bug fix

* address comments

* Update dbms/src/Flash/BatchCommandsHandler.cpp

Co-Authored-By: JaySon <[email protected]>

* revert unnecessary changes
windtalker added a commit that referenced this pull request Oct 25, 2019
* basic framework for coprocessor support in tiflash

* basic support for InterpreterDagRequestV2

* code refine

* tipb submodule use tipb master branch

* rewrite build flow in InterpreterDagRequest

* rename Dag to DAG

* Update tipb submodule

* basic support for selection/limit/topn executor in InterpreterDAGRequest

* basic support for selection/limit/topn executor in InterpreterDAGRequest (#150)

* merge pingcap/cop branch

* Code reorg

* Format

* merge pingcap/cop

* Refine code

* basic support for dag agg executor

* Code refine

* Refine code

* Another way of getting codec flag

* fix cop test regression (#157)

* fix cop test regression

* address comments

* format code

* fix npe during dag execute (#160)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* Add tipb cpp gen in build script

* Fix build error and adjust some formats

* Fix build error

* Fix build error

* Update flash configs

* Format

*  throw exception when meet error duing cop request handling (#162)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver (#166)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver

* fix bug

* 1. refine code, 2. address comments

* update comments

* columnref index is based on executor output schema (#167)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver

* fix bug

* 1. refine code, 2. address comments

* update comments

* columnref index is based on executor output schema

* Move flash/cop/dag to individual library

* DAG planner fix and mock dag request (#169)

* Enhance dbg invoke and add dag as schemaful function

* Add basic sql parse to dag

* Column id starts from 1

* Fix value to ref

* Add basic dag test

* Fix dag bugs and pass 1st mock test

* Make dag go normal routine and add mock dag

* Add todo

* Add comment

* Fix gcc compile error

* Enhance dag test

* Address comments

* Fix DAG get and lock storage

* handle error in cop request (#171)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver

* fix bug

* 1. refine code, 2. address comments

* update comments

* columnref index is based on executor output schema

* handle error in coprocessor request

* refine code

* use Clear to clear a protobuf message completely

* refine code

*  code refine && several minor bug fix (#174)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver

* fix bug

* 1. refine code, 2. address comments

* update comments

* columnref index is based on executor output schema

* handle error in coprocessor request

* refine code

* use Clear to clear a protobuf message completely

* refine code

* code refine && several minor bug fix

* address comments

* address comments

* Fix region id in mock dag

* support udf in (#175)

* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver

* fix bug

* 1. refine code, 2. address comments

* update comments

* columnref index is based on executor output schema

* handle error in coprocessor request

* refine code

* use Clear to clear a protobuf message completely

* refine code

* code refine && several minor bug fix

* address comments

* address comments

* support udf in

* refine code

* address comments

* address comments

* 1. fix decode literal expr error, 2. add all scalar function sig in scalar_func_map (#177)

* add all scalar function sig in scalarFunMap

* fix literal expr decode

* enable ltrim && rtrim

* code refine

* use throw instead of rethrow in DAGDriver.cpp

* some bug fix (#179)

* add all scalar function sig in scalarFunMap

* fix literal expr decode

* enable ltrim && rtrim

* code refine

* use throw instead of rethrow in DAGDriver.cpp

* 1. fix decode UInt literal error, 2. support mysqlDecimal type

* format code

* Support all DAG operator types in mock SQL -> DAG parser (#176)

* Enhance dbg invoke and add dag as schemaful function

* Add basic sql parse to dag

* Column id starts from 1

* Fix value to ref

* Add basic dag test

* Fix dag bugs and pass 1st mock test

* Make dag go normal routine and add mock dag

* Add todo

* Add comment

* Fix gcc compile error

* Enhance dag test

* Address comments

* Enhance mock sql -> dag compiler and add project test

* Mock sql dag compiler support more expression types and add filter test

* Add topn and limit test

* Add agg for sql -> dag parser and agg test

* Add dag specific codec

* type

* Update codec accordingly

* Remove cop-test

* filter column must be uint8 in tiflash (#180)

* add all scalar function sig in scalarFunMap

* fix literal expr decode

* enable ltrim && rtrim

* code refine

* use throw instead of rethrow in DAGDriver.cpp

* 1. fix decode UInt literal error, 2. support mysqlDecimal type

* format code

* filter column must be uint8 in tiflash

* address comments

* address comments

* address comments

* remove useless include

* 1. fix encode null error, 2. fix empty field type generated by TiFlash (#182)

* add all scalar function sig in scalarFunMap

* fix literal expr decode

* enable ltrim && rtrim

* code refine

* use throw instead of rethrow in DAGDriver.cpp

* 1. fix decode UInt literal error, 2. support mysqlDecimal type

* format code

* filter column must be uint8 in tiflash

* address comments

* address comments

* address comments

* remove useless include

* 1. fix encode null error, 2. fix empty field type generated by TiFlash

* check validation of dag exprs field type (#183)

* check validation of dag exprs field type

* format code

* address comments

*  add more coprocessor mock tests (#185)

* check validation of dag exprs field type

* format code

* address comments

* add more filter test

* add data type tests

* remove useless comment

* disable decimal test

*  add some log about implicit cast (#188)

* check validation of dag exprs field type

* format code

* address comments

* add more filter test

* add data type tests

* remove useless comment

* disable decimal test

* add some log about implicit cast

* address comment

* Pass DAG tests after merging master (#199)

* Enhance dbg invoke and add dag as schemaful function

* Add basic sql parse to dag

* Column id starts from 1

* Fix value to ref

* Add basic dag test

* Fix dag bugs and pass 1st mock test

* Make dag go normal routine and add mock dag

* Add todo

* Add comment

* Fix gcc compile error

* Enhance dag test

* Address comments

* Enhance mock sql -> dag compiler and add project test

* Mock sql dag compiler support more expression types and add filter test

* Add topn and limit test

* Add agg for sql -> dag parser and agg test

* Add dag specific codec

* type

* Update codec accordingly

* Remove cop-test

* Pass tests after merging master

* Fix date/datetime/bit encode error (#200)

* Enhance dbg invoke and add dag as schemaful function

* Add basic sql parse to dag

* Column id starts from 1

* Fix value to ref

* Add basic dag test

* Fix dag bugs and pass 1st mock test

* Make dag go normal routine and add mock dag

* Add todo

* Add comment

* Fix gcc compile error

* Enhance dag test

* Address comments

* Enhance mock sql -> dag compiler and add project test

* Mock sql dag compiler support more expression types and add filter test

* Add topn and limit test

* Add agg for sql -> dag parser and agg test

* Add dag specific codec

* type

* Update codec accordingly

* Remove cop-test

* Pass tests after merging master

* Copy some changes from xufei

* Enable date/datetime test

* Enable date/datetime test

* Refine code

* Adjust date/datetime tiflash rep to UInt

* Fix datetime to Int

* Typo

* improve dag execution time collection (#202)

* improve dag execution time collection

* address comment

* update comments

* update comment

* update comment

*  column id in table scan operator may be -1 (#205)

* improve dag execution time collection

* address comment

* update comments

* update comment

* update comment

* column id in table scan operator may be -1

* column id in table scan operator may be -1

* quick fix for decimal encode (#210)

* quick fix for decimal encode

* address comments

* update comments

* support udf like with 3 arguments (#212)

* support udf like with 3 arguments

* address comments

* add some comments

* Flash-473 optimize date and datetime comparison (#221)

* support udf like with 3 arguments

* address comments

* add some comments

* Flash-473 optimize date and datetime comparison

* address comments

* FLASH-479 select from empty table throw error in tiflash (#223)

* 1. select from empty table throw error in tiflash, 2. add some logs, 3. disable timestamp literal in DAG request

* revert unrelated change

* Update flash service port

* fix bug in DAGBlockOutputStream

* fix bug in DAGBlockOutputStream (#230)

* FLASH-475: Support BATCH COMMANDS in flash service (#232)

* Initial batch command support

* Add config to control thread pool size

* Address comments

* init change for array encode

* FLASH-483: Combine raft service and flash service (#235)

* Combine raft service and flash service

* Address comment and fix build error

* Update configs

* Fix build error

* Fix test regression

* Fix null value bug in datum

* FLASH-490: Fix table scan with -1 column ID and no agg (#240)

* Fix table scan with -1 column ID and no agg

* Add break

* Remove useless includes

* Use dag context to store void ft instead of dag query source

* Fix decimal type reverse get

* Change adding smallest column to adding handle column, address comments

* throw error if the cop request is not based on full region scan (#247)

* throw error if the cop request is not based on full region scan

* format code

* FLASH-437 Support time zone in coprocessor (#259)

* do not allow timestamp literal in DAG request

* refine code

* fix cop date type encode error

* support tz info in DAG request

* address comments

* Address comment

* use the new date implementation

* FLASH-489 support key condition for coprocessor query (#261)

* support key condition for coprocessor query

* add tests

* remove useless code

* check validation when build RPNElement for function in/notIn

* address comments

* address comments

* only return execute summaies if requested (#264)

* refine code

* Refine service init (#265)

* fix bug

* fix bug

* FLASH-554 cop check range should be based on region range (#270)

* only return execute summaies if requested

* cop check range should be based on region range

* address comments

* add tests

* minor improve

* add ut for arrow encode

* minor improve (#273)

* update tipb

* Fix mutex on timezone retrieval (#276)

* fix mutex contention

* add const ref

* Fix race condition of batch command handling (#277)

* update tipb version

* set default record_per_chunk to 1024

* address comment

* address comments

* refine code

* refine code

* add mock_dag test

* code refine

* code refine

* address comments

* Fix NULL order for dag (#281)

* refine get actions in DAGExpressionAnalyzer, fix bug in dbgFuncCoprocessor (#282)

* remove duplicate agg funcs (#283)

* 1. remove duplicate agg funcs, 2. for column ref expr, change column_id to column_index since the value stored in column ref expr is not column id

* bug fix

* refine code

* remove useless code

* address comments

* remove uselss include

* address comments

* refine code

* address comments

* format code

* fix typo

* Update dbms/src/Flash/BatchCommandsHandler.cpp

Co-Authored-By: JaySon <[email protected]>

* revert unnecessary changes

* refine code

* fix build error

* refine code

* address comments

* refine code

* address comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants