Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prepare 0.3.2 #777

Merged
merged 6 commits into from
Dec 21, 2021
Merged

Prepare 0.3.2 #777

merged 6 commits into from
Dec 21, 2021

Conversation

zhicwu
Copy link
Contributor

@zhicwu zhicwu commented Dec 13, 2021 •
edited
Loading

  • more tests for ClickHouseInputStream
  • use separate thread pool for I/O(instead of commonPool)
  • remove log4j dependency
  • new options
    • clickhouse-http-client: http_connection_provider defaults to HTTP_URL_CONNECTION, can be changed to HTTP_CLIENT in JDK 11 or above
    • clickhouse-jdbc: customTypeMappings to define mapping between ClickHouse data type and Java class
  • correct behaviour of handling named parameters
    • select :p1(DateTime64(3)) p1, :p2 p2, :p1 p3 has only two parameters: p1 and p2
    • select ? p1, ? p2, ? p3 has 3 parameters
  • optimize LZ4 input and output stream for memory effciency
  • add new data type Bool(instead of treating it as alias to Int8)
  • only show functions under system schema
  • enhance prepared statement for below two use cases
    • insert into ... values
    • insert into ... format ...
  • support more AggregateFunction types

@github-actions
Copy link 

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score    Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   245.516 ± 29.278  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   239.286 ± 25.687  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   237.156 ± 27.516  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   235.730 ± 30.784  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   247.038 ± 28.548  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   246.739 ± 28.367  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   248.922 ± 31.022  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   244.815 ± 28.263  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1050.045 ± 94.970  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1012.921 ± 76.739  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1065.005 ± 65.120  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1055.477 ± 53.199  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   738.557 ± 52.523  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   707.033 ± 54.532  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   711.309 ± 76.160  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   731.522 ± 56.784  ops/s

@enqueue
Copy link
Contributor

enqueue commented Dec 14, 2021

Bear in mind that a JDBC connection pool might create some dozens of connections ahead of time for reuse. If each of these connections spawns another threadpool, we might run into issues.

@zhicwu
Copy link
Contributor Author

zhicwu commented Dec 14, 2021

Bear in mind that a JDBC connection pool might create some dozens of connections ahead of time for reuse. If each of these connections spawns another threadpool, we might run into issues.

max_threads_per_client defaults to zero, but when it's configured with a number greater than zero, each client will have its own thread pool, instead of reusing ClickHouseClient.defaultExecutor. Maybe keep alive timeout should be set for core pool as well to avoid the situation of having too many idle threads.

@zhicwu zhicwu linked an issue Dec 14, 2021 that may be closed by this pull request
Confirmation that clickhouse-jdbc is not affected by CVE-2021-44228? #779
Closed
@github-actions
Copy link 

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   287.056 ±  35.974  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   275.543 ±  28.798  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   272.975 ±  24.080  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   274.608 ±  29.392  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   281.707 ±  35.551  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   284.496 ±  31.182  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   284.215 ±  31.433  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   286.415 ±  30.166  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1626.552 ±  77.295  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1575.407 ±  81.022  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1628.521 ±  57.435  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1589.268 ±  62.086  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20  1104.106 ±  73.055  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20  1066.350 ±  80.787  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20  1077.503 ± 100.513  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20  1020.707 ±  94.470  ops/s

@github-actions
Copy link 

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   279.993 ±  30.792  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   267.521 ±  27.470  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   247.821 ±  45.978  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   268.745 ±  27.913  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   281.262 ±  33.734  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   283.223 ±  33.768  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   280.931 ±  36.562  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   279.858 ±  33.696  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1599.161 ± 100.538  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1594.226 ±  62.095  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1647.391 ±  75.005  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1581.111 ±  66.437  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20  1066.236 ±  96.858  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20  1077.183 ±  72.847  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20  1084.858 ± 103.049  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20  1066.591 ±  85.452  ops/s

@zhicwu zhicwu changed the title Fit and finish Prepare 0.3.2 Dec 16, 2021
@github-actions
Copy link 

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   280.841 ±  30.689  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   273.023 ±  31.249  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   273.944 ±  28.472  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   269.603 ±  27.688  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   282.478 ±  35.168  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   286.499 ±  30.990  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   284.089 ±  31.715  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   285.575 ±  29.491  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1625.814 ±  92.580  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1586.369 ±  82.836  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1648.988 ±  71.028  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1637.752 ±  75.794  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20  1017.791 ± 149.155  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20  1088.756 ±  84.440  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20  1121.829 ±  78.587  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20  1113.738 ± 105.829  ops/s

@zhicwu zhicwu merged commit 48a6281 into ClickHouse:develop Dec 21, 2021
@github-actions
Copy link 

Benchmark                                (client)  (connection)  (statement)   (type)   Mode  Cnt     Score     Error  Units
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20   266.786 ±  31.967  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20   271.687 ±  29.301  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20   266.762 ±  25.757  ops/s
Basic.insertOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20   266.166 ±  27.591  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20   281.392 ±  33.433  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20   280.805 ±  28.723  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20   277.415 ±  32.561  ops/s
Basic.insertOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20   281.632 ±  31.947  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse       normal  default  thrpt   20  1584.398 ± 140.062  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc         reuse     prepared  default  thrpt   20  1555.086 ±  74.058  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new       normal  default  thrpt   20  1594.057 ±  94.675  ops/s
Basic.selectOneRandomNumber  clickhouse-http-jdbc           new     prepared  default  thrpt   20  1603.031 ±  58.615  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse       normal  default  thrpt   20  1062.549 ±  77.554  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc         reuse     prepared  default  thrpt   20  1046.488 ±  92.378  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new       normal  default  thrpt   20  1095.806 ± 112.619  ops/s
Basic.selectOneRandomNumber  clickhouse-grpc-jdbc           new     prepared  default  thrpt   20  1091.176 ±  83.580  ops/s

This was referenced Dec 21, 2021
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Confirmation that clickhouse-jdbc is not affected by CVE-2021-44228?
2 participants
@zhicwu @enqueue