New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature](update) Support `update on current_timestamp` #25884

Merged

dataroaring merged 12 commits into apache:master from bobhan1:update-on-current-timestamp

Nov 23, 2023

Contributor

bobhan1 commented Oct 25, 2023 •

edited

Loading

Proposed changes

If a column is declared with on update current_timestamp, its value will be updated to the current timestamp whenever there is an update on the row.

Example:

mysql> CREATE TABLE t1 (
    ->     k int,
    ->     v int,
    ->     update_time datetime default current_timestamp on update current_timestamp)
    -> UNIQUE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1
    -> PROPERTIES(
    ->     "replication_num" = "1",
    ->     "enable_unique_key_merge_on_write" = "true"
    -> );
Query OK, 0 rows affected (0.01 sec)
mysql> set enable_unique_key_partial_update=true;
Query OK, 0 rows affected (0.00 sec)

mysql> set enable_insert_strict=false;
Query OK, 0 rows affected (0.00 sec)

mysql> insert into t1(k,v) values(1,1),(2,2);
Query OK, 2 rows affected (0.04 sec)
mysql> select * from t1;
+------+------+---------------------+
| k    | v    | update_time         |
+------+------+---------------------+
|    1 |    1 | 2023-10-25 22:38:10 |
|    2 |    2 | 2023-10-25 22:38:10 |
+------+------+---------------------+
2 rows in set (0.03 sec)
mysql> set enable_unique_key_partial_update=false;
Query OK, 0 rows affected (0.00 sec)

mysql> insert into t1(k,v) values(1,100);
Query OK, 1 row affected (0.04 sec)
mysql> select * from t1;
+------+------+---------------------+
| k    | v    | update_time         |
+------+------+---------------------+
|    2 |    2 | 2023-10-25 22:38:10 |
|    1 |  100 | 2023-10-25 22:38:37 |
+------+------+---------------------+
2 rows in set (0.03 sec)

similar to https://dev.mysql.com/doc/refman/8.0/en/timestamp-initialization.html

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

bobhan1 marked this pull request as draft

October 25, 2023 06:49

bobhan1 force-pushed the update-on-current-timestamp branch from ab0f06c to 185078c Compare

October 25, 2023 06:57

Contributor

github-actions bot commented Oct 25, 2023

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment

Contributor

github-actions bot commented Oct 25, 2023

clang-tidy review says "All clean, LGTM! 👍"

bobhan1 force-pushed the update-on-current-timestamp branch from 185078c to f18428c Compare

October 25, 2023 07:29

Contributor

github-actions bot commented Oct 25, 2023

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment

Contributor

github-actions bot commented Oct 25, 2023

clang-tidy review says "All clean, LGTM! 👍"

bobhan1 force-pushed the update-on-current-timestamp branch 4 times, most recently from 8839174 to 1b42500 Compare

October 25, 2023 09:21

bobhan1 marked this pull request as ready for review

October 25, 2023 09:21

Contributor Author

bobhan1 commented Oct 25, 2023

run buildall

1 similar comment

Contributor Author

bobhan1 commented Oct 25, 2023

run buildall

Contributor

github-actions bot commented Oct 25, 2023

clang-tidy review says "All clean, LGTM! 👍"

Contributor Author

bobhan1 commented Oct 25, 2023

run buildall

Contributor

github-actions bot commented Oct 25, 2023

clang-tidy review says "All clean, LGTM! 👍"

doris-robot commented Oct 25, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.26 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162061702 Bytes

doris-robot commented Oct 25, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.06 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162252858 Bytes

bobhan1 force-pushed the update-on-current-timestamp branch from 5b0e59a to 9792975 Compare

October 25, 2023 14:13

Contributor Author

bobhan1 commented Oct 25, 2023

run buildall

doris-robot commented Oct 25, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.07 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162106364 Bytes

bobhan1 force-pushed the update-on-current-timestamp branch from 3e62479 to 00a4538 Compare

October 26, 2023 03:22

Contributor Author

bobhan1 commented Oct 26, 2023

run buildall

doris-robot commented Oct 26, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.73 seconds
stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162376928 Bytes

bobhan1 force-pushed the update-on-current-timestamp branch 2 times, most recently from 8e65d43 to a270928 Compare

October 27, 2023 01:52

Contributor Author

bobhan1 commented Oct 27, 2023

run buildall

doris-robot commented Oct 27, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.27 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162102094 Bytes

zhannngchen reviewed

View reviewed changes

docs/zh-CN/docs/sql-manual/sql-reference/Data-Definition-Statements/Create/CREATE-TABLE.md Show resolved Hide resolved

doris-robot commented Nov 12, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 2630e83e3d4ceeb8ab339930fbe0af1fb8628a46, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5263	5104	5055	5055
q2	370	187	199	187
q3	2085	2093	2037	2037
q4	1476	1466	1429	1429
q5	4117	4115	4139	4115
q6	251	131	137	131
q7	2092	1578	1603	1578
q8	2767	2763	2755	2755
q9	10397	10402	10189	10189
q10	3488	3550	3543	3543
q11	380	251	247	247
q12	459	298	301	298
q13	4525	4117	4116	4116
q14	336	296	292	292
q15	642	579	565	565
q16	692	617	596	596
q17	1148	1127	1090	1090
q18	7897	7281	7397	7281
q19	1690	1716	1706	1706
q20	614	352	351	351
q21	4917	4589	4563	4563
q22	539	415	433	415
Total cold run time: 56145 ms
Total hot run time: 52539 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4975	5014	5033	5014
q2	341	265	235	235
q3	4025	3997	3980	3980
q4	2797	2736	2735	2735
q5	6535	6528	6504	6504
q6	244	124	125	124
q7	3095	2679	2684	2679
q8	4828	4782	4803	4782
q9	17728	17735	17811	17735
q10	4094	4141	4176	4141
q11	772	642	648	642
q12	990	797	814	797
q13	4281	3862	3892	3862
q14	385	357	348	348
q15	637	562	574	562
q16	799	701	689	689
q17	3907	3961	3942	3942
q18	9325	9323	9202	9202
q19	1907	1808	1787	1787
q20	2365	2053	2043	2043
q21	8597	8585	8708	8585
q22	974	900	876	876
Total cold run time: 83601 ms
Total hot run time: 81264 ms

doris-robot commented Nov 12, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.33 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162402639 Bytes

bobhan1 added 12 commits

November 13, 2023 10:25


          update

12299f5


          add docs

ae95313


          add persist

5ebee97


          update

c71bf54


          format

dd4db37


          update

3e8c333


          format

ec3fb77


          update

785baf8


          update

8a86f7a


          update

74dc8fb


          update

323ef07


          add comment in tests

9170f5d

bobhan1 force-pushed the update-on-current-timestamp branch from 2630e83 to 9170f5d Compare

November 13, 2023 02:25

Contributor Author

bobhan1 commented Nov 13, 2023

run buildall

doris-robot commented Nov 13, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 9170f5db69eb57ef643aaf83d0e3ef36f93e4003, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5247	5115	5036	5036
q2	376	184	198	184
q3	2076	2015	2031	2015
q4	1485	1451	1416	1416
q5	4175	4166	4146	4146
q6	253	133	139	133
q7	2089	1605	1608	1605
q8	2772	2765	2769	2765
q9	10480	10329	10202	10202
q10	3498	3550	3565	3550
q11	395	268	262	262
q12	459	307	309	307
q13	4517	4133	4132	4132
q14	330	293	286	286
q15	613	591	557	557
q16	708	616	599	599
q17	1144	1105	1093	1093
q18	7802	7423	7493	7423
q19	1721	1708	1715	1708
q20	588	369	365	365
q21	4956	4607	4592	4592
q22	541	442	428	428
Total cold run time: 56225 ms
Total hot run time: 52804 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4959	5091	4970	4970
q2	347	267	260	260
q3	4029	3979	3956	3956
q4	2818	2775	2756	2756
q5	6481	6520	6492	6492
q6	245	125	131	125
q7	3099	2759	2754	2754
q8	4811	4842	4828	4828
q9	17821	17652	17614	17614
q10	4103	4143	4146	4143
q11	744	668	652	652
q12	991	792	854	792
q13	4288	3910	3877	3877
q14	383	342	365	342
q15	602	558	588	558
q16	768	697	679	679
q17	3895	3894	3915	3894
q18	9391	9210	9322	9210
q19	1886	1772	1796	1772
q20	2401	2072	2033	2033
q21	8922	8907	8813	8813
q22	920	918	867	867
Total cold run time: 83904 ms
Total hot run time: 81387 ms

doris-robot commented Nov 13, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.06 seconds
stream load tsv: 551 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162202697 Bytes

Contributor Author

bobhan1 commented Nov 13, 2023

run feut

Contributor Author

bobhan1 commented Nov 13, 2023

run pipelinex_p0

Contributor Author

bobhan1 commented Nov 13, 2023

run feut

2 similar comments

Contributor Author

bobhan1 commented Nov 13, 2023

run feut

Contributor Author

bobhan1 commented Nov 13, 2023

run feut

zhannngchen approved these changes

View reviewed changes

Contributor

zhannngchen left a comment

LGTM

github-actions bot added the approved label

Contributor

github-actions bot commented Nov 13, 2023

PR approved by at least one committer and no changes requested.

dataroaring approved these changes

View reviewed changes

Contributor

dataroaring left a comment

LGTM

dataroaring merged commit 4b22fc1 into apache:master

seawinde pushed a commit to seawinde/doris that referenced this pull request


          [Feature](update) Support update on current_timestamp (apache#25884)

fe91f0e

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request


          [Feature](update) Support update on current_timestamp (apache#25884)

a2c1155

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved not-merge/2.0 reviewed