fix bugs: #136
Merged
fix bugs: #136
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
morningman
commented
Nov 12, 2017
morningman
commented
- Rename database does not modify the corresponding names in cluster.
- Result of SHOW BROKERS does not match that of in SHOW PROC '/brokers'.
- Bugs when set property for other user using admin user.
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <[email protected]> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <[email protected]> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <[email protected]> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <[email protected]> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <[email protected]> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <[email protected]> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <[email protected]> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <[email protected]> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <[email protected]> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <[email protected]> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <[email protected]> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <[email protected]> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <[email protected]> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <[email protected]> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <[email protected]> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <[email protected]> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <[email protected]> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <[email protected]> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <[email protected]> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <[email protected]> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <[email protected]> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <[email protected]> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <[email protected]> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <[email protected]> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <[email protected]> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <[email protected]> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <[email protected]> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <[email protected]> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <[email protected]> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <[email protected]> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <[email protected]> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <[email protected]> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <[email protected]> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <[email protected]> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <[email protected]> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <[email protected]> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <[email protected]> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <[email protected]> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <[email protected]> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <[email protected]> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <[email protected]> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <[email protected]> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <[email protected]> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <[email protected]> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <[email protected]> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <[email protected]> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <[email protected]> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <[email protected]> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <[email protected]> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <[email protected]> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <[email protected]> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <[email protected]> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <[email protected]> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <[email protected]> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <[email protected]> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <[email protected]> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <[email protected]> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <[email protected]> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <[email protected]> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <[email protected]> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <[email protected]> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <[email protected]> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <[email protected]> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <[email protected]> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <[email protected]> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <[email protected]> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <[email protected]> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <[email protected]> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <[email protected]> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <[email protected]> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <[email protected]> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <[email protected]> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <[email protected]> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <[email protected]> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <[email protected]> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <[email protected]> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <[email protected]> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <[email protected]> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <[email protected]> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <[email protected]> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <[email protected]> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <[email protected]> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <[email protected]> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <[email protected]> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <[email protected]> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <[email protected]> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <[email protected]> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <[email protected]> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <[email protected]> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <[email protected]> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <[email protected]> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <[email protected]> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <[email protected]> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <[email protected]> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <[email protected]> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <[email protected]> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <[email protected]> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <[email protected]> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <[email protected]> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <[email protected]> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <[email protected]> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <[email protected]> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <[email protected]> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <[email protected]> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <[email protected]> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <[email protected]> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <[email protected]> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <[email protected]> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <[email protected]> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <[email protected]> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <[email protected]> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <[email protected]> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <[email protected]> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <[email protected]> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <[email protected]> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <[email protected]> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <[email protected]> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <[email protected]> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <[email protected]> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <[email protected]> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <[email protected]> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <[email protected]> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <[email protected]> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <[email protected]> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <[email protected]> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <[email protected]> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <[email protected]> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <[email protected]> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <[email protected]> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut Co-authored-by: Luwei <[email protected]> Co-authored-by: deardeng <[email protected]>
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jun 20, 2022
Co-authored-by: deardeng <[email protected]> Squashed commit of the following: commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <[email protected]> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <[email protected]> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <[email protected]> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <[email protected]> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <[email protected]> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <[email protected]> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <[email protected]> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <[email protected]> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <[email protected]> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <[email protected]> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <[email protected]> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <[email protected]> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <[email protected]> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <[email protected]> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <[email protected]> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <[email protected]> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <[email protected]> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <[email protected]> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <[email protected]> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <[email protected]> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <[email protected]> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <[email protected]> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <[email protected]> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <[email protected]> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <[email protected]> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <[email protected]> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <[email protected]> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <[email protected]> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <[email protected]> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <[email protected]> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <[email protected]> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <[email protected]> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <[email protected]> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <[email protected]> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <[email protected]> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <[email protected]> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <[email protected]> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <[email protected]> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <[email protected]> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <[email protected]> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <[email protected]> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <[email protected]> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <[email protected]> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <[email protected]> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734)
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jun 20, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <[email protected]> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <[email protected]> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <[email protected]> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <[email protected]> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <[email protected]> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <[email protected]> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <[email protected]> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <[email protected]> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <[email protected]> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <[email protected]> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <[email protected]> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <[email protected]> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <[email protected]> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <[email protected]> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <[email protected]> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <[email protected]> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <[email protected]> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <[email protected]> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <[email protected]> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <[email protected]> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <[email protected]> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <[email protected]> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <[email protected]> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <[email protected]> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <[email protected]> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <[email protected]> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <[email protected]> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <[email protected]> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <[email protected]> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <[email protected]> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <[email protected]> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <[email protected]> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <[email protected]> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <[email protected]> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <[email protected]> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <[email protected]> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <[email protected]> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <[email protected]> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <[email protected]> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <[email protected]> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <[email protected]> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <[email protected]> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <[email protected]> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <[email protected]> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <[email protected]> Co-authored-by: deardeng <[email protected]>
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jul 6, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <[email protected]> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <[email protected]> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <[email protected]> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <[email protected]> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <[email protected]> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <[email protected]> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <[email protected]> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <[email protected]> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <[email protected]> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <[email protected]> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <[email protected]> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <[email protected]> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <[email protected]> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <[email protected]> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <[email protected]> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <[email protected]> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <[email protected]> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <[email protected]> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <[email protected]> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <[email protected]> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <[email protected]> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <[email protected]> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <[email protected]> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <[email protected]> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <[email protected]> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <[email protected]> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <[email protected]> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <[email protected]> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <[email protected]> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <[email protected]> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <[email protected]> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <[email protected]> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <[email protected]> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <[email protected]> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <[email protected]> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <[email protected]> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <[email protected]> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <[email protected]> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <[email protected]> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <[email protected]> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <[email protected]> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <[email protected]> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <[email protected]> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <[email protected]> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <[email protected]> Co-authored-by: deardeng <[email protected]>
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jul 6, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <[email protected]> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <[email protected]> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <[email protected]> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <[email protected]> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <[email protected]> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <[email protected]> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <[email protected]> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <[email protected]> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <[email protected]> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <[email protected]> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <[email protected]> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <[email protected]> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <[email protected]> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <[email protected]> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <[email protected]> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <[email protected]> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <[email protected]> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <[email protected]> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <[email protected]> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <[email protected]> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <[email protected]> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <[email protected]> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <[email protected]> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <[email protected]> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <[email protected]> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <[email protected]> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <[email protected]> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <[email protected]> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <[email protected]> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <[email protected]> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <[email protected]> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <[email protected]> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <[email protected]> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <[email protected]> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <[email protected]> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <[email protected]> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <[email protected]> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <[email protected]> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <[email protected]> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <[email protected]> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <[email protected]> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <[email protected]> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <[email protected]> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <[email protected]> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <[email protected]> Co-authored-by: deardeng <[email protected]>
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jul 6, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <[email protected]> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <[email protected]> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <[email protected]> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <[email protected]> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <[email protected]> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <[email protected]> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <[email protected]> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <[email protected]> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <[email protected]> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <[email protected]> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <[email protected]> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <[email protected]> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <[email protected]> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <[email protected]> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <[email protected]> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <[email protected]> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <[email protected]> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <[email protected]> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <[email protected]> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <[email protected]> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <[email protected]> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <[email protected]> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <[email protected]> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <[email protected]> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <[email protected]> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <[email protected]> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <[email protected]> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <[email protected]> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <[email protected]> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <[email protected]> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <[email protected]> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <[email protected]> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <[email protected]> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <[email protected]> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <[email protected]> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <[email protected]> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <[email protected]> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <[email protected]> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <[email protected]> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <[email protected]> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <[email protected]> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <[email protected]> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <[email protected]> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <[email protected]> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <[email protected]> Co-authored-by: deardeng <[email protected]>
platoneko
added a commit
to platoneko/doris
that referenced
this pull request
Jul 7, 2022
commit e5ead33dd247a6b8e301209b86f19a2089103551 Author: plat1ko <[email protected]> Date: Mon Jun 20 15:37:21 2022 +0800 temporarily avoid src/io conflicts with doris/master commit 2ffee9d8906af61c40203a71fd0c30db14fbb47a Author: plat1ko <[email protected]> Date: Fri Jun 17 20:59:41 2022 +0800 (cold_on_s3) Report remote data size in tablets (apache#195) * be report tablet remote size * update TTabletStat * fe report tablet remote data size commit ee0d856685c1f089f585d032f26203d4397dc3e7 Author: deardeng <[email protected]> Date: Fri Jun 17 16:06:23 2022 +0800 (cold_on_s3) support modify partition add storage policy commit bbf81a905ff3bda38f80f629755ad983c6f6d556 Author: deardeng <[email protected]> Date: Fri Jun 17 10:24:14 2022 +0800 (cold_on_s3) support alter table add storage policy commit 1c662b90d59ef8a870ebb7cc89801433fdab1f8a Author: plat1ko <[email protected]> Date: Thu Jun 16 16:01:02 2022 +0800 (cold_on_s3) directly upload rowset exceeded ttl (apache#186) * directly upload rowset exceed ttl * clang-format commit f17709819a2a51b2783cceee2b32451740f95256 Author: plat1ko <[email protected]> Date: Thu Jun 16 11:39:08 2022 +0800 (cold_on_s3) Implement and rewrite some filesystem methods (apache#183) * remove temporary test code * no need to send fs to LocalFileWriter ctor * rewrite some fs methods * fix opened fd leak commit 1f7a09d90e38a16574f841701fa04b8164c5155a Author: Luwei <[email protected]> Date: Tue Jun 14 18:07:52 2022 +0800 (cold_on_s3) fix log printing (apache#176) commit 41b2ca65c6340e9432e58d53816b380bdfa9e9b7 Author: Luwei <[email protected]> Date: Tue Jun 14 17:02:48 2022 +0800 (cold_on_s3) fix rowset write timestamp bug (apache#175) commit f987fdffdc9d3245e44f226ea19fa489a3951652 Author: deardeng <[email protected]> Date: Tue Jun 14 16:14:56 2022 +0800 (cold_on_s3) fix be core in file cache and fix be report remote capacity error (apache#174) commit 4190cd3784a7c308afdc0f29878a40bc476999f5 Author: deardeng <[email protected]> Date: Tue Jun 14 11:17:23 2022 +0800 (cold_on_s3) fix compile error and thread pool core commit f76186517b65b8bf70dc2fa456f2ec32d791e4e7 Author: plat1ko <[email protected]> Date: Mon Jun 13 19:15:37 2022 +0800 (cold_on_s3) Revert FilePathDesc (apache#136) * change cooldown pick rowset strategy; add/modify some log * revert FilePathDesc * return unsupport error to avoid undefined behavior * patch to reduce path copy * fix be ut commit 9ddac6074840417b7a73a4eb8c3f4da00f97a4d1 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:36 2022 +0800 (cold_on_s3) Add ut for s3 read (apache#147) commit d325739ab1ba4e13539ec0b6401ebc13a2dad851 Author: Luwei <[email protected]> Date: Mon Jun 13 16:07:21 2022 +0800 (cold_on_s3) Add metrics for file system operation (apache#146) * Add metrics for file system operation * fix status code commit b42bb65cc24870f78f3f4a5c7682d0983eb7ac8b Author: deardeng <[email protected]> Date: Mon Jun 13 12:53:10 2022 +0800 (cold_on_s3) support show proc backends return remote used capacity commit fdffe69455fe867d3d95a85311684664abaaf5bb Author: deardeng <[email protected]> Date: Fri Jun 10 16:08:09 2022 +0800 (cold_on_s3) support storage policy use default policy commit 1921e75bce6b85bac61f278b3077a47c916aab00 Author: deardeng <[email protected]> Date: Thu Jun 9 10:40:46 2022 +0800 (cold_on_s3) create table support different partition use their own storage policy commit ebb0b585a1dbc8fae146cc742f8000b1b4033507 Author: plat1ko <[email protected]> Date: Fri Jun 10 22:28:40 2022 +0800 (cold_on_s3) Implement some fs methods (apache#163) * implement some fs methods * add more log info commit 89e820d3f1a9f6b4680bf445ff60a825c315bcab Author: Lei Zhang <[email protected]> Date: Thu Jun 9 23:12:58 2022 +0800 [fix](be) fix asan be set_storage_medium core (apache#9986) (apache#9987) apache#9986 commit ffe2504e24c9afb06b7e843b455f341f1e8864d7 Author: plat1ko <[email protected]> Date: Thu Jun 9 16:17:06 2022 +0800 [hot-fix](cold_on_s3) do not skip 0 segment rowset (apache#143) commit f8cd4a12a838d2ee25d0ec72acc3d346f8c24db9 Author: Luwei <[email protected]> Date: Wed Jun 8 16:09:35 2022 +0800 (cold_on_s3) Optimize cooldown policy (apache#113) * Optimize cooldown policy * fix ut * remove useless log * add ut * add ut * rename var * fix typo and ut commit 08aae82daae19d9d7dc4bc0dd294def6d1ea6157 Author: plat1ko <[email protected]> Date: Wed Jun 8 16:02:41 2022 +0800 provide quick_cooldown switch for regression test (apache#132) commit 704af00135ff3aacfe746ff617efebfe19e7c53a Author: deardeng <[email protected]> Date: Tue Jun 7 18:05:01 2022 +0800 (cold_on_s3) fix multiple fe synchronization resource storage policy exit commit 58f09c47d4128d6cfe4959a61a0d0e3bde0cbc82 Author: platoneko <[email protected]> Date: Thu Jun 2 11:38:46 2022 +0800 (cold_on_s3) fix move bug commit cb4d4d766828103355c47dff8cc37c018c767eb5 Author: deardeng <[email protected]> Date: Thu Jun 2 11:24:16 2022 +0800 (cold_on_s3) fix clang report '_exec_env' is not used commit 6b8f6f2b62e01abfdd30b5c42b9c72d554ec5f4e Author: platoneko <[email protected]> Date: Thu Jun 2 00:05:42 2022 +0800 (cold_on_s3) reformat be code to make ci happy commit 4168d9fb1c2ddad57cff7afb9da20272f1a0d389 Author: platoneko <[email protected]> Date: Wed Jun 1 23:44:56 2022 +0800 (cold_on_s3) fix be ut commit 110fc7421ceba32a8cf46b237e65309d94a60752 Author: platoneko <[email protected]> Date: Wed Jun 1 22:36:37 2022 +0800 (cold_on_s3) fix fe compile error commit a74f84a7f5980c5c7cc5d5c80b0bfd379cfcf11d Author: plat1ko <[email protected]> Date: Thu May 26 11:15:31 2022 +0800 [Enhancement](cold_on_s3) refresh fs map (apache#67) * refresh fs map * fetch cooldown ttl from storage policy commit 71a4f364133e1686d0e86a04ab785a64c47ae701 Author: plat1ko <[email protected]> Date: Tue May 24 21:28:06 2022 +0800 (cold_on_s3) Do not compact remote rowsets; modify pick cooldown rowset strategy (apache#53) * time from the beginning of upload * compact to appropriate state before upload * only do compaction on local rowsets commit b434d2c9a9c4f4d9d81b4fd3bbabb6cd913e7262 Author: plat1ko <[email protected]> Date: Tue May 24 19:22:25 2022 +0800 (cold_on_s3) use noncopy iostream for s3 get object (apache#54) commit 606b98217eb261b9001da77bf766b52c16efb426 Author: plat1ko <[email protected]> Date: Tue May 24 18:42:09 2022 +0800 (cold_on_s3) copy remote rowset meta to new tablet meta (apache#52) commit e218f051e685e27a1392befb40ee7fef832a1515 Author: deardeng <[email protected]> Date: Tue May 24 12:58:43 2022 +0800 (cold_on_s3) support storage policy change fix thrift merge redefine commit 152fa1856fb3034afcf0b3dc4272b0bafac8f630 Author: deardeng <[email protected]> Date: Tue May 24 11:45:13 2022 +0800 (cold_on_s3) support storage policy change and create table add storage policy name (apache#39) * [feature](cold-hot)support storage policy change and create table add storage policy name * support storage policy change and create table add storage policy name, check resource policy exist when create table. * when create table use storage policy check it exist * support storage policy change and create table add storage policy name and when create table use storage policy check it exist * support storage policy change fix some bug. * fix some code style commit 77101dd7f563e1d0ddd03fbe6b35b4196811a87e Author: deardeng <[email protected]> Date: Sat May 21 07:37:43 2022 -0700 (cold_on_s3) support storage policy (apache#28) * [feature](cold-hot)support storage policy * fix review * fix review v1 commit 9923d59e97c25e6d6fb54bf09ed6e0a00a5704c3 Author: plat1ko <[email protected]> Date: Fri May 20 16:22:36 2022 +0800 (cold_on_s3) Batch upload to improve upload speed (apache#38) * test upload duration * use s3 transfer * batch upload * some fix in s3 filesystem commit 29656e73f33e94bb85a954eb31dca9e6eea33cf4 Author: Luwei <[email protected]> Date: Wed May 18 15:08:40 2022 +0800 (cold_on_s3) Implement periodic uploading of rowsets to remote storage (apache#18) * Implement periodic uploading of rowsets to remote storage * add queue size check * add test config for cooldwon * modify read path commit c35ed8bc87d212d442f1b57be0340230de4fb619 Author: plat1ko <[email protected]> Date: Tue May 17 17:22:19 2022 +0800 (cold_on_s3) fe distinguish drop partition (apache#26) commit e216b0320d71d04a4fde829f13fd8c8bfa94854b Author: deardeng <[email protected]> Date: Mon May 16 21:02:51 2022 -0700 (cold_on_s3) support clone task don't download file in s3 (apache#15) * apache#12 support clone task don't download file in s3 * apache#12 add clone ut * fix code format * fix commit apache#12 commit be7521900726c9cd401e3a6503177e0365fc8a9a Author: plat1ko <[email protected]> Date: Tue May 17 12:01:32 2022 +0800 [Enhancement](cold_on_s3) FE distinguish drop table or drop replica; disk capacity distinguish local or remote (apache#24) * disk capacity differentiates between local and remote * fe drop table commit 414a1cb0c5d9777b6eb0b2ad5d7557dc1bf4a243 Author: plat1ko <[email protected]> Date: Mon May 16 11:55:42 2022 +0800 [Enhancement](cold_on_s3) Physically remove remote rowsets when drop table/partition (apache#14) * update is_local logical * remove remote rowsets when drop table/partition commit ed7cf258051716ff99c37812a7fa63ac1fc69f3f Author: Luwei <[email protected]> Date: Mon May 16 11:17:30 2022 +0800 (cold_on_s3) Implement reading rowset data from remote storage (apache#11) * adjust io stack * support s3 * refine code * fix ut * remove useless code * rename rblock * fix err msg commit da6ca91f22d30e9d16e4dd70dbcad30714d321c9 Author: plat1ko <[email protected]> Date: Wed May 11 16:29:54 2022 +0800 (cold_on_s3) Implement tablet cooldown (apache#10) * tablet cooldown * add global_local_filesystem to keep compatible * test tablet cooldown * revert engine_storage_migration_task_test.cpp commit 83f4a3bfcd70cd747941338cd89148fb69d272ae Author: platoneko <[email protected]> Date: Sun May 1 12:06:41 2022 +0800 (cold_on_s3) Rowset write path using fs (apache#7) * replace WritableBlock with FileWriter * local file system * set fs when load dir * fix be ut commit 3585fe6 Author: Xinyi Zou <[email protected]> Date: Tue Jun 7 11:31:49 2022 +0800 [fix][mem tracker] Fix logout load task mem tracker dcheck fail (apache#9943) * fix tracker 0602 * fix format commit 429a41d Author: Xinyi Zou <[email protected]> Date: Tue May 31 19:12:42 2022 +0800 [bugfix] Fix BE core about vectorized join build thread memtracker switch, and FileStat duplicate commit 89d3a36 Author: Pxl <[email protected]> Date: Thu May 26 16:51:01 2022 +0800 [Bug] [Vectorized] add padding when load char type data (apache#9734) Co-authored-by: Luwei <[email protected]> Co-authored-by: deardeng <[email protected]>
starocean999
pushed a commit
to starocean999/incubator-doris
that referenced
this pull request
Jul 26, 2023
# Checklist * 内部Jira: http://39.106.86.136:8090/browse/CIR-1103 * 修复PR：apache#14747 * master是否修复：是 * 1.2-lts是否修复：1.2上没问题 * 1.1-lts是否修复：是 Co-authored-by: smallhibiscus <844981280>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.