bugfix : check if table meta cache should be refreshed in AT mode #4734

Bughue · 2022-06-30T04:09:38Z

I have registered the PR changes.

Ⅰ. Describe what this PR did

AT模式下，在构造record的时候会遍历resultset并从tablemeta里获取表结构。如果数据库表中新建了一个字段，就会报空指针。
at io.seata.rm.datasource.sql.struct.TableRecords.buildRecords(TableRecords.java:195)

旧方案（已废弃）：

先把原本的buildRecords改为private，加上检测，在检测到null时抛出一个固定的TableMetaException异常
对外提供新的buildRecords方法，要求传入StatementProxy，方便在遇到TableMetaException直接刷新meta
新方法在捕捉到异常后会进行tablemeta刷新(靠StatementProxy提供connection、dbtype、resourceid)
现在用双锁来控制并发，内存标识在tableMeta里面（保证同key只会刷新一次，不管是不是并发）
AbstractUndoExecutor里没有StatementProxy，但undo过程如果meta都不对应的话肯定是回滚不了，已忽略

新方案（已实现）：

把原本的buildRecords加上检测，在检测到null时抛出一个固定的TableMetaException异常
AbstractDMLBaseExecutor.executeAutoCommitFalse在捕捉到异常后会提出一个meta刷新事件（加入队列）
DataSourceProxy在初始化的时候就建立一个阻塞队列和一个专门处理meta刷新的单一线程的线程池，他会从队列不断取出刷新事件进行刷新（如果1秒前已经刷过就跳过）

关于单测：在TableRecordsTest模拟了字段不同的情况

Ⅱ. Does this pull request fix one issue?

fixes #4572
fixes #5292

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

codecov-commenter · 2022-07-14T06:15:56Z

Codecov Report

Merging #4734 (16ca49a) into develop (00ca98c) will decrease coverage by 0.07%.
The diff coverage is 56.89%.

❗ Current head 16ca49a differs from pull request most recent head 96bae53. Consider uploading reports for the commit 96bae53 to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@              Coverage Diff              @@
##             develop    #4734      +/-   ##
=============================================
- Coverage      48.63%   48.57%   -0.07%     
+ Complexity      4184     4180       -4     
=============================================
  Files            743      744       +1     
  Lines          26646    26688      +42     
  Branches        3328     3332       +4     
=============================================
+ Hits           12960    12963       +3     
- Misses         12296    12326      +30     
- Partials        1390     1399       +9

Impacted Files	Coverage Δ
...datasource/exec/mysql/MySQLUpdateJoinExecutor.java	`78.34% <ø> (ø)`
...n/java/io/seata/rm/datasource/DataSourceProxy.java	`76.59% <33.33%> (-3.80%)`	⬇️
...ta/rm/datasource/exec/AbstractDMLBaseExecutor.java	`51.85% <50.00%> (-5.30%)`	⬇️
...m/datasource/sql/struct/TableMetaCacheFactory.java	`52.63% <50.00%> (-22.37%)`	⬇️
...ta/rm/datasource/exception/TableMetaException.java	`66.66% <80.00%> (ø)`
...o/seata/rm/datasource/sql/struct/TableRecords.java	`68.46% <100.00%> (+1.48%)`	⬆️
...n/src/main/java/io/seata/common/util/IdWorker.java	`77.08% <0.00%> (-6.25%)`	⬇️
...ource/sql/struct/cache/AbstractTableMetaCache.java	`82.35% <0.00%> (-2.95%)`	⬇️
...rage/redis/store/RedisTransactionStoreManager.java	`74.93% <0.00%> (-1.57%)`	⬇️
...torage/file/store/FileTransactionStoreManager.java	`55.62% <0.00%> (-0.65%)`	⬇️
... and 1 more

…ild_null

rm-datasource/src/main/java/io/seata/rm/datasource/sql/struct/TableRecords.java

…ild_null

CLAassistant · 2022-12-12T13:16:11Z

All committers have signed the CLA.

CLAassistant · 2022-12-12T13:16:13Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

rm-datasource/src/main/java/io/seata/rm/datasource/exception/RmTableMetaException.java

rm-datasource/src/main/java/io/seata/rm/datasource/sql/struct/cache/AbstractTableMetaCache.java

rm-datasource/src/main/java/io/seata/rm/datasource/DataSourceProxy.java

…v_4572_new_feild_null

slievrly · 2023-02-27T15:58:09Z

rm-datasource/src/main/java/io/seata/rm/datasource/sql/struct/TableMetaCacheFactory.java

+        TableMetaRefreshHolder(DataSourceProxy dataSource) {
+            this.dataSource = dataSource;
+            this.lastRefreshFinishTime = System.currentTimeMillis() - TABLE_META_REFRESH_INTERVAL_TIME;
+            this.tableMetaRefreshQueue = new LinkedBlockingQueue<>();


limited the queue length

slievrly · 2023-02-27T16:09:44Z

rm-datasource/src/main/java/io/seata/rm/datasource/sql/struct/TableRecords.java

@@ -55,6 +59,8 @@ public class TableRecords implements java.io.Serializable {

    private List<Row> rows = new ArrayList<Row>();

+    private static final Interner<String> TABLE_NAME_POOL = Interners.newWeakInterner();


where used?

…v_4572_new_feild_null

slievrly

LGTM

…ache#4734) (cherry picked from commit 28b3413)

…ache#4734)

wuwen5 · 2024-07-09T03:37:09Z

rm-datasource/src/main/java/io/seata/rm/datasource/DataSourceProxy.java

@@ -120,20 +98,18 @@ private void init(DataSource dataSource, String resourceGroupId) {
        }
        initResourceId();
        DefaultResourceManager.get().registerResource(this);
-        if (ENABLE_TABLE_META_CHECKER_ENABLE) {


原定时刷新缓存的功能在这次更新后失效了？

tableMeta缓存刷新失效? #6660

Bughue added 5 commits June 27, 2022 17:54

todo

ea0daa4

simulate nullpointerExp

1ac4cbb

fix npe

68f2687

fix npe

dc21dcd

fix npe

42ee542

Bughue changed the title ~~[WIP] bugfix : check if table meta cache should be refreshed in AT mode~~ bugfix : check if table meta cache should be refreshed in AT mode Jun 30, 2022

Bughue added 2 commits June 30, 2022 17:21

format

7de5d38

format

de27bf6

l81893521 self-requested a review July 1, 2022 07:45

先去掉单测，让构建都完成，方便review

af64796

Merge branch 'develop' of github.com:seata/seata into dev_4572_new_fe…

a7800e2

…ild_null

caohdgege reviewed Jul 25, 2022

View reviewed changes

rm-datasource/src/main/java/io/seata/rm/datasource/sql/struct/TableRecords.java Outdated Show resolved Hide resolved

Bughue and others added 5 commits July 26, 2022 14:19

remove t-w-r

1679535

Merge branch 'develop' into dev_4572_new_feild_null

d6b8610

Merge branch 'develop' of github.com:seata/seata into dev_4572_new_fe…

fccf6a9

…ild_null

Merge branch 'develop' into dev_4572_new_feild_null

13c8601

Merge branch 'develop' into dev_4572_new_feild_null

6faf8e7

Bughue and others added 5 commits December 14, 2022 16:17

Merge branch 'develop' into dev_4572_new_feild_null

fbade58

fix merge

329618c

fix test

4056fc2

fix test

9397cca

Merge branch 'develop' into dev_4572_new_feild_null

b2d712f

funky-eyes mentioned this pull request Feb 2, 2023

Refresh tableMeta when SQL throw a missing field exception #5292

Closed

Merge branch 'develop' into dev_4572_new_feild_null

2689577

funky-eyes reviewed Feb 3, 2023

View reviewed changes

Merge branch 'develop' into dev_4572_new_feild_null

c3de060

funky-eyes added this to the 1.7.0 milestone Feb 3, 2023

Bughue and others added 8 commits February 15, 2023 11:26

tableMeta refresh event

ccf9ecf

check style

e8b9a6b

check style

c901589

currentTimeMillis

072f609

check style

026f1b5

Merge branch 'develop' into dev_4572_new_feild_null

996bfb5

comment

1ac5bf2

Merge branch 'develop' into dev_4572_new_feild_null

711e659

slievrly reviewed Feb 21, 2023

View reviewed changes

rm-datasource/src/main/java/io/seata/rm/datasource/DataSourceProxy.java Outdated Show resolved Hide resolved

Bughue and others added 10 commits February 22, 2023 14:08

comment

eb47a94

Merge remote-tracking branch 'origin/dev_4572_new_feild_null' into de…

332081b

…v_4572_new_feild_null

fix test

ef87ff3

Merge branch 'develop' into dev_4572_new_feild_null

3b85ba1

fix test

de81450

Merge remote-tracking branch 'origin/dev_4572_new_feild_null' into de…

155c5f0

…v_4572_new_feild_null

fix test

a8d8781

fix test

febc6a0

Merge branch 'develop' into dev_4572_new_feild_null

9672177

Merge branch 'develop' into dev_4572_new_feild_null

16ca49a

slievrly reviewed Feb 27, 2023

View reviewed changes

Bughue added 3 commits February 28, 2023 10:40

optimize

8b22aef

Merge remote-tracking branch 'origin/dev_4572_new_feild_null' into de…

5b4e206

…v_4572_new_feild_null

optimize

96bae53

slievrly approved these changes Feb 28, 2023

View reviewed changes

slievrly merged commit 28b3413 into apache:develop Feb 28, 2023

Bughue added a commit to Bughue/seata that referenced this pull request Feb 28, 2023

bugfix : check if table meta cache should be refreshed in AT mode (ap…

0f1f8af

…ache#4734) (cherry picked from commit 28b3413)

Bughue mentioned this pull request Feb 28, 2023

bugfix : check if table meta cache should be refreshed in AT mode #5395

Merged

1 task

l81893521 pushed a commit to l81893521/seata that referenced this pull request Mar 28, 2023

bugfix : check if table meta cache should be refreshed in AT mode (ap…

029f35d

…ache#4734)

wuwen5 reviewed Jul 9, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bugfix : check if table meta cache should be refreshed in AT mode #4734

bugfix : check if table meta cache should be refreshed in AT mode #4734

Bughue commented Jun 30, 2022 •

edited

Loading

codecov-commenter commented Jul 14, 2022 •

edited

Loading

CLAassistant commented Dec 12, 2022 •

edited

Loading

CLAassistant commented Dec 12, 2022

slievrly Feb 27, 2023

slievrly Feb 27, 2023

slievrly left a comment

wuwen5 Jul 9, 2024

wuwen5 Jul 9, 2024

		@@ -55,6 +59,8 @@ public class TableRecords implements java.io.Serializable {

		private List<Row> rows = new ArrayList<Row>();

		private static final Interner<String> TABLE_NAME_POOL = Interners.newWeakInterner();

bugfix : check if table meta cache should be refreshed in AT mode #4734

bugfix : check if table meta cache should be refreshed in AT mode #4734

Conversation

Bughue commented Jun 30, 2022 • edited Loading

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

codecov-commenter commented Jul 14, 2022 • edited Loading

Codecov Report

CLAassistant commented Dec 12, 2022 • edited Loading

CLAassistant commented Dec 12, 2022

slievrly Feb 27, 2023

Choose a reason for hiding this comment

slievrly Feb 27, 2023

Choose a reason for hiding this comment

slievrly left a comment

Choose a reason for hiding this comment

wuwen5 Jul 9, 2024

Choose a reason for hiding this comment

wuwen5 Jul 9, 2024

Choose a reason for hiding this comment

Bughue commented Jun 30, 2022 •

edited

Loading

codecov-commenter commented Jul 14, 2022 •

edited

Loading

CLAassistant commented Dec 12, 2022 •

edited

Loading