session: add a batch commit session variable for the large transaction #8293
Conversation
jackysp
added
type/enhancement
|
I'm afraid if a transaction use too many statements, TiDB OOM quickly and this config can't help. |
|
I remember that the memory amplification is terrible, and count = 5000 is big enough for most cases |
| // The last history could not be "commit" statement. | ||
| // It means it is impossible to start a new transaction at the end of the transaction (the "commit" statement). | ||
| // Because after the server executed "commit" statement, the session is out of the transaction. | ||
| se.sessionVars.SetStatusFlag(mysql.ServerStatusInTrans, true) |
There was a problem hiding this comment.
What will happen if the last statement is a rollback, will there be a corner case that this line put the session into a new transaction while it should be rollbacked ?
There was a problem hiding this comment.
rollback is the same as commit.
There was a problem hiding this comment.
I will update the comment and add a rollback test case.
There was a problem hiding this comment.
s1:
set @@autocommit = 0;
insert into t value (1);
rollback;
s2:
insert into t values (1);
insert into t values (2);
s1:
select * from t; // what it see ?
There was a problem hiding this comment.
So the statement count limit is 1 for this case?
|
I've tag a requre LGTM3 to this PR. |
|
LGTM |
1 similar comment
|
IMHO, it's better let Just as description above, this config only useful in prepare data situation and only can be "safe" use in prepare data situation. make variable in server level will be problem, just imagine: we have a tidb running under heavy online transaction processing, now we want add a table and using a "import program" to import some data in new table. we just want to import fast in import program's sql, the sql from online processing should never affect by So, we should better use session variable, and set it in program's database dsn jdbc:mysql://localhost:4000/db?characterEncoding=UTF-8&sessionVariables=batch_commit=100root@tcp(localhost:4001)/db?strict=true&batch_commit=100 |
|
PTAL @lysu |
|
/run-all-tests |
lysu
added
status/LGT2
There was a problem hiding this comment.
I think this config option is risky enough and need another LGTM @shenli
|
/run-common-test |
jackysp
changed the title
1 similar comment
|
@jackysp CI failed. |
|
/run-all-tests |
tiancaiamao
added
status/LGT3
What problem does this PR solve?
tidb_batch_insert / tidb_batch_delete only split the statement into several batches. If the transaction contains many statements, TiDB could not commit it in batch.
What is changed and how it works?
Add a session level variable to control if the transaction should return an error or start a new transaction when it exceeds the statements limit of the transaction. It will be useful when importing data.
Check List
Tests
TPC-C data importing test works fine on it.
Code changes
Related changes
PTAL @tiancaiamao @lysu
This change is