*: node crash with OOM despite memory limits set #20609
Comments
|
Now : |
|
Hi @guotie, what workload are you running? The |
|
it finally crashed |
|
|
|
Now, the cockroach panic on connect , the process was killed by os because of out of memory when client connect to server. |
|
Hi @guotie, what workload were you running against the server? It's possible that you need to devote more RAM to your instance to prevent OOM errors. |
|
@guotie in addition to Jordan's question, I would like to remind you it is unsafe to have the sum of the RocksDB cache size and the max SQL memory exceed 50-60% of your total available RAM. This is because CockroachDB also uses data in memory in addition to these two amounts. |
petermattis
added
the
C-investigation
|
Closing this as unactionable. If you see something like this again, please check the |
jordanlewis
added
the
X-unactionable
|
@jordanlewis I meet the same question. my server has 96G RAM cockroachdb config; PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND just now the network is bad and can't upload it here. |
|
Can you upload it to this issue? It shouldn't be too big. Or email it to me [email protected] |
|
@jordanlewis upload it just now! thank you. and i also send it to the email. |
knz
added
O-community
|
@victorggsimida, what workload are you running to get this issue? There is a very large amount of memory tied up in the SQL parser. Are you running queries with very large strings?
|
the picture you show means that parse use 12G RAM. But the total memory used by cockroachdb is Where the rest memory use for ?
I user haproxy to balance query to different cockroachdb node. but only this node has memory problem |
|
I'll bet on long-lived connections with boatloads of prepared queries, never released -- the ASTs / input syntax strngs are preserved but never reused. Maybe we could test that by placing a limit on the max number of prepared statements in a session. (Postgres has one - we could use the exact same) |
|
@victorggsimida please explain how the client is sending the insert SQL to the haproxy. Also please share haproxy configuration. How long does it keep the connection open? |
|
@knz client connect the ip+1234, with the threading pool developed by ourselves. the connect is long connection; |
|
See my comment above. I think your connection pool is using prepared statements and never releases the prepared statements. That would be the cause of the large memory usage. |
|
@victorggsimida please check http://initd.org/psycopg/articles/2012/10/01/prepared-statements-psycopg/ -- are you doing anything similar to this? |
|
@knz
no . absolutely no. did not user Prepare in my sqls. I use like this: no Prepare in my SQLs. |
|
@knz hi, is there more information I need to provide ? |
|
I don't have further questions at that time but perhaps Jordan has. |
|
@jordanlewis hi Is there any information i need to provide to solve |
|
@victorggsimida how many concurrent connections do you have in your app? |
|
@jordanlewis concurrent connections is set to 5 and 60 seperately. but the problem is existed in these two concurrency. And the memory used by cockroachdb is always in the specific one node. when I start the app (extract data from mysql, and load data into cockrochdb with upsert sql), the problem happens |
|
Please share the exact syntax of the upsert statements, and also the output of |
|
UPSERT INTO {db}.{table} ({fields},synctime) VALUES ({value_fields},'{now}') |
|
How many value fields do you have per statement? How large is the table you are inserting into? Are you running upsert between begin..commit, or as individual statements? |
|
|
@knz |
|
Ok so a transaction with "tens of thousands" of values in a single mutation statement will not work and will cause various of the problems you observe, including server crashes if you do this multiple times in concurrent connections. We know of this problem and the workaround is simple: modify/insert fewer rows/values at a time! What about you try again with a max of 100-1000 values per statement, and a max of 10000 rows modified per transaction. Would that work better? |
|
@knz
|
|
@knz I still have one question: why always one node with memory problem? because I use haproxy as load balance. and haproxy installed in another machine not in the cluster |
|
You need to check in the web ui the number of SQL transactions/statements per node. I suspect that one node is taking more than its fair share despite the load balancer. |
|
Closing again as unactionable. If you have a new problem we can help you with, please open a new issue and we'll take a look! |
|
We are facing the same problem, looks like cockroachdb can't limit its memory usage. |
|
@coderplay many things have changed since 2017. You will need to provide much more information for us to be able to help you. You can open a new issue if you have a reproducible issue. Please include information like your schema, workload, and data in the heap_profiler directory of your log output. Thanks! |
|
@jordanlewis The use case and the failures are almost the same as what it was in 2017. Do you suggest posting the information here, or in another thread? |
|
You can post here. If you have something reproducible, open a new thread. |
|
@jordanlewis just posted on #76282 |
Is this a question, feature request, or bug report?
QUESTION
Have you checked our documentation at https://cockroachlabs.com/docs/stable/? If you could not find an answer there, please consider asking your question in our community forum at https://forum.cockroachlabs.com/, as it would benefit other members of our community.
Prefer live chat? Message our engineers on our Gitter channel at https://gitter.im/cockroachdb/cockroach.
BUG REPORT
Please supply the header (i.e. the first few lines) of your most recent
log file for each node in your cluster. On most unix-based systems
running with defaults, this boils down to the output of
grep -F '[config]' cockroach-data/logs/cockroach.log
I set memory usage less than 30%, but now , cockroach use 70% memory
The text was updated successfully, but these errors were encountered: