MySQL server has gone away #121

moneroexamples · 2019-01-12T04:01:58Z

# ERR: SQLException in /home/mwo2/openmonero/src/MySqlAccounts.cpp(select) on line 258
# ERR: MySQL server has gone away
# ERR: SQLException in /home/mwo2/openmonero/src/MySqlAccounts.cpp(insert) on line 285
# ERR: MySQL server has gone away

Sometimes it happens. Don't know yet why.

Previously had simillar problem:

#10

The text was updated successfully, but these errors were encountered:

italocoin-project · 2019-01-21T19:34:26Z

This could be fixed with force connect check this maybe can help, i haven't encountered this issue maybe because of this

italocoin-project@d34cf46

moneroexamples · 2019-01-21T21:40:41Z

Thx. Will check that.

moneroexamples · 2019-01-22T00:59:59Z

Thanks. I checked the code and this may work, and will reconnect you. But the problem I see is that the query data will be lost. The code that is calling these queries will not try to re-call it again. So you may end up with missing outputs or inputs in your database after reconnecting in catch part.

I think for reconnection, the better place would be before the query is executed, i.e., where conn->check_if_connected();. So maybe this could be modified, so that it reconnects.

There can also be other reasons why the connection is failing, e.g., default settings in mysql/mariadb being inadequate.

I will try to investigate what's happening. Maybe mysql/mariadb logs will some info about that.

italocoin-project · 2019-01-22T01:56:52Z

No problem, i see, this could bring up some problems if it would miss outputs or inputs from mysql DB, but then, if the input gets missed, on a rescan it should be reeaded, because the rescan will find the imputs/outputs from blockchain's db, isn't it? let me know if you find a better way to deal with timeouts, thanks :)

moneroexamples · 2019-01-22T02:35:51Z

no problem and yes, would have to rescan to find missing txs/outputs/inputs.

I just setup regular mariadb on my linux box, instead of using docker, and will try to get some logs out of it.

moneroexamples · 2019-01-22T21:57:23Z

I added ReconnectOption to mysql, so now it will reconnect when connection drops.

openmonero/src/db/MySqlAccounts.cpp

Lines 228 to 229 in 9e3b72f

    
           conn = make_shared<MySqlConnector>( 
        
                   new mysqlpp::ReconnectOption(true));

There still can be a problem, but at least connection should not be dropped permanently. If I manually kill the connection in mariadb I get this, but then it reconnects:

# ERR: SQLException in /home/mwo2/openmonero/src/db/MySqlAccounts.cpp(select) on line 267                                                                                                                                                                                   
# ERR: Lost connection to MySQL server during query

In the current devel branch I also added Ctrl+C support to greacfully stop openmonero, and the logging got extended.

italocoin-project · 2019-01-22T23:48:32Z

You think this option is better then force connect that i've mentioned above?
nice work with Ctrl+C support, great job man.

I have another issue: MDB_READERS_FULL limit
basicaly when i get MANY connections i get the MDB_READERS_FULL limit under heavy load the db leaks readers or something. Did you encountered this problem?

moneroexamples · 2019-01-23T00:28:05Z

You think this option is better then force connect that i've mentioned above?

I think so. Its directly implemented in mysql++ and propagated to mysql. It's also recommended in mysql++ docs regarding the timeouts: https://tangentsoft.com/mysqlpp/doc/html/userman/tutorial.html#conn-timeout

Will see how it will work. Can always remove it if its not helpful.

I have another issue: MDB_READERS_FULL limit

I had it maybe only once. It was also reported before: #97

This is one reason why Ctr+C was added. With it, openmoenro will close the lmdb's blockchain database that it opens for reading. You can see this when you run openmonero with -m 3 (level 3 log for monero). This will show you that database is closed properly now when Ctrl+C is detected:

2019-01-23 00:16:01.732     7f8f8095ac00        TRACE   blockchain      src/cryptonote_core/blockchain.cpp:547  Blockchain::deinit
2019-01-23 00:16:01.732     7f8f8095ac00        TRACE   blockchain      src/cryptonote_core/blockchain.cpp:549  Stopping blockchain read/write activity
2019-01-23 00:16:01.732     7f8f8095ac00        TRACE   blockchain.db.lmdb      src/blockchain_db/lmdb/db_lmdb.cpp:1448 BlockchainLMDB::close
2019-01-23 00:16:01.732     7f8f8095ac00        TRACE   blockchain.db.lmdb      src/blockchain_db/lmdb/db_lmdb.cpp:1464 BlockchainLMDB::sync
2019-01-23 00:16:01.733     7f8f8095ac00        TRACE   blockchain      src/cryptonote_core/blockchain.cpp:564  Local blockchain read/write activity stopped successfully
2019-01-23 00:16:01.733     7f8f8095ac00        TRACE   blockchain.db.lmdb      src/blockchain_db/lmdb/db_lmdb.cpp:1178 BlockchainLMDB::~BlockchainLMDB

When you force close it, openmonero has no chance to close the lmdb which it opens and it seems to keep accumulating the open connections. Before I just deleting lmdb.lck (e.g. in home/mwo/.bitmonero/lmdb/lock.mdb) and it solved the problem in my case so that new one is created. Not sure about this though.

Similar issue was also reported before for monero monero-project/monero#2741 with suggested solution.

when i get MANY connections

How many is MANY?

italocoin-project · 2019-01-23T00:36:57Z

How many is MANY?

Send more than 126 different login request to openmonero and you will get the issue, let me know

yes, now it actually closes properly, what if it gets killed?

moneroexamples · 2019-01-23T00:47:15Z

126 different login request to the mywallet

I see. Probably will need to rethink how lmdb is accessed in openmonero. At the moment, there will be 126 search tx threads accessing the lmdb at the same time. So maybe will have to implement some pooling when only few threads access lmdb at the same time, or move to using rpc to deamon instead of accessing lmdb directly.

what if it gets killed?

Thank it will be like before, all connections, threads will be terminated immedietly. I think it's better to let the all threads, mysql and lmdb close properly when openmonero stops.

italocoin-project · 2019-01-23T00:54:11Z

or move to using rpc to deamon instead of accessing lmdb directly

rpc to daemon could solve the problem for good.

maybe will have to implement some pooling when only few threads access lmdb at the same time

It means that someone will have to wait until the threads gets freed again, i think we should test them both

moneroexamples · 2019-01-23T01:00:59Z

rpc to daemon should also be much easier to implement, as the second possible solution would require major changes to architecture of openmonero. But rpc will also wont be simultaneous, as is needs also synchronization.

italocoin-project · 2019-01-23T01:15:09Z

rpc to daemon should also be much easier to implement, as the second possible solution would require major changes to architecture of openmonero.

I agree

But rpc will also not be simultaneous, as is needs also synchronization.

Thats also true, but i think the best solution is the rpc to daemon, how is mymonero handling with many connections? I assume that they have over 126 consecutive login requests

moneroexamples · 2019-01-23T01:28:26Z

mymonero handling with many connections?

Don't know. Their backend is closed sourced. But from what I know they use go and mysql, so I think rpc also would be easiest.

But for mass scale, applications such as openmonero/mymonero are "easily" parallelized, as you can spin up, e.g., 10 instances of the backend and daemons, on 10 servers. One possible way of doing this with some details has already been proposed for openmonero: #95 (comment)

italocoin-project · 2019-01-23T01:40:53Z

so I think rpc also would be easiest.

Yes, that would be the best option, do you plan to implement RPC to DAEMON soon? And about more instances, that would be a good idea too, a system to sync all mysql db on all instances would solve the panic in users if they would see that the wallet syncs from 0 again

moneroexamples · 2019-01-23T01:47:36Z

Yes, that would be the best option, do you plan to implement RPC to DAEMON soon?

Yes, though "soon" in monero world does not mean anything:-) But yes, this and mysql-thing will be the issues I will be focusing the most at present. When complete? No timeframe on this. Hopefully "soon":-)

Just spin up new branch for that: https://github.com/moneroexamples/openmonero/tree/new_rpc So all new changes will be there for experimentation.

italocoin-project · 2019-01-23T01:54:02Z

Nice, i will merge it tomorrow morning and test out the branch, i will let you know if i face issues, thanks :)

moneroexamples · 2019-01-23T01:57:59Z

Not sure I will have there anything new for tomorrow, but this is the branch when the rpc changes will be taking place.

No problem. Good you can provide extra info and much needed testing for openmonero. Thanks.

italocoin-project · 2019-01-23T02:18:36Z

Thank you for your hard work, i really like your projects and work, you can count on my help for testing out, i use openmonero for my project to, thanks for that :)

moneroexamples · 2019-01-23T06:06:11Z

I made a seprate issue for the LMDB readers limit: #127

moneroexamples mentioned this issue Jan 23, 2019

MDB_READERS_FULL: Environment maxreaders limit reached #127

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MySQL server has gone away #121

MySQL server has gone away #121

moneroexamples commented Jan 12, 2019 •

edited

Loading

italocoin-project commented Jan 21, 2019

moneroexamples commented Jan 21, 2019

moneroexamples commented Jan 22, 2019 •

edited

Loading

italocoin-project commented Jan 22, 2019

moneroexamples commented Jan 22, 2019

moneroexamples commented Jan 22, 2019

italocoin-project commented Jan 22, 2019 •

edited

Loading

moneroexamples commented Jan 23, 2019 •

edited

Loading

italocoin-project commented Jan 23, 2019 •

edited

Loading

moneroexamples commented Jan 23, 2019

italocoin-project commented Jan 23, 2019

moneroexamples commented Jan 23, 2019 •

edited

Loading

italocoin-project commented Jan 23, 2019

moneroexamples commented Jan 23, 2019

italocoin-project commented Jan 23, 2019

moneroexamples commented Jan 23, 2019

italocoin-project commented Jan 23, 2019

moneroexamples commented Jan 23, 2019 •

edited

Loading

italocoin-project commented Jan 23, 2019

moneroexamples commented Jan 23, 2019

MySQL server has gone away #121

MySQL server has gone away #121

Comments

moneroexamples commented Jan 12, 2019 • edited Loading

italocoin-project commented Jan 21, 2019

moneroexamples commented Jan 21, 2019

moneroexamples commented Jan 22, 2019 • edited Loading

italocoin-project commented Jan 22, 2019

moneroexamples commented Jan 22, 2019

moneroexamples commented Jan 22, 2019

italocoin-project commented Jan 22, 2019 • edited Loading

moneroexamples commented Jan 23, 2019 • edited Loading

italocoin-project commented Jan 23, 2019 • edited Loading

moneroexamples commented Jan 23, 2019

italocoin-project commented Jan 23, 2019

moneroexamples commented Jan 23, 2019 • edited Loading

italocoin-project commented Jan 23, 2019

moneroexamples commented Jan 23, 2019

italocoin-project commented Jan 23, 2019

moneroexamples commented Jan 23, 2019

italocoin-project commented Jan 23, 2019

moneroexamples commented Jan 23, 2019 • edited Loading

italocoin-project commented Jan 23, 2019

moneroexamples commented Jan 23, 2019

moneroexamples commented Jan 12, 2019 •

edited

Loading

moneroexamples commented Jan 22, 2019 •

edited

Loading

italocoin-project commented Jan 22, 2019 •

edited

Loading

moneroexamples commented Jan 23, 2019 •

edited

Loading

italocoin-project commented Jan 23, 2019 •

edited

Loading

moneroexamples commented Jan 23, 2019 •

edited

Loading

moneroexamples commented Jan 23, 2019 •

edited

Loading