Internal error: database is locked (repo.go / serv.go)

[cookiengineer]

• Gitea version (or commit ref): 1.1.1 built with: bindata, sqlite
• Git version: 2.13.0 on client and 2.12.2 on server
• Operating system: Arch Linux to Arch Linux ARM
• Database: SQLite
• Can you reproduce the bug at https://try.gitea.io: No
• Log gist:

Description

When trying to push sometimes gitea answers with:

[$] git push backup development -f
Gitea: Internal error
UpdatePublicKey: database is locked
error: failed to push some refs to 'git@git-mirror:Artificial-Engineering/lycheejs.git'

When checking gitea via web interface everything seemed normal, no errors shown in dashboard or admin interface.

Afterwards I restarted the gitea service (using systemctl stop gitea; systemctl start gitea) and everything seems to work nicely again.

The gitea.log file has these entries (literally hundreds of them):

2017/05/20 14:34:20 [I] Log Mode: File(Info)
2017/05/20 14:34:20 [I] XORM Log Mode: File(Info)
2017/05/20 14:34:20 [I] Cache Service Enabled
2017/05/20 14:34:20 [I] Session Service Enabled
2017/05/20 14:34:21 [I] Git Version: 2.12.2
2017/05/20 14:34:21 [I] SQLite3 Supported
2017/05/20 14:34:21 [I] Run Mode: Production
(...)
2017/05/20 14:34:21 [...gitea/models/repo.go:2042 CheckRepoStats()] [E] Update repository count 'num_watches'[2]: database table is locked: repository
(... literally dozens of times the same message ... )
2017/05/20 14:34:21 [...gitea/models/repo.go:2042 CheckRepoStats()] [E] Update repository count 'num_watches'[60]: database table is locked: repository
2017/05/20 14:34:23 [I] Listen: http://0.0.0.0:3000

The serv.log contains this entry around the same time:

2017/05/10 19:38:05 [T] No update task is presented: 7565f22d-5db4-4d34-b071-ae2b77a8fa46
2017/05/20 14:32:14 [...io/gitea/cmd/serv.go:318 runServ()] [F] UpdatePublicKey: database is locked
2017/05/20 14:32:48 [...io/gitea/cmd/serv.go:318 runServ()] [F] UpdatePublicKey: database is locked

The xorm.log file looks pretty normal, no errors, just some SQL queries and everything looks like when it worked afterwards.

I thought I report this bug anyways, because I'm unsure whether something really painful can happen in future in case this happens again.

[typeless]

Complementary information:

https://gist.github.com/mrnugget/0eda3b2b53a70fa4a894

[bkcsoft]

Could you test if this fixes the issue? #1961 (applying the diff and building on v1.1.2 should be straight forward)

[typeless]

@bkcsoft This bug is hard to reproduce. My Gitea server has only got one per month since April. There were 10 cases in March though. Of course, the numbers have something to do with the variations of the user activities as well.

It would be nice if there is a test that can reliably trigger the problem.

[cookiengineer]

@bkcsoft: Same for me, I cannot reliably trigger this problem. It's pretty random when it happens, though I suspect that in my case it's related to turning off the Raspberry Pi's power while it has a job/process running.

[bkcsoft]

One way of triggering it could be to have 10 users create PRs to the same repo at the same time.

[typeless]

I think we should have the xorm library protect the db connection with a lock for SQLite. SQLite doesn't provide out-of-box multithreading safety for shared-single-connection-over-shared-cache scenario, which is how it is configured in Gitea. Otherwise Gitea would probably need to initiate a connection for every goroutine.

P.S. The 'database table is locked' error can be reproduced, quite reliably, by our integration tests already.

[lafriks]

I agree that xorm library should protect sqlite connection from multiple threads

[lunny]

@typeless @lafriks you are welcome to open issue or send PR to xorm to do that.

[lunny]

continue via #2040