sqlite: set busy timeout to 100s

[Luap99]

Only one process can write to the sqlite db at the same time, if another process tries to use it at that time it fails and a database is locked error is returned. If this happens sqlite should keep retrying until it can write. To do that we can just set the _busy_timeout option. A 100s timeout should be enough even on slower systems but not to much in case there is a deadlock so it still returns in a reasonable time.

[NO NEW TESTS NEEDED] I think we strongly need to consider some form of parallel stress testing to catch bugs like this.

Fixes #20809

Does this PR introduce a user-facing change?

Fix `database is locked` errors with the new sqlite database backend.

[Luap99]

@mheon @vrothberg @edsantiago PTAL

[edsantiago]

Now testing in #17831

PS thank you

[vrothberg]

I believed that one of the options above took care of setting a default busy value already but I may be wrong.

Changes LGTM. Did you manage to manually reproduce and validate the fix is solving it?

[giuseppe]

LGTM

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: giuseppe, Luap99

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Luap99,giuseppe]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Luap99]

Did you manage to manually reproduce and validate the fix is solving it?

Yes using the reproducer from the issue.

[mheon]

I'm going to drop a
/lgtm
/hold

Even if this does turn out to not be a complete fix for some reason I see no harm in merging it.

[giuseppe]

/hold cancel

[jpalus]

One question though -- what "writing to the database" mean exactly? Is it like single update statement, single transaction, single "connection"? In other words what needs to fit within those 100s? If I have a big container with lots of files and I do podman rm on it which takes > 100s does that mean no write can occur before it, but only afterwards?

[mheon]

From the SQLite locking model, this should be a single transaction. Connections are allowed to be concurrent, locks are only taken on write transactions.

[Luap99]

One question though -- what "writing to the database" mean exactly? Is it like single update statement, single transaction, single "connection"? In other words what needs to fit within those 100s? If I have a big container with lots of files and I do podman rm on it which takes > 100s does that mean no write can occur before it, but only afterwards?

To be clear here the lock is only taken for db writes, those should generally be fast. We do not have the db locked while we delete the container storage for example. The db stores only "metadata" so the data should not be to big and operations should be fast.