SQLite Locking Fails in WSL: OperationalError locking protocol

[mikelococo]

Describe the bug

When running under the Windows 10 Windows Subsystem for Linux (WSL), sqlite locking errors are encountered and RMD hangs or exits: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) locking protocol

There's a WSL bug around some SQlite problems, specifically with WAL: microsoft/WSL#2395. Though I tried setting session().execute("PRAGMA journal_mode=DELETE") and also tried TRUNCATE in redditdownloader/sql/__init__.py and it made no difference.

I'm not sure there's really anything you can do about this, I file this bug less because it's likely to result in a fix in the RMD source code, and more to help other users that that hit this issue so they have something to search. I suppose maybe you could try to detect WSL and warn pointing to this bug.

Environment Info

• OS: Ubuntu 18.04 in WSL in Windows 10 Pro 1903, Python 3.6.8
• RMD Version: master at b9c7f81

Screenshots/Information

I set "concurrent_downloads": 1 to reduce output noise and also try to reduce the probability of locking issues.

$ python redditdownloader

====================================
   Reddit Media Downloader 3.0.0
====================================
    (By ShadowMoose @ Github)

Loaded settings file [/home/mike/.local/share/RMD/settings.json].
Loaded Source:  MyRedditUser Saved
Remaining: 0/0+

File: None
  Handler: None
   Status: Starting up...

Authenticating via OAuth...
Authenticated as [MyRedditUser]

Remaining: 0/0+

File: None
  Handler: None
   Status: Starting up...

Remaining: 0/0+

File: None
  Handler: None
   Status: Starting up...

Remaining: 0/0+

File: Process RedditElementLoader:
None
  Handler: None
   Status: Starting up...

Traceback (most recent call last):
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 580, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: locking protocol

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "redditdownloader/processing/redditloader.py", line 30, in run
    self.load()
  File "redditdownloader/processing/redditloader.py", line 47, in load
    self._scan_sources()
  File "redditdownloader/processing/redditloader.py", line 67, in _scan_sources
    post = self._session.query(sql.Post).filter(sql.Post.reddit_id == r.id).first()
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3240, in first
    ret = list(self[0:1])
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3018, in __getitem__
    return list(res)
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3342, in __iter__
    return self._execute_and_instances(context)
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3367, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
    return meth(self, multiparams, params)
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
    distilled_params,
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1253, in _execute_context
    e, statement, parameters, cursor, context
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1473, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 152, in reraise
    raise value.with_traceback(tb)
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1249, in _execute_context
    cursor, statement, parameters, context
  File "/home/mike/code/reddit-media-downloader3/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 580, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) locking protocol
[SQL: SELECT posts.reddit_id AS posts_reddit_id, posts.author AS posts_author, posts.type AS posts_type, posts.title AS posts_title, posts.body AS posts_body, posts.parent_id AS posts_parent_id, posts.subreddit AS posts_subreddit, posts.over_18 AS posts_over_18, posts.created_utc AS posts_created_utc, posts.num_comments AS posts_num_comments, posts.score AS posts_score, posts.source_alias AS posts_source_alias
FROM posts
WHERE posts.reddit_id = ?
 LIMIT ? OFFSET ?]
[parameters: ('t3_dj986s', 1, 0)]
(Background on this error at: http://sqlalche.me/e/e3q8)
All finished.

Additional context

Disabling WAL doesn't seem to help, despite what is suggested in the WSL issue. Probably nothing can be done here other than warn if WSL is detected in the platform, but having a bug report around (even if its closed wontfix) might help others debug the problem more quickly.

[shadowmoose]

Thanks for the report.
I'll look into this, but on the surface it does seem like something outside of RMD's scope. At the least, if I can detect WSL, it's probably a good idea to warn about it. Having the issue documented here is a good plan as well.

[shadowmoose]

Okay, so it looks like that bug may not be fixable from the Python side. Because of this, I unfortunately won't be able to make RMD support WSL at this time.

As of e262e74 I've added a warning on the user-friendly Run.py startup script to help prevent database corruption.

Anybody who runs into the issue will hopefully either find this report, or will be redirected here. If it becomes possible to patch this issue through RMD, I'm happy to revisit this problem.