How should MemcacheUnexpectedCloseError be handled?

[oseppala]

When the course configuration is loaded from cache we occasionally get MemcacheUnexpectedCloseError. Either memcache is not up of , connection fails or the server was booted.

It would be possible to load the config from file instead of the cache but the current try-catch-else does not catch the memcacheunexpectedcloseerror and thus the method is exited.

gitmanager/access/config.py

Lines 351 to 367 in a3443f6

	# Try cached version.
	try:
	config = cache.get(cache_key)
	except ValueError as e:
	LOGGER.error(f"Failed to get config from cache: {e}")
	else:
	if config and config.is_valid():
	return config
	LOGGER.debug('Loading course "%s"' % (course_key))
	config = CourseConfig.load(course_key, source)
	try:
	cache.set(cache_key, config)
	except ValueError as e:
	LOGGER.error(f"Failed to set config to cache: {e}")

Should we use a retryingclient https://pymemcache.readthedocs.io/en/latest/apidoc/pymemcache.client.retrying.html or just load the config from a file?

stack trace:

Traceback (most recent call last):
File "/gitmanager/builder/builder.py", line 631, in build_course
config = CourseConfig.get(course_key, build_config_source)
File "/gitmanager/access/config.py", line 353, in get
config = cache.get(cache_key)
File "/usr/local/lib/python3.8/dist-packages/django/core/cache/backends/memcached.py", line 75, in get
return self._cache.get(key, default)
File "/usr/local/lib/python3.8/dist-packages/pymemcache/client/hash.py", line 361, in get
return self._run_cmd('get', key, default, default=default, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymemcache/client/hash.py", line 334, in _run_cmd
return self._safely_run_func(
File "/usr/local/lib/python3.8/dist-packages/pymemcache/client/hash.py", line 214, in _safely_run_func
result = func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/pymemcache/client/base.py", line 619, in get
return self._fetch_cmd(b'get', [key], False).get(key, default)
File "/usr/local/lib/python3.8/dist-packages/pymemcache/client/base.py", line 1027, in _fetch_cmd
buf, line = _readline(self.sock, buf)
File "/usr/local/lib/python3.8/dist-packages/pymemcache/client/base.py", line 1440, in _readline
raise MemcacheUnexpectedCloseError()
pymemcache.exceptions.MemcacheUnexpectedCloseError