Possible issue with open method when using rocksdict #573
Comments
|
Here's the solution I implemented. Let me know if this is bad/wrong and whether it's a fix that needs to be implemented. def open(self, path: Path, *, read_only: bool = False) -> DB:
"""Open RocksDB database using this configuration."""
if self.use_rocksdict:
db_options = self.as_options()
db_options.set_db_paths(
[rocksdict.DBPath(str(path), self.target_file_size_base)]
)
db_access_type = (
# rocksdict.AccessType.ReadWrite <== This line changes
rocksdict.AccessType.read_write()
if self.ttl is None
else rocksdict.AccessType.with_ttl(self.ttl)
)
db = DB(str(path), options=self.as_options(), access_type=db_access_type)
db.set_read_options(rocksdict.ReadOptions())
return db
else:
return rocksdb.DB(str(path), self.as_options(), read_only=read_only)and to resolve the NoneType nfiles value error we modify the platforms module to check for Windows OS (which can't make use of the resource package) so it returns 1 instead of None. """Platform/OS utilities."""
import platform
import subprocess # nosec: B404
from typing import Optional
def max_open_files() -> Optional[int]:
"""Return max number of open files, or :const:`None`."""
try:
from resource import RLIM_INFINITY, RLIMIT_NOFILE, getrlimit
except ImportError:
if platform.system() == "Windows": # <=== add a check for Windows OS to return something.
return 512
else:
return None
else:
_, hard_limit = getrlimit(RLIMIT_NOFILE)
if hard_limit == RLIM_INFINITY:
# macOS bash always returns infinity, even though there
# is an actual system limit.
if platform.system() == "Darwin":
output = subprocess.check_output( # nosec
[
"sysctl",
"-q",
"kern.maxfilesperproc",
]
)
# $ sysctl -q kern.maxfilesperproc
# kern.maxfilesperproc: 24576
_, _, svalue = output.decode().partition(":")
return int(svalue.strip())
return None
return hard_limit |
|
this seems to suggest that the default is 512. |
|
I'm experiencing the exact same issue |
|
Are you also using Windows, or Linux? If Linux, then the issue with ReadWrite might be universal and is a bug that needs fixing. The |
|
I'm seeing the exact same issue with Linux. |
|
So you're seeing this then? [INFO] Executing _on_partitions_assigned
[INFO] opening partition 2 for gen id 3 app id 3 #<== Numbers here might be different each time
[WARNING] [^----Store: rocksdb:<table_name>]: DB for partition 2 retries timed out
[INFO] Closing rocksdb on stop
[ERROR] [^-App]: Crashed reason=TypeError("argument 'nfiles': 'NoneType' object cannot be interpreted as an integer")That would be interesting, as it would imply the issue for you is coming from the platforms module. I'd suggest adding some debug statements in there (just for yourself) so you can see why it's returning the None. If it's the |
|
Sorry I realise I didn't specify I was running into the second issue in the OP: Additional info: I only see this using v0.10.17. As soon as I install v0.10.16 this problem goes away. |
|
Ah, so it's the ReadWrite one. |
|
Oooo. Interesting that it goes away with v0.10.16. |
Yes this resolved the error |
Indeed, I also have the KeyError, but the KeyError is part of rocksdict expected behaviour when the database partition has not yet been initiated by the library (when faust starts and starts reading the data that's persisted by rocksdb) |
|
Sorry I didn't get to address this sooner, I'll have this patched ASAP. |
Steps to reproduce
Trying to run with store=rocksdb:// on windows
Expected behavior
I expect it can store data in tables and recover quickly when changelogs are long
Actual behavior
Errors with [^----Store: rocksdb:<table name>]: DB for partition 2 retries timed out.
I tracked it down to the fact that the resource module (used in faust/utils/platforms) is a Unix only thing so the
_max_open_filesvariable defaults toNone, which then flows on to a crash because it's trying to set the_max_open_filesondb.options(in faust/stores/rocksdb.py) which does not like a None.Setting this default to 1 instead of None resolved that issue, but then I encountered another issue where the open method tries to call
rocksdict.AccessType.ReadWrite, which doesn't exist.Looking at the
AccessTypemethod shows it doesn't haveReadWrite, but does haveread_write, so I replaced withrocksdict.AccessType.read_write()and it worked.I'm fairly certain there is no issue here except that I'm trying to use this with Windows. But I did get it working, so I'm not sure whether there's something here to explore or this is a big don't.
The text was updated successfully, but these errors were encountered: