feat: add GLOBAL_WORKING_DIR and GLOBAL_WORKING_PROCESS_DIR config parameteres #3014
Conversation
amadeusz-ds
requested review from
badGarnet,
cragwolfe,
micmarty-deepsense,
qued and
pawel-kmiecik
|
|
||
| @property | ||
| def STORAGE_DIR(self) -> str: | ||
| """Path to Unstructured storage directory.""" |
There was a problem hiding this comment.
STORAGE_DIR is a misleading name, which has permanent or at least caching connotations. could this instead be TMP_STORAGE_DIR?
There was a problem hiding this comment.
There are parameters STORAGE_DIR and STORAGE_TMPDIR
There was a problem hiding this comment.
Maybe UNSTRUCTURED_DIR and UNSTRUCTURED_TMPDIR as those by default point to ~/.cache/unstructured and ~/.cache/unstructured/tmp/{gid} respectively
There was a problem hiding this comment.
or UNSTRUCTURED_CACHE_DIR and UNSTRUCTURED_TMP_DIR?
There was a problem hiding this comment.
made the changes, waiting for greenlight @cragwolfe
…-usage-of-temporary-storage
amadeusz-ds
changed the title
| @@ -160,7 +160,6 @@ def _try_process_document(self, doc: Path) -> Optional[list]: | |||
| @abstractmethod | |||
| def _process_document(self, doc: Path) -> list: | |||
| """Should return all metadata and metrics for a single document.""" | |||
| pass | |||
There was a problem hiding this comment.
It was removed by the linter
There was a problem hiding this comment.
If a docstring is added, the pass keyword is optional for functions.
…-usage-of-temporary-storage
amadeusz-ds
changed the title
| @@ -161,7 +163,12 @@ def test_save_elements_with_output_dir_path_none(): | |||
| ) | |||
|
|
|||
| # Verify that the images are saved in the expected directory | |||
| expected_output_dir = os.path.join(tmpdir, "figures") | |||
| if storage_enabled: | |||
There was a problem hiding this comment.
personally I'd prefer to see more usages of pathlib, but I believe it's not the mail goal of this PR, just a side note
This PR introduces GLOBAL_WORKING_DIR and GLOBAL_WORKING_PROCESS_DIR controlling where temporary files are stored during partition flow, via tempfile.tempdir.
Edit:
Renamed prefixes from STORAGE_ to UNSTRUCTURED_CACHE_
Edit 2:
Renamed prefixes from UNSTRUCTURED_CACHE to GLOBAL_WORKING_DIR_