Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SOQLDataset.shuffle generates an error when used in a macro #560

Open
dcinzona opened this issue Dec 10, 2021 · 1 comment
Open

SOQLDataset.shuffle generates an error when used in a macro #560

dcinzona opened this issue Dec 10, 2021 · 1 comment

Comments

@dcinzona
Copy link

dcinzona commented Dec 10, 2021

If you use the SOQLDataset plugin in a macro, the system attempts to delete the file ( maybe here cleanup()) when the context is closed using TempDirectory().cleanup() - but I'm not sure why it is sending the filename rather than the folder.

If I move this same shuffle declaration into a recipe, not a macro called by the recipe, I no longer get the error.

Recipe
Account.yml

- plugin: snowfakery.standard_plugins.Salesforce.SOQLDataset
- plugin: snowfakery.standard_plugins.Salesforce.SalesforceQuery
- include_file: account.macro.yml
- object: Account
  include: account
  fields:
    __user:
      SOQLDataset.shuffle:
      #SalesforceQuery.random_record:
        fields: Id
        from: User
        where: IsActive = true  LIMIT 10
    OwnerId: ${{__user.Id}}

Macro
account.macro.yml

- plugin: snowfakery.standard_plugins.Salesforce.SOQLDataset
- plugin: snowfakery.standard_plugins.Salesforce.SalesforceQuery
- macro: account
  fields:
    __user:
      SOQLDataset.shuffle:
      #SalesforceQuery.random_record:
        fields: Id, LastName
        from: User
        where: IsActive = true
    OwnerId: ${{__user.Id}}
    Name: ${{__user.LastName}}

I'm guessing that once the macro is loaded, it gets pulled out of memory and the context is closed, and so it's trying to clean up everything - but not sure why it is trying to call cleanup() on the actual queryresults.db rather than the enclosing directory.

Here is the stack trace:

Traceback (most recent call last):
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 820, in _cleanup
    cls._rmtree(name)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 816, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 749, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 627, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 808, in onerror
    cls._rmtree(path)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 816, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 749, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 608, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\Users\GustavoTandeciarz\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 605, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\GUSTAV~1\\AppData\\Local\\Temp\\tmp3tsiuqwe\\queryresults.db'
@dcinzona
Copy link
Author

Looks like this might definitely be windows specific and perhaps related to this
https://bugs.python.org/issue42796

But the cleanup routine could still fail due to a sharing violation from some other process (likely a child process) setting the directory or a child directory as the working directory. Also, a sharing violation when trying to delete a regular file in the tree causes the cleanup routine to fail with NotADirectoryError because the onerror function calls rmtree() if unlink() fails with a PermissionError.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant