-
Notifications
You must be signed in to change notification settings - Fork 63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to access lf.source_directory()
in federated execution
#2378
Comments
This is one of the reasons the feature was questioned in the first place. The circumstances under which you can possibly know where your source directory is located are very limited. For one, this won't work if the binary gets transferred to a remote system and the sources are not. It also won't work if you create a Docker image that doesn't make the sources available in the runner stage. If the path is absolute, it won't work in someone else's file system. If it is relative, it won't work if you move the binary. Etc... I honestly still doubt there is a way to make this "work" universally, so likely we'll have to narrow down scope. We had a discussion about this recently, and I recall @edwardalee advocated that the feature "as-is" was incomplete but useful enough in the near term to be merged and could be fixed later. Could I ask what are you trying to accomplish with this feature? |
In this specific case, I'm trying to install a the custom python serializer package using pip in the preamble (#2375). This would require locating the python package on the file system and installing it with pip. However the python package path could only be relative, because we want the path to work in different environments. I'm trying to use However I have encountered the need for something similar to
In these applications I used workarounds such as changing the operating directory with |
For Python there is already established, well-thought-out, and hardened tooling for all of this. The key for solving this issue would be a proper integration with the Python ecosystem (in particular packaging) and using |
The
lf.source_directory()
cannot be accessed in federated execution.Example program:
Error:
The text was updated successfully, but these errors were encountered: