Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to access lf.source_directory() in federated execution #2378

Open
Depetrol opened this issue Jul 23, 2024 · 3 comments
Open

Unable to access lf.source_directory() in federated execution #2378

Depetrol opened this issue Jul 23, 2024 · 3 comments
Labels
bug Something isn't working python Related to the Python target

Comments

@Depetrol
Copy link
Collaborator

The lf.source_directory() cannot be accessed in federated execution.
Example program:

target Python {
    coordination: decentralized
}
reactor Server{
reaction (startup){=
    print("LF source directory", lf.source_directory())
=}
}
federated reactor {
    server = new Server()
}

Error:

    print("LF source directory", lf.source_directory())
RuntimeError: LF_SOURCE_DIRECTORY constant is not defined.
@Depetrol Depetrol added bug Something isn't working python Related to the Python target labels Jul 23, 2024
@lhstrh
Copy link
Member

lhstrh commented Jul 24, 2024

This is one of the reasons the feature was questioned in the first place. The circumstances under which you can possibly know where your source directory is located are very limited. For one, this won't work if the binary gets transferred to a remote system and the sources are not. It also won't work if you create a Docker image that doesn't make the sources available in the runner stage. If the path is absolute, it won't work in someone else's file system. If it is relative, it won't work if you move the binary. Etc...

I honestly still doubt there is a way to make this "work" universally, so likely we'll have to narrow down scope. We had a discussion about this recently, and I recall @edwardalee advocated that the feature "as-is" was incomplete but useful enough in the near term to be merged and could be fixed later.

Could I ask what are you trying to accomplish with this feature?

@Depetrol
Copy link
Collaborator Author

In this specific case, I'm trying to install a the custom python serializer package using pip in the preamble (#2375). This would require locating the python package on the file system and installing it with pip. However the python package path could only be relative, because we want the path to work in different environments. I'm trying to use lf.source_directory() as the base for the relative import.

However I have encountered the need for something similar to lf.source_directory() in various other cases, and I think the functionality could be quite useful. Because the generated python files are located in the src-gen directory, it would inevitably break relative paths.

  • Loading config files. For example, using Hydra.
  • Accessing data files for ML applications.
  • Importing relative python files

In these applications I used workarounds such as changing the operating directory with os.chdir to a absolute location on my file system before imports.

@cmnrd
Copy link
Collaborator

cmnrd commented Jul 25, 2024

For Python there is already established, well-thought-out, and hardened tooling for all of this. The key for solving this issue would be a proper integration with the Python ecosystem (in particular packaging) and using importlib.resources instead of a homebrew mechanism.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working python Related to the Python target
Projects
None yet
Development

No branches or pull requests

3 participants