Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Link to cluster info #9633

Merged
merged 1 commit into from
Jul 22, 2024
Merged

docs: Link to cluster info #9633

merged 1 commit into from
Jul 22, 2024

Conversation

tara-det-ai
Copy link
Contributor

@tara-det-ai tara-det-ai commented Jul 11, 2024

How to access user data

Problem: it isn't clear what the interface is to access the values from inside the model code.

Users can call their own arbitrary user_data from the Experiment Configuration file. This would be things like data sources and containers or even credentials (which are obscurable). To do this, the user uses data (not metadata.data) in the Experiment configuration yaml file. Which is documented here. However, it is not clear how Determined accesses the values of this user data from inside the model code.

Solution:
Help users understand that the way to access the data:
https://docs.determined.ai/latest/reference/experiment-config-reference.html#data ...

is here
https://docs.determined.ai/latest/reference/training/api-det-reference.html#determined.ClusterInfo.user_data

another way
another way is just the xpath, for example, under the "username" section we could have task_container_defaults.registry_auth.username

there are 3 ways to retrieve data config and one way to get all of it:

From DET_EXPERIMENT_CONFIG env var like:
expconf = os.environ.get("DET_EXPERIMENT_CONFIG")

  • You can then json.loads(expconf) and handle it like json

From context

  • data_config = train_context.get_data_config()

From cluster_info

  • data = det.get_cluster_info().user_data

@cla-bot cla-bot bot added the cla-signed label Jul 11, 2024
@determined-ci determined-ci added the documentation Improvements or additions to documentation label Jul 11, 2024
@determined-ci determined-ci requested a review from a team July 11, 2024 00:01
Copy link

netlify bot commented Jul 11, 2024

Deploy Preview for determined-ui canceled.

Name Link
🔨 Latest commit 41ab05f
🔍 Latest deploy log https://app.netlify.com/sites/determined-ui/deploys/669e7f1bdc6aa80008ff7347

@tara-det-ai tara-det-ai force-pushed the docs/Link-to-cluster-info branch 2 times, most recently from 6560685 to 2d54322 Compare July 16, 2024 21:27
@tara-det-ai tara-det-ai merged commit 1c7f4b5 into main Jul 22, 2024
65 of 86 checks passed
@tara-det-ai tara-det-ai deleted the docs/Link-to-cluster-info branch July 22, 2024 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants