Add an initialization action to change the node manager's log directory path to local SSDs #1020
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
datasherlock
commented
Oct 6, 2022
datasherlock
commented
- If there are no local SSDs, the script will not add any property to yarn-site and the default boot disk path will be used
- If there are multiple mounts, it will create a comma separated list of paths for the property based on the MAX_MNT_DISK_FOR_LOGS setting.
- If MAX_MNT_DISK_FOR_LOGS is greater than the actual disks, then the actual disk count will be used. If not, the configuration will be honoured.
- The fluentd configuration file is modified to monitor the new path for pushing to Cloud Logging
|
/gcbrun |
|
@jainyogesh - Can you please help me find a reviewer for this PR? |
|
@ranu010101 , @vinayakumarb, @medb - can one of you please help review it or suggest another reviewr |
- Enclosed variables in {}
- Enclosed variables in double quotes where applicable.
jayadeep-jayaraman
requested changes
Jan 3, 2023
| # This function validates the MAX_MNT_DISK_FOR_LOGS variable and ensures that the value is capped at the max disks available | ||
| function get_max_local_ssd() { | ||
| max_disk_no=0 | ||
| for line in $(df -h --output=target); do |
There was a problem hiding this comment.
To be doubly sure let us also add the check for device names. Nothing stops someone from creating /mnt directory in root mount
There was a problem hiding this comment.
df returns only the filesystems and the mounts points they are mounted on. Even if someone were to create a directory called 'mnt' in /, df wouldn't return the path unless there's a device/filesystem mounted at that mount point.
jayadeep-jayaraman
approved these changes
Jan 6, 2023
|
/gcbrun |
jayadeep-jayaraman
approved these changes
Jan 6, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.