HDDS-5500. Fix broken kubernetes test #2464
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ts reverts k3s until a better solution is found.
GeorgeJahad
changed the title
adoroszlai
approved these changes
Jul 27, 2021
There was a problem hiding this comment.
I think the problem is a change in the k3s tool,
Thanks @GeorgeJahad for the fix.
FYI I also noticed that K3S was changed between the succesful and failed runs, so I have tried reverting to the previous version:
adoroszlai@910e9a9
https://github.com/adoroszlai/hadoop-ozone/runs/3155669162#step:6:55
but SCM startup still failed:
https://github.com/adoroszlai/hadoop-ozone/runs/3155669162#step:6:2079
It turns out that it failed due to another change (Ozone Runner image upgrade).
Now with the image upgrade reverted I have verified that K3S downgrade fixes the problem and reverting the downgrade still reproduces it.
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Jul 30, 2021
* master: (48 commits) HDDS-5514. Skip check for UNHEALTHY containers for datanode finalize. (apache#2469) HDDS-5279. OFS mkdir -p does not work when Volume is not pre-created (apache#2412) HDDS-5328. Remove delete container command from admin CLI (apache#2456) HDDS-5382. Increase default container report interval to 60 mins (apache#2363) HDDS-5378 Add APIs to retrieve Namespace Summary from Recon (apache#2417) HDDS-5466. Refactor BlockOutputStream. (apache#2442) HDDS-5465. Delete redundant code when set、add and remove bucket acl (apache#2439) HDDS-5184. Use separate DB profile for Datanodes. (apache#2214) HDDS-5494. Reduce retry in Kubernetes test (apache#2461) HDDS-5414. Data buffers incorrectly filtered for Ozone Insight (apache#2387) HDDS-5450. Avoid refresh pipeline for S3 headObject (apache#2431) HDDS-5500. New k3s version breaks kubernetes test (apache#2464) HDDS-5489. Install OS-specific flekszible (apache#2462) Multi-raft style placement with permutations for offline data generator (apache#2434) HDDS-5484. Intermittent failure in TestReplicationManager#testMovePrerequisites (apache#2454) HDDS-5443 Create and then recreate a bucket with a randomized name (apache#2436) HDDS-5492. Disable failing kubernetes test (apache#2459) HDDS-4330. Bootstrap new OM node (apache#1494) HDDS-5418. Let Recon send reregisterCommand to Datanodes if DatanodeDetails changed (apache#2392) HDDS-5479. s3g bucket list failed when there is non-english key name. (apache#2450) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
kubernetes tests are failing because the latest k3s has changed the permissions of mount persistentVolumes to be root only.
this breaks all our persistent volume tests which use the "hadoop" user.
the tests were disabled in a previous PR:
#2459
This PR re-enables them by reverting k3s to the last working version, (and re-enabling the disabled test.)
The problem is described in this k3s issue:
Cannot write data to local PVC #3704
k3s-io/k3s#3704
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5500
How was this patch tested?
kubernetes.sh tests