initial commit for hot shard tool #1299

AnnigeriShambu · 2022-07-21T16:06:25Z

Description

Profile analyzer is a script to findout the top team reads. This PR adds a plugin to run the script as a Job to analyse the cluster.

Type of change

Please select one of the options below.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation
Other

Discussion

Are there any design details that you would like to discuss further?

Testing

Please describe the tests that you ran to verify your changes. Unit tests?
Manual testing?
Yes, Did Manual testing for both 6.3 and 7.1 clusters. Below is the output from the local cluster:

 k logs test-cluster-hot-shard-tool-wbxl5
/app/./transaction_profiling_analyzer.py:544: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  logger.warn("Can't find sortedcontainers so disabling ReadCounter")
Processed 0 transactions, 0 invalid

NOTE: shard locations are current and may not reflect where an operation was performed in the past

Test case output:

go test  ./... -coverprofile cover.out
?   	github.com/FoundationDB/fdb-kubernetes-operator	[no test files]
ok  	github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta1	1.039s	coverage: 41.2% of statements
ok  	github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2	1.333s	coverage: 46.8% of statements
?   	github.com/FoundationDB/fdb-kubernetes-operator/cmd/po-docgen	[no test files]
ok  	github.com/FoundationDB/fdb-kubernetes-operator/controllers	82.059s	coverage: 81.7% of statements
ok  	github.com/FoundationDB/fdb-kubernetes-operator/fdbclient	1.251s	coverage: 8.4% of statements
ok  	github.com/FoundationDB/fdb-kubernetes-operator/internal	1.396s	coverage: 71.4% of statements
ok  	github.com/FoundationDB/fdb-kubernetes-operator/internal/removals	1.108s	coverage: 61.2% of statements
ok  	github.com/FoundationDB/fdb-kubernetes-operator/internal/replacements	1.516s	coverage: 63.8% of statements
?   	github.com/FoundationDB/fdb-kubernetes-operator/kubectl-fdb	[no test files]
Running Suite: FDB plugin - /Users/shambugouda_annigeri/go/src/github.com/FoundationDB/fdb-kubernetes-operator/kubectl-fdb/cmd
==============================================================================================================================
Random Seed: 1662527212

Will run 2 of 83 specs
S2022/09/07 06:06:52 creating job test-hot-shard-tool
•2022/09/07 06:06:52 creating job test-hot-shard-tool
•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

Ran 2 of 83 Specs in 0.018 seconds
SUCCESS! -- 2 Passed | 0 Failed | 0 Pending | 81 Skipped
PASS | FOCUSED
FAIL	github.com/FoundationDB/fdb-kubernetes-operator/kubectl-fdb/cmd	1.493s
ok  	github.com/FoundationDB/fdb-kubernetes-operator/mock-kubernetes-client/client	1.590s	coverage: 75.9% of statements
?   	github.com/FoundationDB/fdb-kubernetes-operator/pkg/fdbadminclient	[no test files]
?   	github.com/FoundationDB/fdb-kubernetes-operator/pkg/podclient	[no test files]
ok  	github.com/FoundationDB/fdb-kubernetes-operator/pkg/podmanager	2.042s	coverage: 5.6% of statements
?   	github.com/FoundationDB/fdb-kubernetes-operator/setup	[no test files]
FAIL
make: *** [test] Error 1

Do we need to perform additional testing once this is merged, or perform in a larger testing environment?

Documentation

Did you update relevant documentation within this repository?

If this change is adding new functionality, do we need to describe it in our user manual?

If this change is adding or removing subreconcilers, have we updated the core technical design doc to reflect that?

If this change is adding new safety checks or new potential failure modes, have we documented and how to debug potential issues?

Follow-up

Are there any follow-up issues that we should pursue in the future?

Does this introduce new defaults that we should re-evaluate in the future?

foundationdb-ci · 2022-07-21T17:01:22Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 39302a4
Duration 0:54:45
Result: ✅ SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

johscheuer

Commenting here because I was asked for a review.

config/deployment/profile_analyzer.yaml

kubectl-fdb/cmd/profile_analyzer.go

sample-apps/fdb-profile-analyzer/Dockerfile

foundationdb-ci · 2022-07-28T10:50:51Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 872d311
Duration 1:17:50
Result: ❌ FAILED
Error: Error while executing command: make -C tests -kj prOperator. Reason: exit status 2
Build Logs (available for 30 days)

foundationdb-ci · 2022-07-28T15:42:20Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: af7e7cc
Duration 1:13:15
Result: ❌ FAILED
Error: Error while executing command: make -C tests -kj prOperator. Reason: exit status 2
Build Logs (available for 30 days)

foundationdb-ci · 2022-07-29T16:47:52Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: b827a0d
Duration 1:06:32
Result: ❌ FAILED
Error: Error while executing command: make -C tests -kj prOperator. Reason: exit status 2
Build Logs (available for 30 days)

foundationdb-ci · 2022-08-03T11:02:38Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 8c2ebf7
Duration 1:03:32
Result: ✅ SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

johscheuer

Is there a reason we dropped the initial "hardcoded" Job approach completely and force users to use a template file? To my understanding the template approach was meant to be an addition and not a replacement.

kubectl-fdb/cmd/profile_analyzer.go

foundationdb-ci · 2022-08-09T16:22:34Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 59a3a43
Duration 0:57:57
Result: ✅ SUCCEEDED
Error: N/A
Build Logs (available for 30 days)

foundationdb-ci · 2022-08-09T16:50:02Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 41b58ff
Duration 1:21:32
Result: ❌ FAILED
Error: Error while executing command: make -C tests -kj prOperator. Reason: exit status 2
Build Logs (available for 30 days)

foundationdb-ci · 2022-08-10T13:52:23Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 75cefe9
Duration 2:02:24
Result: ❌ FAILED
Error: Error while executing command: make -C tests -kj prOperator. Reason: exit status 2
Build Logs (available for 30 days)

foundationdb-ci · 2022-08-10T14:11:56Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 830c054
Duration 2:03:52
Result: ❌ FAILED
Error: Error while executing command: make -C tests -kj prOperator. Reason: exit status 2
Build Logs (available for 30 days)

foundationdb-ci · 2022-08-10T17:44:31Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 5a20f07
Duration 2:14:25
Result: ❌ FAILED
Error: Error while executing command: make -C tests -kj prOperator. Reason: exit status 2
Build Logs (available for 30 days)

kubectl-fdb/cmd/profile_analyzer.go

sample-apps/fdb-profile-analyzer/sample_template.yaml

foundationdb-ci · 2022-08-26T12:14:23Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 16da7e9
Duration 1:58:12
Result: ✅ SUCCEEDED
Error: N/A
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

foundationdb-ci · 2022-08-30T16:33:12Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 03fe3ce
Duration 1:47:34
Result: ❌ FAILED
Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

kubectl-fdb/cmd/profile_analyzer_test.go

foundationdb-ci · 2022-09-02T15:47:59Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: d655fe2
Duration 4:07:06
Result: ❌ FAILED
Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

foundationdb-ci · 2022-09-02T19:51:48Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: a25f2ca
Duration 4:07:05
Result: ✅ SUCCEEDED
Error: N/A
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

foundationdb-ci · 2022-09-02T21:05:28Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: ae8e9c8
Duration 4:06:52
Result: ✅ SUCCEEDED
Error: N/A
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

johscheuer

I think we are pretty close to get this merged as an initial working version. We can improve the setup in other PRs.

Have you thought about the Docker image release process and how should it be versioned?

kubectl-fdb/cmd/profile_analyzer.go

sample-apps/fdb-profile-analyzer/sample_template.yaml

foundationdb-ci · 2022-09-07T09:29:26Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 900f5ad
Duration 4:06:12
Result: ❌ FAILED
Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

foundationdb-ci · 2022-09-07T09:34:17Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: a49f837
Duration 4:05:52
Result: ❌ FAILED
Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

foundationdb-ci · 2022-09-07T09:34:45Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 76ed4b6
Duration 4:08:16
Result: ✅ SUCCEEDED
Error: N/A
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

foundationdb-ci · 2022-09-07T10:05:20Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 0adb852
Duration 4:05:51
Result: ❌ FAILED
Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

foundationdb-ci · 2022-09-07T10:06:31Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 58d061f
Duration 4:08:30
Result: ❌ FAILED
Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

johscheuer

Could you answer the unresolved questions?

foundationdb-ci · 2022-09-08T18:21:49Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 4555882
Duration 4:06:07
Result: ❌ FAILED
Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

johscheuer

Please add a note in the command help that this feature is experimental and needs some additional work. Otherwise I'm fine with merging it.

kubectl-fdb/cmd/profile_analyzer_test.go

kubectl-fdb/cmd/profile_analyzer.go

sample-apps/fdb-profile-analyzer/Dockerfile

sample-apps/fdb-profile-analyzer/sample_template.yaml

foundationdb-ci · 2022-09-12T18:54:13Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: 3976714
Duration 4:06:05
Result: ❌ FAILED
Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

kubectl-fdb/cmd/profile_analyzer.go

Update comment in plugin

johscheuer

I'm okay with merging the current state but we have to follow up with the open tasks.

foundationdb-ci · 2022-09-14T13:10:32Z

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

Commit ID: b0972a1
Duration 4:06:07
Result: ❌ FAILED
Error: Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1
Build Logs (available for 30 days)
Build Artifact (available for 30 days)

initial commit

39302a4

johscheuer requested changes Jul 21, 2022

View reviewed changes

Use Job instead of deployment

872d311

refactore more code

af7e7cc

add configmap

b827a0d

Use tempaltes for profile analyzer

8c2ebf7

johscheuer reviewed Aug 3, 2022

View reviewed changes

kubectl-fdb/cmd/profile_analyzer.go Outdated Show resolved Hide resolved

kubectl-fdb/cmd/profile_analyzer.go Outdated Show resolved Hide resolved

kubectl-fdb/cmd/profile_analyzer.go Outdated Show resolved Hide resolved

kubectl-fdb/cmd/profile_analyzer.go Outdated Show resolved Hide resolved

Shambugouda annigeri added 2 commits August 9, 2022 16:24

Use batchv1 API to create Job

59a3a43

Update required flags

41b58ff

Merge branch 'main' into hot_shard_tool

75cefe9

AnnigeriShambu marked this pull request as ready for review August 10, 2022 11:49

Add dockerfile

830c054

AnnigeriShambu changed the title ~~[WIP] initial commit for hot shard tool~~ initial commit for hot shard tool Aug 10, 2022

Fix formats

5a20f07

johscheuer requested changes Aug 15, 2022

View reviewed changes

add test case

16da7e9

Adding test case for profile analyzer

03fe3ce

johscheuer reviewed Aug 31, 2022

View reviewed changes

kubectl-fdb/cmd/profile_analyzer_test.go Outdated Show resolved Hide resolved

kubectl-fdb/cmd/profile_analyzer_test.go Show resolved Hide resolved

AnnigeriShambu requested a review from johscheuer September 2, 2022 12:39

Update test case

a25f2ca

change string split

ae8e9c8

johscheuer requested changes Sep 6, 2022

View reviewed changes

Shambugouda annigeri added 4 commits September 7, 2022 06:11

fix PR comments and add test case for 7.1

900f5ad

Fix test case

76ed4b6

remove focused test case

a49f837

Add test case for 6.3 and 7.1

58d061f

AnnigeriShambu requested a review from johscheuer September 7, 2022 05:58

lint the code

0adb852

johscheuer reviewed Sep 7, 2022

View reviewed changes

pin version in pip

4555882

AnnigeriShambu requested a review from johscheuer September 8, 2022 14:16

johscheuer requested changes Sep 9, 2022

View reviewed changes

add experimental comment

3976714

johscheuer reviewed Sep 14, 2022

View reviewed changes

kubectl-fdb/cmd/profile_analyzer.go Outdated Show resolved Hide resolved

Update kubectl-fdb/cmd/profile_analyzer.go

b0972a1

Update comment in plugin

johscheuer approved these changes Sep 14, 2022

View reviewed changes

johscheuer merged commit 2bd11bf into FoundationDB:main Sep 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initial commit for hot shard tool #1299

initial commit for hot shard tool #1299

AnnigeriShambu commented Jul 21, 2022 •

edited

Loading

foundationdb-ci commented Jul 21, 2022

johscheuer left a comment

foundationdb-ci commented Jul 28, 2022

foundationdb-ci commented Jul 28, 2022

foundationdb-ci commented Jul 29, 2022

foundationdb-ci commented Aug 3, 2022

johscheuer left a comment

foundationdb-ci commented Aug 9, 2022

foundationdb-ci commented Aug 9, 2022

foundationdb-ci commented Aug 10, 2022

foundationdb-ci commented Aug 10, 2022

foundationdb-ci commented Aug 10, 2022

foundationdb-ci commented Aug 26, 2022

foundationdb-ci commented Aug 30, 2022

foundationdb-ci commented Sep 2, 2022

foundationdb-ci commented Sep 2, 2022

foundationdb-ci commented Sep 2, 2022

johscheuer left a comment

foundationdb-ci commented Sep 7, 2022

foundationdb-ci commented Sep 7, 2022

foundationdb-ci commented Sep 7, 2022

foundationdb-ci commented Sep 7, 2022

foundationdb-ci commented Sep 7, 2022

johscheuer left a comment

foundationdb-ci commented Sep 8, 2022

johscheuer left a comment

foundationdb-ci commented Sep 12, 2022

johscheuer left a comment

foundationdb-ci commented Sep 14, 2022

initial commit for hot shard tool #1299

initial commit for hot shard tool #1299

Conversation

AnnigeriShambu commented Jul 21, 2022 • edited Loading

Description

Type of change

Discussion

Testing

Documentation

Follow-up

foundationdb-ci commented Jul 21, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

johscheuer left a comment

Choose a reason for hiding this comment

foundationdb-ci commented Jul 28, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Jul 28, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Jul 29, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Aug 3, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

johscheuer left a comment

Choose a reason for hiding this comment

foundationdb-ci commented Aug 9, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Aug 9, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Aug 10, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Aug 10, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Aug 10, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Aug 26, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Aug 30, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Sep 2, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Sep 2, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Sep 2, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

johscheuer left a comment

Choose a reason for hiding this comment

foundationdb-ci commented Sep 7, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Sep 7, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Sep 7, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Sep 7, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

foundationdb-ci commented Sep 7, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

johscheuer left a comment

Choose a reason for hiding this comment

foundationdb-ci commented Sep 8, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

johscheuer left a comment

Choose a reason for hiding this comment

foundationdb-ci commented Sep 12, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

johscheuer left a comment

Choose a reason for hiding this comment

foundationdb-ci commented Sep 14, 2022

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

AnnigeriShambu commented Jul 21, 2022 •

edited

Loading