Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release version 2.1.0 #425

Closed
16 of 22 tasks
gaiksaya opened this issue Jun 21, 2022 · 4 comments
Closed
16 of 22 tasks

Release version 2.1.0 #425

gaiksaya opened this issue Jun 21, 2022 · 4 comments
Assignees
@Shivamdhar
Labels
v2.1.0

Comments

@gaiksaya
Copy link
Member

gaiksaya commented Jun 21, 2022 •
edited by Shivamdhar
Loading

This is a component issue for 2.1.0.
Coming from opensearch-build#1818. Please follow the following checklist.
Please refer to the DATES in that post.

How to use this issue

This Component Release Issue

This issue captures the state of the OpenSearch release, on component/plugin level; its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help.
Any release related work can be linked to this issue or added as comments to create visiblity into the release status.

Release Steps

There are several steps to the release process; these steps are completed as the whole component release and components that are behind present risk to the release. The component owner resolves the tasks in this issue and communicate with the overall release owner to make sure each component are moving along as expected.

Steps have completion dates for coordinating efforts between the components of a release; components can start as soon as they are ready far in advance of a future release. The most current set of dates is on the overall release issue linked at the top of this issue.

The Overall Release Issue

Linked at the top of this issue, the overall release issue captures the state of the entire OpenSearch release including references to this issue, the release owner which is the assignee is responsible for communicating the release status broadly. Please contact them or @mention them on that issue for help.

What should I do if my plugin isn't making any changes?

If including changes in this release, increment the version on 2.0 branch to 2.1.0 for Min/Core, and 2.1.0.0 for components. Otherwise, keep the version number unchanged for both.

Preparation

  • Assign this issue to a release owner.
  • Finalize scope and feature set and update the Public Roadmap.
  • All the tasks in this issue have been reviewed by the release owner.
  • Create, update, triage and label all features and issues targeted for this release with v2.1.0.

CI/CD

  • All code changes for 2.1.0 are complete.
  • Ensure working and passing CI.
  • Check that this repo is included in the distribution manifest.

Pre-Release

  • Update to the 2.1.0 release branch in the distribution manifest.
  • Increment the version on the parent branch to the next development iteration.
  • Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
  • Confirm that all changes for 2.1.0 have been merged.
  • Add this repo to the manifest for the next developer iteration.

Release Testing

  • Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
  • Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
  • Sanity Testing: Sanity testing and fixing of critical issues found.
  • File issues for all intermittent test failures.

Release

  • Complete documentation.
  • Verify all issued labeled for this release are closed or labelled for the next release.

Post Release

  • Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
  • Add this repo to the manifest of the next eventual security patch version.
  • Suggest improvements to this template.
  • Conduct a retrospective, and publish its results.
@gaiksaya gaiksaya added the v1.2.0 release label Jun 21, 2022
@gaiksaya gaiksaya changed the title Release version 1.2 Release version 2.1.0 Jun 21, 2022
@gaiksaya gaiksaya added v2.1.0 and removed v1.2.0 release labels Jun 21, 2022
@gaiksaya gaiksaya mentioned this issue Jun 21, 2022
Closed
40 tasks
This was referenced Jun 24, 2022
Merged
Merged
@Shivamdhar
Copy link
Contributor

Executed benchmark script part of k-NN perf-tool (https://github.com/opensearch-project/k-NN/tree/main/benchmarks/perf-tool):

python3.8 knn-prf-tool.py test sample-configs/nmslib-sift-hnsw/test.yml results/output.json

Results

{
  "metadata": {
    "test_name": "nmslib_sift_hnsw",
    "test_id": "Test workflow for nmslib hnsw",
    "date": "07/01/2022 23:26:18",
    "python_version": "3.8.5 (default, Feb 18 2021, 01:24:20) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-12)]",
    "os_version": "Linux-5.10.118-111.515.amzn2.x86_64-x86_64-with-glibc2.2.5",
    "processor": "x86_64, 36 cores",
    "memory": "464601088 (used) / 73020035072 (available) / 74167287808 (total)"
  },
  "results": {
    "test_took": 217352.58001048514,
    "create_index_took_total": 15070.491211983608,
    "ingest_took_total": 112612.91173801874,
    "refresh_index_store_kb_total": 1456328.9907226562,
    "refresh_index_took_total": 9922.444153984543,
    "force_merge_took_total": 504.73290649824776,
    "query_took_total": 79242.0,
    "query_took_p50": 8.0,
    "query_took_p90": 9.0,
    "query_took_p99": 10.5,
    "query_memory_kb_total": 652757.0,
    "query_recall@K_total": 0.9995080000000001,
    "query_recall@1_total": 1.0
  }

@Shivamdhar
Copy link
Contributor

Executed benchmark script part of k-NN osb (https://github.com/opensearch-project/k-NN/tree/main/benchmarks/osb):

export PARAMS_FILE=/home/ec2-user/k-NN/benchmarks/osb/params/no-train-params.json
export PROCEDURE=no-train-test
opensearch-benchmark execute_test --target-hosts $URL:$PORT --workload-path ./workload.json --workload-params ${PARAMS_FILE} --test-procedure=${PROCEDURE} --pipeline benchmark-only

Results:

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                         Metric |                    Task |    Value |   Unit |
|---------------------------------------------------------------:|------------------------:|---------:|-------:|
|                     Cumulative indexing time of primary shards |                         |  1.04825 |    min |
|             Min cumulative indexing time across primary shards |                         | 0.274683 |    min |
|          Median cumulative indexing time across primary shards |                         | 0.341467 |    min |
|             Max cumulative indexing time across primary shards |                         |   0.4321 |    min |
|            Cumulative indexing throttle time of primary shards |                         |        0 |    min |
|    Min cumulative indexing throttle time across primary shards |                         |        0 |    min |
| Median cumulative indexing throttle time across primary shards |                         |        0 |    min |
|    Max cumulative indexing throttle time across primary shards |                         |        0 |    min |
|                        Cumulative merge time of primary shards |                         |        0 |    min |
|                       Cumulative merge count of primary shards |                         |        0 |        |
|                Min cumulative merge time across primary shards |                         |        0 |    min |
|             Median cumulative merge time across primary shards |                         |        0 |    min |
|                Max cumulative merge time across primary shards |                         |        0 |    min |
|               Cumulative merge throttle time of primary shards |                         |        0 |    min |
|       Min cumulative merge throttle time across primary shards |                         |        0 |    min |
|    Median cumulative merge throttle time across primary shards |                         |        0 |    min |
|       Max cumulative merge throttle time across primary shards |                         |        0 |    min |
|                      Cumulative refresh time of primary shards |                         |   2.3526 |    min |
|                     Cumulative refresh count of primary shards |                         |       25 |        |
|              Min cumulative refresh time across primary shards |                         |  0.63625 |    min |
|           Median cumulative refresh time across primary shards |                         | 0.786383 |    min |
|              Max cumulative refresh time across primary shards |                         | 0.929967 |    min |
|                        Cumulative flush time of primary shards |                         |        0 |    min |
|                       Cumulative flush count of primary shards |                         |        0 |        |
|                Min cumulative flush time across primary shards |                         |        0 |    min |
|             Median cumulative flush time across primary shards |                         |        0 |    min |
|                Max cumulative flush time across primary shards |                         |        0 |    min |
|                                        Total Young Gen GC time |                         |    0.088 |      s |
|                                       Total Young Gen GC count |                         |        6 |        |
|                                          Total Old Gen GC time |                         |        0 |      s |
|                                         Total Old Gen GC count |                         |        0 |        |
|                                                     Store size |                         |  2.86994 |     GB |
|                                                  Translog size |                         |  1.22342 |     GB |
|                                         Heap used for segments |                         |        0 |     MB |
|                                       Heap used for doc values |                         |        0 |     MB |
|                                            Heap used for terms |                         |        0 |     MB |
|                                            Heap used for norms |                         |        0 |     MB |
|                                           Heap used for points |                         |        0 |     MB |
|                                    Heap used for stored fields |                         |        0 |     MB |
|                                                  Segment count |                         |       45 |        |
|                                                 Min Throughput |      custom-vector-bulk |  28373.1 | docs/s |
|                                                Mean Throughput |      custom-vector-bulk |  49612.1 | docs/s |
|                                              Median Throughput |      custom-vector-bulk |  53331.4 | docs/s |
|                                                 Max Throughput |      custom-vector-bulk |  78027.5 | docs/s |
|                                        50th percentile latency |      custom-vector-bulk |  16.6189 |     ms |
|                                        90th percentile latency |      custom-vector-bulk |  24.7767 |     ms |
|                                        99th percentile latency |      custom-vector-bulk |  39.5298 |     ms |
|                                      99.9th percentile latency |      custom-vector-bulk |  4960.76 |     ms |
|                                       100th percentile latency |      custom-vector-bulk |  21882.7 |     ms |
|                                   50th percentile service time |      custom-vector-bulk |  16.6189 |     ms |
|                                   90th percentile service time |      custom-vector-bulk |  24.7767 |     ms |
|                                   99th percentile service time |      custom-vector-bulk |  39.5298 |     ms |
|                                 99.9th percentile service time |      custom-vector-bulk |  4960.76 |     ms |
|                                  100th percentile service time |      custom-vector-bulk |  21882.7 |     ms |
|                                                     error rate |      custom-vector-bulk |        0 |      % |
|                                                 Min Throughput |    refresh-target-index |     0.01 |  ops/s |
|                                                Mean Throughput |    refresh-target-index |     0.01 |  ops/s |
|                                              Median Throughput |    refresh-target-index |     0.01 |  ops/s |
|                                                 Max Throughput |    refresh-target-index |     0.01 |  ops/s |
|                                       100th percentile latency |    refresh-target-index |  82981.1 |     ms |
|                                  100th percentile service time |    refresh-target-index |  82981.1 |     ms |
|                                                     error rate |    refresh-target-index |        0 |      % |
|                                                 Min Throughput | knn-query-from-data-set |      451 |  ops/s |
|                                                Mean Throughput | knn-query-from-data-set |   678.37 |  ops/s |
|                                              Median Throughput | knn-query-from-data-set |   709.65 |  ops/s |
|                                                 Max Throughput | knn-query-from-data-set |   734.07 |  ops/s |
|                                        50th percentile latency | knn-query-from-data-set |  12.6933 |     ms |
|                                        90th percentile latency | knn-query-from-data-set |  15.2904 |     ms |
|                                        99th percentile latency | knn-query-from-data-set |    21.08 |     ms |
|                                      99.9th percentile latency | knn-query-from-data-set |  34.8925 |     ms |
|                                     99.99th percentile latency | knn-query-from-data-set |  183.734 |     ms |
|                                       100th percentile latency | knn-query-from-data-set |  183.885 |     ms |
|                                   50th percentile service time | knn-query-from-data-set |  12.6933 |     ms |
|                                   90th percentile service time | knn-query-from-data-set |  15.2904 |     ms |
|                                   99th percentile service time | knn-query-from-data-set |    21.08 |     ms |
|                                 99.9th percentile service time | knn-query-from-data-set |  34.8925 |     ms |
|                                99.99th percentile service time | knn-query-from-data-set |  183.734 |     ms |
|                                  100th percentile service time | knn-query-from-data-set |  183.885 |     ms |
|                                                     error rate | knn-query-from-data-set |        0 |      % |


---------------------------------
[INFO] SUCCESS (took 167 seconds)
---------------------------------

export PARAMS_FILE=/home/ec2-user/k-NN/benchmarks/osb/params/train-params.json
export PROCEDURE=train-test
opensearch-benchmark execute_test --target-hosts $URL:$PORT --workload-path ./workload.json --workload-params ${PARAMS_FILE} --test-procedure=${PROCEDURE} --pipeline benchmark-only

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                         Metric |                    Task |      Value |             Unit |
|---------------------------------------------------------------:|------------------------:|-----------:|-----------------:|
|                     Cumulative indexing time of primary shards |                         |    1.62143 |              min |
|             Min cumulative indexing time across primary shards |                         |     0.2503 |              min |
|          Median cumulative indexing time across primary shards |                         |   0.267283 |              min |
|             Max cumulative indexing time across primary shards |                         |   0.836567 |              min |
|            Cumulative indexing throttle time of primary shards |                         |          0 |              min |
|    Min cumulative indexing throttle time across primary shards |                         |          0 |              min |
| Median cumulative indexing throttle time across primary shards |                         |          0 |              min |
|    Max cumulative indexing throttle time across primary shards |                         |          0 |              min |
|                        Cumulative merge time of primary shards |                         |   0.866333 |              min |
|                       Cumulative merge count of primary shards |                         |         24 |                  |
|                Min cumulative merge time across primary shards |                         | 0.00826667 |              min |
|             Median cumulative merge time across primary shards |                         |   0.152008 |              min |
|                Max cumulative merge time across primary shards |                         |    0.55405 |              min |
|               Cumulative merge throttle time of primary shards |                         |   0.759867 |              min |
|       Min cumulative merge throttle time across primary shards |                         |          0 |              min |
|    Median cumulative merge throttle time across primary shards |                         |    0.11985 |              min |
|       Max cumulative merge throttle time across primary shards |                         |   0.520167 |              min |
|                      Cumulative refresh time of primary shards |                         |    0.14305 |              min |
|                     Cumulative refresh count of primary shards |                         |         67 |                  |
|              Min cumulative refresh time across primary shards |                         |  0.0250167 |              min |
|           Median cumulative refresh time across primary shards |                         |  0.0391333 |              min |
|              Max cumulative refresh time across primary shards |                         |  0.0397667 |              min |
|                        Cumulative flush time of primary shards |                         |    0.03415 |              min |
|                       Cumulative flush count of primary shards |                         |          1 |                  |
|                Min cumulative flush time across primary shards |                         |          0 |              min |
|             Median cumulative flush time across primary shards |                         |          0 |              min |
|                Max cumulative flush time across primary shards |                         |    0.03415 |              min |
|                                        Total Young Gen GC time |                         |       0.06 |                s |
|                                       Total Young Gen GC count |                         |          7 |                  |
|                                          Total Old Gen GC time |                         |          0 |                s |
|                                         Total Old Gen GC count |                         |          0 |                  |
|                                                     Store size |                         |    3.09407 |               GB |
|                                                  Translog size |                         |    1.33393 |               GB |
|                                         Heap used for segments |                         |          0 |               MB |
|                                       Heap used for doc values |                         |          0 |               MB |
|                                            Heap used for terms |                         |          0 |               MB |
|                                            Heap used for norms |                         |          0 |               MB |
|                                           Heap used for points |                         |          0 |               MB |
|                                    Heap used for stored fields |                         |          0 |               MB |
|                                                  Segment count |                         |         68 |                  |
|                                                 Min Throughput |            delete-model |     137.28 |            ops/s |
|                                                Mean Throughput |            delete-model |     137.28 |            ops/s |
|                                              Median Throughput |            delete-model |     137.28 |            ops/s |
|                                                 Max Throughput |            delete-model |     137.28 |            ops/s |
|                                       100th percentile latency |            delete-model |    7.00644 |               ms |
|                                  100th percentile service time |            delete-model |    7.00644 |               ms |
|                                                     error rate |            delete-model |          0 |                % |
|                                                 Min Throughput |       train-vector-bulk |      68475 |           docs/s |
|                                                Mean Throughput |       train-vector-bulk |    75823.8 |           docs/s |
|                                              Median Throughput |       train-vector-bulk |    77733.9 |           docs/s |
|                                                 Max Throughput |       train-vector-bulk |    78654.8 |           docs/s |
|                                        50th percentile latency |       train-vector-bulk |    17.1311 |               ms |
|                                        90th percentile latency |       train-vector-bulk |    25.3084 |               ms |
|                                        99th percentile latency |       train-vector-bulk |     96.666 |               ms |
|                                      99.9th percentile latency |       train-vector-bulk |    126.474 |               ms |
|                                       100th percentile latency |       train-vector-bulk |    199.319 |               ms |
|                                   50th percentile service time |       train-vector-bulk |    17.1311 |               ms |
|                                   90th percentile service time |       train-vector-bulk |    25.3084 |               ms |
|                                   99th percentile service time |       train-vector-bulk |     96.666 |               ms |
|                                 99.9th percentile service time |       train-vector-bulk |    126.474 |               ms |
|                                  100th percentile service time |       train-vector-bulk |    199.319 |               ms |
|                                                     error rate |       train-vector-bulk |          0 |                % |
|                                                 Min Throughput |     refresh-train-index |     142.69 |            ops/s |
|                                                Mean Throughput |     refresh-train-index |     142.69 |            ops/s |
|                                              Median Throughput |     refresh-train-index |     142.69 |            ops/s |
|                                                 Max Throughput |     refresh-train-index |     142.69 |            ops/s |
|                                       100th percentile latency |     refresh-train-index |    6.61596 |               ms |
|                                  100th percentile service time |     refresh-train-index |    6.61596 |               ms |
|                                                     error rate |     refresh-train-index |          0 |                % |
|                                                 Min Throughput |       ivfpq-train-model |       0.01 | models_trained/s |
|                                                Mean Throughput |       ivfpq-train-model |       0.01 | models_trained/s |
|                                              Median Throughput |       ivfpq-train-model |       0.01 | models_trained/s |
|                                                 Max Throughput |       ivfpq-train-model |       0.01 | models_trained/s |
|                                       100th percentile latency |       ivfpq-train-model |     130760 |               ms |
|                                  100th percentile service time |       ivfpq-train-model |     130760 |               ms |
|                                                     error rate |       ivfpq-train-model |          0 |                % |
|                                                 Min Throughput |      custom-vector-bulk |    72796.8 |           docs/s |
|                                                Mean Throughput |      custom-vector-bulk |      78309 |           docs/s |
|                                              Median Throughput |      custom-vector-bulk |    79128.1 |           docs/s |
|                                                 Max Throughput |      custom-vector-bulk |    80157.3 |           docs/s |
|                                        50th percentile latency |      custom-vector-bulk |    14.2077 |               ms |
|                                        90th percentile latency |      custom-vector-bulk |    18.8638 |               ms |
|                                        99th percentile latency |      custom-vector-bulk |    97.5693 |               ms |
|                                      99.9th percentile latency |      custom-vector-bulk |    333.256 |               ms |
|                                       100th percentile latency |      custom-vector-bulk |     358.68 |               ms |
|                                   50th percentile service time |      custom-vector-bulk |    14.2077 |               ms |
|                                   90th percentile service time |      custom-vector-bulk |    18.8638 |               ms |
|                                   99th percentile service time |      custom-vector-bulk |    97.5693 |               ms |
|                                 99.9th percentile service time |      custom-vector-bulk |    333.256 |               ms |
|                                  100th percentile service time |      custom-vector-bulk |     358.68 |               ms |
|                                                     error rate |      custom-vector-bulk |          0 |                % |
|                                                 Min Throughput |    refresh-target-index |      47.97 |            ops/s |
|                                                Mean Throughput |    refresh-target-index |      47.97 |            ops/s |
|                                              Median Throughput |    refresh-target-index |      47.97 |            ops/s |
|                                                 Max Throughput |    refresh-target-index |      47.97 |            ops/s |
|                                       100th percentile latency |    refresh-target-index |    20.4669 |               ms |
|                                  100th percentile service time |    refresh-target-index |    20.4669 |               ms |
|                                                     error rate |    refresh-target-index |          0 |                % |
|                                                 Min Throughput | knn-query-from-data-set |     553.91 |            ops/s |
|                                                Mean Throughput | knn-query-from-data-set |     659.58 |            ops/s |
|                                              Median Throughput | knn-query-from-data-set |     669.97 |            ops/s |
|                                                 Max Throughput | knn-query-from-data-set |      695.5 |            ops/s |
|                                        50th percentile latency | knn-query-from-data-set |    13.7852 |               ms |
|                                        90th percentile latency | knn-query-from-data-set |    17.0937 |               ms |
|                                        99th percentile latency | knn-query-from-data-set |    23.8833 |               ms |
|                                      99.9th percentile latency | knn-query-from-data-set |    52.4814 |               ms |
|                                     99.99th percentile latency | knn-query-from-data-set |    84.9405 |               ms |
|                                       100th percentile latency | knn-query-from-data-set |    1029.88 |               ms |
|                                   50th percentile service time | knn-query-from-data-set |    13.7852 |               ms |
|                                   90th percentile service time | knn-query-from-data-set |    17.0937 |               ms |
|                                   99th percentile service time | knn-query-from-data-set |    23.8833 |               ms |
|                                 99.9th percentile service time | knn-query-from-data-set |    52.4814 |               ms |
|                                99.99th percentile service time | knn-query-from-data-set |    84.9405 |               ms |
|                                  100th percentile service time | knn-query-from-data-set |    1029.88 |               ms |
|                                                     error rate | knn-query-from-data-set |          0 |                % |


---------------------------------
[INFO] SUCCESS (took 245 seconds)
---------------------------------

@gaiksaya
Copy link
Member Author

gaiksaya commented Jul 7, 2022

Hi @Shivamdhar ,

We found below issues tagged with v2.1.0 that were not closed or resolved in this release. Can you take a look and close the ones that are resolved or retag them with next version you see fit?
Thanks!

#425  Release version 2.1.0                v2.1.0               about 1 day ago
#307  Deprecate the "Master" nomenclature  enhancement, v2.1.0  about 2 months ago
#234  Reorganize packages                  Refactoring, v2.1.0  about 3 months ago

@Shivamdhar
Copy link
Contributor

Thanks @gaiksaya ! updated the issues to reflect correct tags.

Release notes are published here https://github.com/opensearch-project/k-NN/releases/tag/2.1.0.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Shivamdhar Shivamdhar

Labels
v2.1.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Shivamdhar @jmazanec15 @gaiksaya