Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding documentation on codec versioning #337

Merged
merged 1 commit into from
Mar 30, 2022
Merged

Adding documentation on codec versioning #337

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions DEVELOPER_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
- [Debugging](#debugging)
- [Backwards Compatibility Testing](#backwards-compatibility-testing)
- [Adding new tests](#adding-new-tests)
- [Codec Versioning](#codec-versioning)
- [Submitting Changes](#submitting-changes)

# Developer Guide
Expand Down Expand Up @@ -265,6 +266,26 @@ Here, we are testing BWC Tests with BWC version of plugin as 1.0.0.0. Make sure

Before adding any new tests to Backward Compatibility Tests, we should be aware that the tests in BWC are not independent. While creating an index, a test cannot use the same index name if it is already used in other tests. Also, adding extra operations to the existing test may impact other existing tests like graphCount.

## Codec Versioning
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add this(with a hyperlink) to the index on the top of this file ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ack


Starting from 2.0 release the new versioning for codec has been introduced. Two positions will be used to define the version,
in format 'X.Y', where 'X' corresponds to underlying version of Lucene and 'Y' is the version of the format.

Codec version is used in following classes and methods:
- org.opensearch.knn.index.codec.KNNXYCodec.KNNXYCodec
- org.opensearch.knn.index.codec.KNNFormatFactory.createKNNXYFormat

These classes and methods are tied directly to Lucene version represented by 'X' part.
Other classes use the delegate pattern so no direct tie to Lucene version are related to format and represented by 'Y'

- BinaryDocValues
- CompoundFormat
- DocValuesConsumer
- DocValuesReader

Version '910' is going to be the first such new version. It corresponds to Lucene 9.1 that is used by the underlying OpenSearch 2.0 and initial
version of the format classes. If in future we need to adjust something in format logic, we only increment the 'Y' part and version became '911'.

## Submitting Changes

See [CONTRIBUTING](CONTRIBUTING.md).
Expand Down