Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[documentation] Adds documentation to help fix local Lucene code errors #928

Merged
merged 4 commits into from
Oct 11, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 47 additions & 1 deletion DEVELOPER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,12 @@
- [Run Single-node Cluster Locally](#run-single-node-cluster-locally)
- [Run Multi-node Cluster Locally](#run-multi-node-cluster-locally)
- [Debugging](#debugging)
- [Major Dependencies](#major-dependencies)
- [Backwards Compatibility Testing](#backwards-compatibility-testing)
- [Adding new tests](#adding-new-tests)
- [Supported configurations](#supported-configurations)
- [Submitting Changes](#submitting-changes)
- [Building On Lucene Version Updates](#building-on-lucene-version-updates)

# Developer Guide

Expand Down Expand Up @@ -88,12 +90,22 @@ Please follow these formatting guidelines:
OpenSearch neural-search uses a [Gradle](https://docs.gradle.org/6.6.1/userguide/userguide.html) wrapper for its build.
Run `gradlew` on Unix systems.

Build OpenSearch neural-search using `gradlew build`
Build OpenSearch neural-search using `gradlew build`. This command will
also run Integration Tests and Unit Tests.
brianf-aws marked this conversation as resolved.
Show resolved Hide resolved

```
./gradlew build
```

## Run Unit Tests
If you want to strictly test that your unit tests are passing
you can run the following.

```
./gradlew test
```


## Run OpenSearch neural-search

### Run Single-node Cluster Locally
Expand Down Expand Up @@ -227,6 +239,12 @@ Additionally, it is possible to attach one debugger to the cluster JVM and anoth
./gradlew :integTest -Dtest.debug=1 -Dcluster.debug=1
```

#### Major Dependencies
Currently, the major dependencies that Neural Search depends on are [ML-Commons](https://github.com/opensearch-project/ml-commons) and [K-NN](https://github.com/opensearch-project/k-NN).
Make sure to check on them when you observe a failure that affects Neural Search.
See [Building on Lucene Version updates](#building-on-lucene-version-updates) as an example where K-NN caused a build failure.
Also, please note that it may take time for developers to create a fix for your current dependency issue.

## Backwards Compatibility Testing

The purpose of Backwards Compatibility Testing and different types of BWC tests are explained [here](https://github.com/opensearch-project/opensearch-plugins/blob/main/TESTING.md#backwards-compatibility-testing). The BWC tests (i.e. Restart-Upgrade, Mixed-Cluster and Rolling-Upgrade scenarios) should be added with any new feature being added to Neural Search.
Expand Down Expand Up @@ -292,3 +310,31 @@ original PR with an appropriate label `backport <backport-branch-name>` is merge
run successfully on the PR. For example, if a PR on main needs to be backported to `2.x` branch, add a label
`backport 2.x` to the PR and make sure the backport workflow runs on the PR along with other checks. Once this PR is
merged to main, the workflow will create a backport PR to the `2.x` branch.

## Building On Lucene Version Updates
There may be a Lucene version update that can affect your workflow causing errors like
`java.lang.NoClassDefFoundError: org/apache/lucene/codecs/lucene99/Lucene99Codec` or
`Provider org.opensearch.knn.index.codec.KNN910Codec.KNN910Codec could not be instantiated`. In this case
we can observe there may be an issue with a dependency with [K-NN](https://github.com/opensearch-project/k-NN).
This results in having issues with not being able to do `./gradlew run` or `./gradlew build`.

You can check this [K-NN PR](https://github.com/opensearch-project/k-NN/pull/2195) as an example of this event happening or this [Neural Search PR](https://github.com/opensearch-project/neural-search/pull/913#issuecomment-2400189329) that shows a developer going
through the same build issue.

**Follow the steps to remedy the gradle run issue.**
brianf-aws marked this conversation as resolved.
Show resolved Hide resolved
1. From your cloned neural search repo root directory `rm -rf build .gradle`
2. Clear the following directories from your gradle folder located in your root directory
1. `cd ~/.gradle`
2. `rm -rf caches workers wrapper daemon`
3. `cd -` switch back the previous directory (i.e. the neural search repo root directory)
3. Finally run `./gradlew run`

**Follow the steps to remedy the gradle build issue**

**PREREQ:** Make sure you have OpenSearch repo cloned locally

1. From your cloned neural search repo root directory `rm -rf build .gradle`
2. Delete the .gradle folder and .m2 folder. `rm -rf ~/.gradle ~/.m2`
3. Head over to your OpenSearch cloned repo root directory
1. `./gradlew publisToMavenLocal`
4. Finally run `./gradlew build` from the neural search repo
Loading