Automated mechanism to update library versions across all repos under OpenSearch project

[bbarani]

Is your feature request related to a problem? Please describe

We are seeing CVE's and vulnerabilities across multiple repos under OpenSearch project that require upgrading / downgrading of the impacted libraries. This is currently a manual process that involves scanning all repositories and updating the reference of impacted libraries.

Describe the solution you'd like

Automated workflow to scan and raise the PR with appropriate versions of the upgraded library versions across all the repos.

Describe alternatives you've considered

No response

Additional context

No response

[prudhvigodithi]

My thoughts

• This can be implemented with custom gradle plugin/task that can take an input of impacted libraries and version to update them.
• Now this custom gradle plugin/task can be part of every plugin build.gradle file.
• Using GH workflow (similar to existing version-increment) we can call the task across for all repos.

./gradlew updateLibraries -P<LIBRARY1>=<VERSION> -P<LIBRARY2>=<VERSION>

• This task internal can use ant.replaceregexp to update the file which contains the impacted library.

Adding @dblock

[ps48]

@prudhvigodithi This looks like a clean initial approach for all OpenSearch plugins. But, we'll need a similar approach for OpenSearch dashboard plugins. Instead of gradle there, we can have node/yarn scripts or use npm-check-updates

My thoughts

* This can be implemented with custom gradle plugin/task that can take an input of impacted libraries and version to update them.

* Now this custom gradle plugin/task can be part of every plugin `build.gradle` file.

* Using GH workflow (similar to existing [version-increment](https://github.com/opensearch-project/opensearch-build/blob/main/.github/workflows/increment-plugin-versions.yml#L78)) we can call the task across for all repos.

[ZilongX]

It would be awesome if we can have this automated feature built into our current infra (even partially would save us a bunch of efforts).

Some thoughts / wonderings :

• Where do we retrieve the to-be-fixed CVE list ? From Github or directly from https://nvd.nist.gov/vuln ?
• We'll need to distinguish packages for some major dependencies, for example Node.js, Angular.js which would still need massive manual efforts to bump versions or upgrade.
• Base on Semantic Versioning enforced in place, there would be no breaking changes permitted in a patch version bump of OpenSearch, it would be great if we can add this check (would change of dependency versions introduce in breaking changes) if possible

[ZilongX]

And agreed with @ps48, we might need different approaches for OpenSeach v.s. OpenSearch Dashboards

[ZilongX]

Not sure if https://docs.github.com/en/code-security/dependabot/dependabot-security-updates/configuring-dependabot-security-updates this dependabot offered by github would help in some way as well.

[gaiksaya]

I believe depandabot is already doing what is asked for here. Is there other than that, that we are trying to achieve?

[bbarani]

@gaiksaya I dont think depandabot works as expected all the time. I haven't seen it creating automated PR's for affected libraries in multiple repos and it still had multiple open issues. Reference: opensearch-project/OpenSearch#3782

[davidlago]

Thanks for bringing this up. I agree with @gaiksaya that this is in theory what dependabot should be doing for us, so perhaps worth looking into why it's failing in some cases rather than building something else on the side?

[prudhvigodithi]

I see this way, there are chances sometimes we have to bump a library version that might not be identified as CVE (CVE sources are vast) by dependent bot, its worth to have a mechanism that can be done across for all plugins consistently.

Another scenario I see is opensearch-project/OpenSearch#4556, jackson and snakeyml dependencies are updated in core, in order for the plugins to be consistent and align with core this mechanism would be useful.

[owaiskazi19]

Dependabot doesn't scan dependencies present in version.properties file present in OpenSearch core repo - opensearch-project/OpenSearch#3782. It only inspects dependencies present in build.gradle.

[VachaShah]

@bbarani Dependabot works to upgrade the dependencies for multiple package ecosystems by scanning the dependency manifests. For example, in case of gradle, it scans the build.gradle files and upgrades dependencies.

In order to allow Dependabot to upgrade dependencies, a dependabot.yml config needs to be included in .github folder with all the paths that have a dependency manifest. See https://github.com/opensearch-project/OpenSearch/blob/main/.github/dependabot.yml for example where it scans build.gradle files in multiple sub-projects.

On top of Dependabot, we have also added a workflow which performs some required actions on a PR commit like ./gradlew updateSHAs and ./gradlew spotlessApply in case of the OpenSearch repo. This workflow helps to perform required gradle tasks so there is minimal manual effort when Dependabot opens PRs on a repo. See https://github.com/opensearch-project/OpenSearch/blob/main/.github/workflows/dependabot_pr.yml for more details. Each repo can implement a similar kind of workflow to ease automation.

Adding Dependabot to all plugins will help keep the dependencies up-to-date and will also keep versions consistent across them. Are there any specific plugin repos that you have seen issues with for Dependabot integration?

[bbarani]

@VachaShah Apologize for delay in my response. I have noticed multiple times when Dependabot didn't work as expected mainly due to the nature of the change log. For Example: CVE-2022-42003 has been fixed in Jackson 2.13.4.2 but its not yet updated across all the external sources that dependabot uses to track so we would need to wait until it reports it on our repo rather than proactively upgrading to patched version across all repos now since we know the fixed version. Also, as mentioned by @owaiskazi19 Dependabot doesn't scan dependencies present in version.properties file present in OpenSearch core repo - opensearch-project/OpenSearch#3782 so we can use this utility as a general purpose utility not targeted around remediating CVE but also to upgrade dependency to a specific version across multiple repos.