Standard for Split/Renamed Repos

[derek-ho]

Issue:
Recently, observability repo was renamed to dashboards-observability, and dashboards-reports was renamed to dashboards-reporting. To reflect this, the manifests for 2.4.1 and 1.3.7 were updated with the new repo names. Going forward, the backend folders of these repos will be moved to observability and reporting, respectively. This causes some confusion in terms of what to do with historical manifests for OS and OSD since they will not be accurate anymore and may need to be changed if artifacts need to be rebuilt.

Let us come to a decision and document it for any other similar situations going forward

Ask:

• Are there any other changes that need to be made from the build-repo side to support this name change and repo splitting/creation?

[derek-ho]

@CEHENKLE @dblock @peterzhuamazon @gaiksaya Please add any thoughts you have so we can set the standard and continue with the repo splitting/creation process, thanks! CC @vamsi-amazon

[derek-ho]

The same applies to sql repo, which will now be split into dashboards-query-workbench, sql-cli, sql-odbc, and sql-jdbc. @YANG-DB @sejli

[dblock]

What confusion or actual problem does this change cause? GitHub continues to redirect to old names and all git references remain stable. AFAIK no changes in previous manifests are required and everything continues to work.

[derek-ho]

@dblock here's an example to make the problem more clear in the 2.4.1 opensearch manifest file there is the following entry for opensearch-observability:

    repository: https://github.com/opensearch-project/dashboards-observability.git
    ref: tags/2.4.1.0
    platforms:
      - linux
      - windows
    working_directory: opensearch-observability
    checks:
      - gradle:properties:version
      - gradle:dependencies:opensearch.version

Once the new repo observability is created and the backend folder is moved there this entry will no longer be the correct location to get the plugin. The same issue exists for the front end plugin in manifests older than 1.3.7 and 2.4.1. Additionally we are looking into preserving git history in the new repo, but don't yet have a guarantee that we will have SHA/branching preservation in the new repo, so any references (I saw a few in the repo, but not sure if they are relevant) or dependencies may be broken CC @pjfitzgibbons

[pjfitzgibbons]

Yep. 👀 There are multiple workflows we'll have to iron out as soon as we get these repos online. We'll also have to copy/paste several workflows into the "new" repo to make builds run right. Infra/Build teams can help us with those, as well.

[pjfitzgibbons]

As far as history, I am leaning toward simply making a "cleanup" commit that excises the "unused" side of the codebase for each repo... past history will be duplicate in each side, some well-known tag will identify when we split.

I am open to thoughts and other approaches to this.

[dblock]

@dblock here's an example to make the problem more clear in the 2.4.1 opensearch manifest file there is the following entry for opensearch-observability:

    repository: https://github.com/opensearch-project/dashboards-observability.git
    ref: tags/2.4.1.0
    platforms:
      - linux
      - windows
    working_directory: opensearch-observability
    checks:
      - gradle:properties:version
      - gradle:dependencies:opensearch.version

Once the new repo observability is created and the backend folder is moved there this entry will no longer be the correct location to get the plugin.

I am confused what got renamed to what. Above you say "observability repo was renamed to dashboards-observability". So what is that we have in the manifest and what did we rename what to?

If you rename a repo, it has no effect on the manifest.

If you move a folder, you don't move it on an existing tag, you move it in another branch (e.g. main). So nothing should change there either.

[gaiksaya]

I think some information is missing from this issue. The issue will be in case of renaming and creating a new repo with old name.
For example, I believe the plan is to rename the current dashboards-reporting repo to reporting (backend plugin)
and then create a new repo called dashboards-reporting for dashboards plugin.
What should be the approach here @dblock ?

[dblock]

@gaiksaya I think we shouldn't do this and create a new repo called reporting.

[derek-ho]

@dblock yea reporting should be ok, but for observability we don't really have a choice, since we want to follow opensearch org naming convention of not including opensearch in the name. Thus, the only two feasible names are observability and dashboards-observability. Thus the conflict with no redirect available.... Regardless of if there is a redirect available or not, one half of the code base will be found at a different repo location, causing an issue with manifest accuracy.

[dblock]

@derek-ho I don't think I understand. The manifests for versions that have shipped are locked a revision. In that revision the code will not have moved yet. Nothing will break.

[derek-ho]

@dblock ok... I am not familiar with how manifest versions are locked on revisions. I just wanted to make sure we don't break anything from the build repo side. For example for 2.4.0, the build repo would be looking at https://github.com/opensearch-project/dashboards-observability.git under the folder opensearch-observability for the backend plugin, but once the new repo is created and redirect breaks, this would ideally be located at https://github.com/opensearch-project/observability.git. In case we need to rebuild anything I just wanted to make sure nothing will break or fail.

[dblock]

Your example is incorrect I think - 2.4.0 has a reference to observability here: https://github.com/opensearch-project/opensearch-build/blob/main/manifests/2.4.0/opensearch-dashboards-2.4.0.yml#L20

The original plan was to rename observability to dashboards-observability and create observability, which would create exactly the problem that you describe. Thus, it was a bad idea. Instead we left https://github.com/opensearch-project/observability alone, and https://github.com/opensearch-project/dashboards-observability was created (it's currently private waiting for some basic setup/files for a new repo).

Does this make sene? I believe this should be closed.

[derek-ho]

@dblock sorry I mistyped - I was talking about 2.4.1 I think this issue still exists for this manifest file since it was changed to dashboards-observability to be safe for the release - https://github.com/opensearch-project/opensearch-build/blob/main/manifests/2.4.1/opensearch-dashboards-2.4.1.yml#L20. We can close it, but just wanted to raise the concern and make sure everyone is on the same page about a path forward.

[dblock]

Ok, so for 2.4.1 that manifest is wrong - I imagine that happened half way through the rename and now we went back.You should PR a fix for it and change it back to observability like in 2.4.0 because now that https://github.com/opensearch-project/dashboards-observability exists and is empty, that manifest will break.

[pjfitzgibbons]

Yeah, eventually (Soon) dashboards-observabillity will not contain the backend code... so your suggested PR is necessary. Thanks and Kindest Regards, Peter Fitzgibbons (224) 307-9689

…

On Wed, Dec 21, 2022 at 10:32 AM Daniel (dB.) Doubrovkine < ***@***.***> wrote: Ok, so for 2.4.1 that manifest is wrong - I imagine that happened half way through the rename and now we went back.You should PR a fix for it and change it back to observability like in 2.4.0 because now that https://github.com/opensearch-project/dashboards-observability exists and is empty, that manifest will break. — Reply to this email directly, view it on GitHub <#3005 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACYNT47B7DLGTD7ZAHVG3WONEKXANCNFSM6AAAAAAS52MWIU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[derek-ho]

@dblock @pjfitzgibbons @gaiksaya @peterzhuamazon @gaiksaya @CEHENKLE @YANG-DB @rupal-bq @anirudha can we make a final decision about this - any concerns? Insights team (observability, sql, reporting) wants to release 2.5.0 with separate repos, and we need confirmation from build/infra team on if this is ok/any changes that need to be made to support this. Example PR to deprecate code from old repo: opensearch-project/observability#1362. New repos will soon be on parity with old ones. What will we be doing with old manifest files?

[dblock]

I re-read the thread and I am still confused. Old manifest files should continue working because of GitHub redirects and stable references. Do you have an example of a specific manifest file that is now broken with the changes above?

What will we be doing with old manifest files?

If such example doesn't exist I propose to close this issue.

[pjfitzgibbons]

We just want the build team to verify for us that the builds are working properly. As of yesterday (or wed?) I think 2.5 build was failing, though we don't specifically know the issues. Thanks and Kindest Regards, Peter Fitzgibbons (224) 307-9689

…

On Thu, Jan 5, 2023 at 3:33 PM Daniel (dB.) Doubrovkine < ***@***.***> wrote: I re-read the thread and I am still confused. Old manifest files should continue working because of GitHub redirects and stable references. Do you have an example of a specific manifest file that is now broken with the changes above? — Reply to this email directly, view it on GitHub <#3005 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACYNRGQEGGUAQSYYTPETLWQ5K3JANCNFSM6AAAAAAS52MWIU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[bbarani]

Closing this issue as we have officially split the monorepos With 2.5.0 release.