[PROPOSAL] Developer docs

[ashwin-pc]

What kind of business use case are you trying to solve? What are your requirements?

Currently we do not have easily accessible documentation for a developer who wants to either contribute to the repository or build plugins for OpenSearch and OpenSearch Dashboards. We should have an easily accessible place (Preferably a public website or as a section within the doc site) that allows developers to understand the architecture and major functional paradigms of the software. Some common topics include:

• The major architectural building blocks of the product
• How to build a plugin
• Core plugins, their API's, how they work and how they can be used.
• Testing principles

What is the problem? What is preventing you from meeting the requirements?

Each repository has many readme's that partially cover some of the architectural details of the project but finding them isnt easy and they are sometimes even outdated or incomplete. If a new developer wants to build a plugin, they must refer to these readme's or Elastic's documentation for information that may not be accurate. This makes developing for Opensearch extremely slow and difficult for someone who does not already know how everything works.

What are you proposing? What do you suggest we do to solve the problem or improve the existing situation?
Each repository used to have a doc site generator that was removed during the fork. It was meant to be replaced by the documentation site.

• Is bringing that to the documentation website still planned?
• Can we bring back the documentation generator until we have a better solution or until the documentation website is able to cover those topics?

What are your assumptions or prerequisites?

What are remaining open questions?

[ashwin-pc]

@elfisher @AMoo-Miki

[kavilla]

This issue mentions how to build a plugin, this issue: #171 can be a sub issue of this issue.

[elfisher]

It feels like this should be part of CONTRIBUTING.md and other documentation within each repo. Is there a reason we think the developer guides should go into the end-user documentation site versus documented within the project repos?

[kavilla]

That's a good call out. Right now we have a lot of developer friendly docs in our repos. So these do exist in our repo but it is so many that it would seem developers are not able to find it properly based on the issues that get opened.

[ashwin-pc]

There are 2 personas that i'm trying to solve for here.

1. A plugin developer: These are users that ideally should not need to look at the OSD codebase to make plugins that extend OSD. This is mostly possible today with the current plugin architeture, but the documentation needed to help such a developer is incomplete and scattered across the repo. This makes it such that a plugin developer must also be very familiar with OSD code and adding unnecessary friction to developing plugins.
2. OSD Developers: These are users who contribute directly OSD. They too require some of the information that plugin developers need, but they should also be familiar with OSD code and can rely to some extent on readme's in the code.

While markdown is powerful, there are some pitfalls by only relying on it to address our gaps in documentation for the two persona's mentioned above:

1. The docs are scattered (they are kept local to the component they are describing) so finding the relevant information can become tedious.
2. Cant incorporate features such as tutorials, search and nav tree to name a few
3. Requires the user to explore the OSD repo (For plugin developers, this is not ideal)

What i would ideally love to see is:

For plugin developers:
Either a section within the end user docs for plugin development specific documentation. It can be something as simple as a few tutorials with links to other resources like Graphana, or a plugin handbook like Wordpress

For OSD contributors:
A generated docsite that makes accessing the scattered readme's in the repo more easier line we used to have with asciidocs before the fork or like we have with EUI.

[dblock]

Two aspects omitted in the conversation above:

1. Branching. OpenSearch main requires JDK 17 right now, while 1.x requires JDK 11. There are dozens of branches in the product, times dozens of branches for plugins, and each has a different JDK and nuances of development that were true at the time of the development. Developer documentation needs to branch with the repos, and I am not sure how separate documentation can ever achieve supporting this kind of matrix.
2. Keeping documentation up-to-date. Right now when I develop a feature there's almost 0 chance it gets documented along with the feature, causing huge drift. If anything, we should be bringing documentation closer to the code rather than the other way around to solve this.

I do think we need developer guides that one can find from opensearch.org documentation. Those should describe how to start contributing to the various parts of the ecosystem with links to each repo. Those guides could/should also contain links or imported references for API documentation for various versions of the product.

[ashwin-pc]

@dblock Those are very good callouts. To your points:

1. Yes docs should be tied to branching in a way, but the docs i referring to are tied closely to the release version and not the branch specifically. And since we follow semver, within a major version we should not have changes that vary branch to branch. A good example of this is the API's, plugins available, high level system architecture. The docsite already supports this form of versioning. Branch specific contribution guidelines can still reside in the code since they can be easily put in readmes but more complex topics are quite hard to parse and find without a dedicated UI.
2. We already have a mechanism to trigger docsite updates when a new PR is created. So while i agree that having it stay beside the code, I'm not sure i agree that all that information is best stored there.

Guess its confusing to talk about it by being abstract about what these docs are so here are some examples of docs that i'd love to see in a website rather than a readme:

1. Tutorials like how to create a plugin.
2. High level architecture of how OpenSerach Core and Dashboards work
3. API docs
4. Speaking as a Dashboards dev. The core plugins available such as the saved object, expressions and data plugin. What they are, how to use them and their API's with examples.
5. Testing and Design patterns

The wordpress plugin handbook is a very good example of this.

the nuances of development can always stay in a branch right next to the code, but right now we dont have even a high level set of docs that new devs can refer to that cover some of the basics that a plugin dev needs to know that is easily searchable and readable.

[dblock]

I don't have any objections to creating a plugin development handbook part of https://github.com/opensearch-project/documentation-website.

The generated docsite is easy, sounds you're already doing it. No objections to moving some of the content out to ^.