Add more context data to embed API

[agjohnson]

While implementing embed docs topics via the API for our project admin pages, I got to play with the API and found some pieces that I wanted.

These might make good additions as API return or potentially we just do this as a javascript client side (which is ultimately how I'm manipulating the content now).

• More metadata around headers, such as heading level. I'd like to display the topics in a nested menu, as they don't make as much sense in sequential order
• Embed API should accept a language argument. I wasn't able to get this working and had to fall back to using a full URL
• Embed API could use accept language header of the client to determine the translation to give, this would be a great middle ground. That way topics embedded are also translated. For instance, we request embed from https://docs.readthedocs.io/en/stable always, but the embed API would use accept language header to lookup a translation, if it exists, and return the embed data from that translation.
• Same, but include an explicit translation argument or something for the accepted language
• Probably more of a client side feature, it would be great to break up the content block a lot more. Parsing out content from the sphinx output was too tedious, but it is possible. Data could include a list of sections with section header, section header nesting level, perhaps a truncated version of the content?

We should decide if any of these are worth pursuing and decide where they might land -- either as API features or on the client side javascript library TBD.

[humitos]

Embed API should accept a language argument. I wasn't able to get this working and had to fall back to using a full URL

In the v3 of this API we are only allowing ?url= argument and removing all the other ones to avoid Sphinx-specific concept here. So, I think this point is outdated now.

Embed API could use accept language header of the client to determine the translation to give, this would be a great middle ground. That way topics embedded are also translated. For instance, we request embed from https://docs.readthedocs.io/en/stable always, but the embed API would use accept language header to lookup a translation, if it exists, and return the embed data from that translation.

Same, but include an explicit translation argument or something for the accepted language

What would be the use case for this? I imagine that a set of English documentation should use Embed API with English URLs, Spanish docs should use Spanish URLs, and so on. What's a useful case to do http://docs.readthedocs.io/en/development/install.html?translation=ja from a project's English documentation. In this case, the tooltip will be in Japanese but the whole documentation in English, which is not good for me.

In case the front-end shows different languages on the same URL, the client could tweak the URL to change /en/ to /ja/ instead of adding ?translation=ja to an URL for an English document.

Besides, this is a Read the Docs-specific feature that won't have any effect on external sites.

(In case we really want this behavior on the Embed API, we should think about implementing this at the core of Read the Docs: hitting a Japanese version of a documentation's page that does not exist but the English/source language does, it should redirect to it. Example: https://docs.readthedocs.io/ja/development/install.html -> https://docs.readthedocs.io/en/development/install.html)

[agjohnson]

In the v3 of this API we are only allowing ?url= argument and removing all the other ones to avoid Sphinx-specific concept here. So, I think this point is outdated now.

How are we planning on returning localized embed content to the user? The embedding source needs to specifically ask for the language as part of the URL?

What would be the use case for this?

To clarify, I mentioned that this was for project dashboard page -- I'm using the embed client directly. It sounds like you're mostly considering the use case of my suggestions for hoverxref.

My use case was that I would like localized embedded content in the dashboard UI -- not in our docs. Currently, it sounds like what you're describing is that I would:

• Get the locale requested by the user on the dashboard
• Manually edit the URL to use /es/latest
• Hope there is content at /es/latest
• Embeds have localized strings

So hopefully that illustrates the problem a bit more. If there is no spanish locale, with content at /es/latest, embed API will return nothing, correct?

What's a useful case to do http://docs.readthedocs.io/en/development/install.html?translation=ja from a project's English documentation.

I'm describing a translation argument to the embed API, not our docs pages. It would make things a lot nicer if we had a way to specify that the embed API should return content localized in Spanish first, and if no translation exists, return content for the default translation instead.

I only suggested a translation or language or whatever argument, but I think it is important. I don't think we can do this with just the URL.

[humitos]

The embedding source needs to specifically ask for the language as part of the URL?

Yes.

If there is no spanish locale, with content at /es/latest, embed API will return nothing, correct?

Yes. It will return 404.

I'm describing a translation argument to the embed API, not our docs pages.

Me too. The only way of using Embed APIv3 is passing ?url= argument as ?url=http://docs.readthedocs.io/en/development/install.html. In my example, I just added the &translation=ja argument as in ?url=http://docs.readthedocs.io/en/development/install.html&translation=ja and I think it doesn't make too much sense; the client should just do one call for /ja/latest/ (translation), and if 404 is returned another one for /en/latest/ (default language)

[agjohnson]

I think some of this confusion comes from how you are using the endpoint in hoverxref and how customers would be using the endpoint directly, or via a JS client. Did we combine the concept of 2 or more API endpoints into one at some point? I'm not exactly sure where my use case -- reusing documentation metadata like TOC and heading, available from Sphinx -- is supposed to be sourced from now. It might help to talk through the use case I have, as a user of the embed data directly.

Me too.

I'm confused then, what is the URL http://docs.readthedocs.io/en/development/install.html?translation=ja that you posted?

In the v3 of this API we are only allowing ?url= argument and removing all the other ones to avoid Sphinx-specific concept here. So, I think this point is outdated now.

So this is confusing then, it doesn't seem like there is anything Sphinx specific about language, version, and path params. As a direct user of the API, these params seemed like a better design than url because I probably want to ask the API for a variable language or even version -- both core RTD features. Why did we move this to just url?

Overall, this isn't a huge blocker. Users shouldn't need to use this endpoint directly. We should have a JS client library, and that client should handle reconstruction of URLs with variable language/versions. This does seem a bit unnecessary though. It seems like we're duplicating application logic in JS because of a constraint we introduced in the middle to make the endpoint driven by url params. Again, this is for usage outside hoverxref mostly.

[humitos]

As a direct user of the API, these params seemed like a better design than url because I probably want to ask the API for a variable language or even version -- both core RTD features. Why did we move this to just url?

We move away from it because:

• there were Sphinx-specific arguments involved, like doc=
• to solve that we added path=; and we are guessing the doc from the path now
• url= is required to support intersphinx and pages not hosted at RTD
• having 2 different ways of using the API is confusing (url= or all these other arguments)
• URL contains all the arguments in just one argument
• path= is a little confusing because it's not what a regular URL path is --it's the path but without the /en/stable/ part of it (language and version)
• language, version, path don't make sense for external pages

Because of these points (and maybe others that I don't have in mind now), we thought it would be good to simplify its usage to the general case that support all the cases we had.

The simpler to better to me, and I think that's the usage of url= argument. Unless there are cases that are not supported/possible by url= I wouldn't introduce the extra complexity.

(re multiple arguments, we are talking about this at #8222 (comment) as well --just linking as reference here-- it's a 🌶️ topic, hehe)

[agjohnson]

So I thought about this a bit more after our discussion.

The above does make sense, though I might have an broader interpretation of how these endpoints might be used because I'm considering a different feature -- driven by a common JS client library that we would maintain. If we were to support application resolution of default locale/version with our embed API endpoint, two endpoints seems like it would be the best direction -- one endpoint for querying embed data on your RTD project (supporting language/version/path), one endpoint for your RTD project to query embed data other projects.

I thought a bit through how I might author a client JS library, and what usage would look like, and I came back to the conclusion that an endpoint that only supports url is probably okay. Like I mentioned, it does push core application logic back on the client JS, but this would at least be hidden from the user.

For illustration, this is what I would like to maintain in my application, as a potential user of Read the Docs toctree embedded inside my application:

I would bring a JS client library into my application's HTML. An illustration of what a template in my application HTML could look like would be:

<div
  data-embed="toc"
  data-embed-url="https://docs.readthedocs.io/{{ request.user.locale }}/{{ settings.VERSION }}/features/foo.html"
  data-embed-fallback-url="https://docs.readthedocs.io/en/stable/features/foo.html">
</div>

This is actually fairly clean, it's not bad compared to other options.

While manipulating the URL and fallback URL directly isn't great, the endpoint doesn't matter too much if we are not going to support resolving to a default locale/version in our API endpoints. If we are pushing this resolution onto a client JS library, the best we can do is to try a URL, and retry the fallback URL if we get a 404. The above example does require 2 requests whenever the preferred URL doesn't exist, and only supports the combinations preferred_locale/preferred_version and default_locale/default_version.

Multiple requests can be avoided by resolving locale/version at the API endpoint, if we are to implement this. Multiple requests isn't awful, but does double latency.

[agjohnson]

Anyways, that leaves heading nesting on my list, and we weren't clear on that one, because it sounded hard to do by parsing HTML.

[humitos]

I think we moved away from the idea of showing "automatic section links" in our new dashboard, and we changed this pattern to use links manually added in the template itself. I think we can close this issue without implementing it in the new Embed APIv3.

[agjohnson]

Well, we moved away mostly because this feature was removed in the API and we couldn't continue building on top. For now, we're manually adding links and link text to the application, and these are bound to break or page titles/etc will go out of sync.

I still think there is a really good user feature that we're not developing here. The feature I've wanted to give users is a way to specify a page name in an external site and our API and some JS turn that page name into a link to the page, with the page title resolved. This is basically intersphinx in HTML, for external sites. It would be great if the page TOC was surfaced too, but that is sounding hard.

But maybe the best direction here would instead be an embedded search result feature for external sites. In this case, the user would instead only provide a search term, and client-side JS would hit our API to get the results and output the page and page links. This would return hits to specific sections, and would automatically get up to date page title information/etc.

This wouldn't involve parsing anything we're not parsing already, and would make users more familiar with our search. This might be an entry point for using our addons JS on external sites even -- for example, an embedded search block web component that can open our search as you type feature.

Maybe it's even already possible to build on top of search like this?

[humitos]

IMHO (and pretty personal also), I think my understanding of what you are describing looks complex comparing the value / use case we are talking about here: "show some help links (with page titles in sync) in the dashboard". Maybe this issue could be treated as a new feature request and we can have a more high level conversation about it. That conversation may help me understand more the benefits of a feature like this and be more motivated to collaborate with the idea.

Currently, the pattern we have for our own dashboard and our workflow seems enough. Mainly, because we don't update our documentation that often to get those page titles out of sync frequently. Even if they get out of sync, it's not a huge problem if the titles don't exactly match and also, they are easy to update.

I wouldn't build this if it's just of us and we don't have customers asking for this feature with more details about their exact use-case.

[agjohnson]

use case we are talking about here: "show some help links (with page titles in sync) in the dashboard"

I'd say an embedded search feature would be a bit more than this use case though, as the point would also be to surface the documentation search UI.

I wouldn't build this if it's just of us and we don't have customers asking for this feature with more details about their exact use-case.

Yeah this would definitely be more on the experimental side. But it would take a proof of concept either way I think. Users aren't always going to tell us what features to build, we'd need to give them something to use first for something like this.