[Observability] Create context container to enable Observability plugin registry function

[cauemarcondes]

closes #68534

Creates a context registry on the Observability plugin. The idea is that any plugin can registry its own way to fetch data.

Registring:

plugins.observability.dashboard.register({
  appName: ObservabilityApp,
  fetchData: (params: FetchDataParams) => Promise<FetchDataResponse[]
  hasData: () => Promise<boolean>
});

• fetchData: Returns the data that will be used to display the charts on the Overview page.
• hasData: Returns a boolean indicating if there's any data available in the APP indices. It'll be used to define if the Landing page should be opened instead of the Overview page, in case of all APPs return false.
  • terminateAfter: 1 can be used in the ES request since no data needs to be returned.
  • When a user doesn't have read privilege to the indices, it should return false.

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[cauemarcondes]

retest

[weltenwort]

I can't comment on the APM side of things, but this looks like a good start. For the data fetching to work, though, the service also needs to expose a way for the solutions to register custom context providers, which make data access available within their respective data fetcher functions.

[cauemarcondes]

retest

[weltenwort]

I think we're getting close 📈 I really like the symmetric substructure of the obs data service 👍 I just left a few suggestions about making its API more consistent below.

[jasonrhodes]

• I was wondering to have this name pre-defined on Observability like apm, logs, uptime, metrics, WDYT?

Can we bump this type up and export/share it in all the places we need to use these names for shared interfaces between the apps? https://github.com/elastic/kibana/blob/master/x-pack/plugins/observability/public/hooks/use_track_metric.tsx#L20

[jasonrhodes]

• context: is any information you want to inject to the function that will be executed

I thought "context" was provided by Kibana core, with pre-scoped clients etc similar to how HTTP routing handlers work. If we are providing the values for context, do we know exactly what values we are going to provide so that all apps have what they need to make their calls?

This part is still really fuzzy to me.

[cauemarcondes]

f we are providing the values for context, do we know exactly what values we are going to provide so that all apps have what they need to make their calls?

Maybe I misunderstand your question, but each plugin should define its own context. The Observability itself won't add any information.
e.g:

// plugin.ts
public setup(core: CoreSetup, plugins: ApmPluginSetupDeps) {
  plugins.observability.dataAccess.registerContext(
      this.initializerContext.opaqueId,
      'createCallApmApi',
      () => createCallApmApi
  );
  plugins.observability.dataAccess.registerProvider(
      this.initializerContext.opaqueId,
      'apm',
      getObservabilityChartData
  );
}

[jasonrhodes]

but each plugin should define its own context

Got it, so I'm trying to wrap my head around these layers of abstraction. How are these two things different:

1:

import { callApi } from '../callApi';

const getChartData = (context, { start, end }) => {
  context.callApi({ start, end, otherArgs });
}

plugins.observability.dataAccess.registerContext(
  this.initializerContext.opaqueId,
  'callApi',
  () => callApi
);
plugins.observability.dataAccess.registerProvider(
  this.initializerContext.opaqueId,
  'apm',
  getChartData
);

vs

2:

import { callApi } from '../callApi';

const getChartData = (context, { start, end }) => {
  callApi({ start, end, otherArgs });
}

plugins.observability.dataAccess.registerProvider(
  this.initializerContext.opaqueId,
  'apm',
  getChartData
);

[cauemarcondes]

but each plugin should define its own context

Got it, so I'm trying to wrap my head around these layers of abstraction. How are these two things different:

Conceptually, none. In both cases, you'll have callApi available, one through the context and another with closure.

But what if your getChartData needs to use some information available in another plugin lets say security ? You still could use closure to solve it, but you'd need to keep the function in the same scope where the licensing plugin is available.

public setup(core: CoreSetup, plugins: ApmPluginSetupDeps) {
  plugins.observability.dataAccess.registerProvider(
        this.initializerContext.opaqueId,
        'apm',
        () => {
          // do anything cool
          const isSecurityEnabled = plugins.security?.license.isEnabled();
        }
      );
}

And if you want to isolate your function in another file, you wouldn't have the security plugin in scope, so you'd do:

//plugin.ts
public setup(core: CoreSetup, plugins: ApmPluginSetupDeps) {
  plugins.observability.dataAccess.requiresAppContext(
        this.initializerContext.opaqueId,
        () => ({
          security: plugins.security,
        })
      );
  plugins.observability.dataAccess.registerProvider(
        this.initializerContext.opaqueId,
        'apm',
       getChartData
      );
}

//get_chart_data.ts
const getChartData = (context, { start, end }) => {
  const isSecurityEnabled = context.security?.license.isEnabled(); 
  // do anything cool
}

[cauemarcondes]

@elasticmachine merge upstream

[jasonrhodes]

But what if your getChartData needs to use some information available in another plugin lets say security ? You still could use closure to solve it, but you'd need to keep the function in the same scope where the licensing plugin is available.

Right, ok. And if this function was being used in your own plugin, you could pass it the things it needs directly or using some React context/hook combination. The alternative would be to have the observability plugin call each registered function with a set of commonly needed interfaces, e.g. licensing, security, es client, etc and then the registered methods would use them as passed in.

I can see the advantage of defining your own context giving you the ultimate flexibility in what your registered function can do, but it seems to come at the expense of making everything much harder to follow. I'm curious if anyone else has thoughts on this pattern?

[jasonrhodes]

Another thought: at the very least if each calling plugin need to pre-register its own function's context value, could it at least happen inside of registerProvider? Something like:

plugins.observability.dataAccess.registerProvider({
  opaqueId: this.initializerContext.opaqueId,
  name: 'apm',
  handler: getChartData,
  providedContext: { // or just context, whatever
    getData: () => {}, // all provided values here
  }
});

I'm not sure how much control over this API we have vs what comes from the registry context tooling we are using from core, but I think that at least might make it a little clearer how these things are related...

[smith]

This looks good for a first pass. I share some of @jasonrhodes concerns about the indirection and complexity, but don't have any constructive feedback for how we would do it differently. This is a great place to start and evolve from as needs change and we have experience using this in our plugins.

[cauemarcondes]

The alternative would be to have the observability plugin call each registered function with a set of commonly needed interfaces, e.g. licensing, security, es client, etc

We can do it. I can pre-register some plugins in advance.

at the very least if each calling plugin need to pre-register its own function's context value, could it at least happen inside of registerProvider?

I will create an example as you propose.

[cauemarcondes]

@jasonrhodes I made the changes as you suggested, it looks simpler in my opinion. Take a look here 0fcbcd2

plugins.observability.dataAccess.registerProvider({
  pluginOpaqueId: this.initializerContext.opaqueId,
  dataType: 'apm',
  handler: getObservabilityChartData,
  providedContext: {
    licensing: plugins.licensing,
    home: plugins.home,
  },
});

Then to access it in the handler function you'd do context.[dataType].[contextName].

WDYT?

[jasonrhodes]

I created Observability own registry for the sake of comparison

Cool so if I understand correctly, in this case we'd need to explicitly type the contract that the obs plugin provides when it calls these dataFetcher callbacks, right? Do we know what that contract would need to look like?

[weltenwort]

Are plugins using this registry provider outside of core? I think I heard that alerting uses it in some way? I think its apparent lack of documentation/examples is another thing that’s made it hard to follow or grok (maybe I’m missing those?).

A quick grep shows it's used outside of core by the data plugin to register search strategies.

Inside core it's also used by...

• the application service to register applications.
• the http plugin to register routes.

I'm surprised the number is so low. I would have expected things like the task manager (and thereby alerting), visualizations, expression functions and so on to use it to for their registries, but they implement custom mechanisms. Maybe because they predate the context service?

I think our use case is a perfect match, but if it turns out that not using it is simpler and easier I can follow that argument.

[weltenwort]

I created Observability own registry for the sake of comparison.

What's the argument against coupling this to the obs plugin life-cycle? An implicit singleton seem unnecessarily hacky. I hope we're not conflating this with the context container topic. Those are different aspects.

[cauemarcondes]

I created Observability own registry for the sake of comparison

Cool so if I understand correctly, in this case we'd need to explicitly type the contract that the obs plugin provides when it calls these dataFetcher callbacks, right? Do we know what that contract would need to look like?

@jasonrhodes I believe this is what you want:

kibana/x-pack/plugins/observability/public/typings/data_fetcher/index.ts

Line 40 in a4b0bfe

export type DataFetcher = (searchParams: SearchParams) => Promise<FetcherResponse[]>;

[jasonrhodes]

@cauemarcondes ok so if a dataFetcher needs access to e.g. the security plugin, etc. it will need to manage that itself via a closure when it registers the fetcher?

I thought we were going to explore providing a contract of core APIs at call time, but that's less flexible than letting the apps manage that for themselves, so I'm okay with either.

[cauemarcondes]

@cauemarcondes ok so if a dataFetcher needs access to e.g. the security plugin, etc. it will need to manage that itself via a closure when it registers the fetcher?

Correct.

[sorenlouv]

The context container approach is overly complex for what we need imo. If we need the additional complexity down the road we can always add it. But let's start simple if possible.

[jasonrhodes]

This LGTM as a great starting point, thanks @cauemarcondes! I think we will probably realize we are all sharing enough of the same deps that we may eventually provide them via the FetchDataParams as a common contract/interface, but we can definitely add that later once we see how each solution is building its fetch data function.

[sorenlouv]

lgtm

[cauemarcondes]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: aa6a94d

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
apm	1610	-1	1611
infra	1610	-1	1611
observability	19	-1	20
uptime	1409	-1	1410
total	-	-4	-

page load asset size

beta

id	value	diff	baseline
/bundles/plugin/observability/observability.plugin.js	102.9KB	+561.0B	102.4KB

History

• 💚 Build #53813 succeeded f2d58df
• 💔 Build #53662 failed 7d09e97
• 💚 Build #53543 succeeded 4f85e9a
• 💔 Build #53273 failed a4b0bfe
• 💚 Build #53241 succeeded fe126ac

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream