Allow custom new platform plugin paths in integration tests

[azasypkin]

We used to create custom plugins for various integration tests that expose custom endpoints (e.g. to serve as fake IdP) and whatnot. Currently it's possible to load new platform plugins from custom paths, but only when Kibana is run in dev mode. Unfortunately this doesn't help since integration tests on CI are run against "prod" Kibana.

/cc @legrego

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[joshdover]

@epixa Previously in #33865 we decided to only support this in development. Any problem with opening this up to production mode?

[epixa]

The issue is that third party plugins cannot reliably use relative imports with this setting, which is why we removed it from the distributions. I don't think that can change until we support/recommend root level imports instead. We'd have to play around with them in that case.

If integration tests are the only driver here, why not update the setup logic to move them into the plugin directory just like any other external plugin?

[pgayvallet]

@epixa

If integration tests are the only driver here, why not update the setup logic to move them into the plugin directory just like any other external plugin?

I'm not sure to see how this could work, specifically because of the relative imports you speak about:

I.E:
We got a test plugin in x-pack/test/plugin_api_integration/plugins/encrypted_saved_objects which is importing

import {
  PluginSetupContract,
  PluginStartContract,
} from '../../../../plugins/encrypted_saved_objects/server';

If we wants to move it from x-pack/test/plugin_api_integration/plugins to x-pack/plugins before running the suite, this import will actually breaks, as the plugin is no longer in folder that import was relative to.

So it seems root level imports are required for this to work.

One other option would be to use 'rebased' imports, meaning that the files in x-pack/test/plugin_api_integration/plugins/encrypted_saved_objects should use relative imports as if it's actual path was x-pack/plugins/encrypted_saved_objects. But this will makes developing test plugins a nightmare, as we will loose all IDE support.

As this is only for integration tests (and as integration tests needs to run in production mode), can't we just simply add an additional 'internal' flag to allow --plugin-path in production mode? Something 'generic' such as --enable-plugin-path-in-production, or something more 'honest' such as --integration-test-mode.

We could then deprecate/remove this option when we actually supports absolute imports and implements what you suggest then (using abs imports + moving test plugins during test setup)

WDYT?

[pgayvallet]

For the record, after a slack discussion with @azasypkin

Soo, there are at least two sets of security-related API integration tests that use custom test-only plugins:

1. https://github.com/elastic/kibana/tree/master/x-pack/test/plugin_api_integration/plugins/encrypted_saved_objects
2. https://github.com/elastic/kibana/tree/master/x-pack/test/oidc_api_integration/fixtures/oidc_provider
   In the first case we want to test how other plugins would work with the public API NP plugin exposes, and in the second case plugin simulates 3rd party service we'd normally use in production (acts as Identity Provider). Both these test plugins expose HTTP endpoints we use in tests.

[epixa]

A stopgap configuration that makes it clear it is for testing (like your integration-test-mode flag) would probably be fine, though to be clear: removing this flag would be a breaking change and would have to be limited to a major version.

The best option IMO is to just do root level imports and remove this restriction entirely. The proposal for root level imports was widely favored, so now just a matter of doing it. @spalger was exploring an approach with babel, and I played briefly with a symlink approach that worked well. I don't have time to work on it at this point, and I don't know where @spalger landed on his effort, but if he's stalled as well and you wanted to tackle this plugin path problem, perhaps you could implement it?

[pgayvallet]

Relates to #40446 then, I guess.

[pgayvallet]

After a discussion with the platform team on slack:

• Absolute path handling is obviously something we want to implement. I will take over @spalger work as he currently do not have the capacity to do it now. However, this was not really planned and it's not a small ticket (even if spencer did most of the job already), so this will probably have to wait after 7.6, I don't have full capacity for this right now (Add absolute path import handling in kibana #52995).

• In the meantime, we came to the conclusion that when An alternative to relative imports for local source code #40446 fully lands, the choice we did about blocking custom plugin paths in production will not make much sense anymore, as the absolute path handling will suppress the main (only?) reason we decided to disable them in the first place. This is why for now, we decided to re-enable the custom plugin paths for NP plugin in production, which will address this ticket. We will add an explicit warning stating custom plugins using relatives imports to kibana are discouraged and might not work properly in production. Once abs paths are implemented and when (if?) we totally enforce all imports to be absolute, we could then remove the warning.