[Feature] Support Multiple Data Source when Import saved objects from uploading files

[yujin-emma]

Is your feature request related to a problem? Please describe.

When multiple data source enabled, user can add sample data and use the dev tool with one specified data source. Import saved objects accepts uploaded files, but the API does not know the data source. These objects cannot display since the it does not know where to fetch the data.

We want to provide user the ability to import saved objects after selecting one data source. When user import saved object from cluster where not enabled multiple data source feature, the imported saved objects will attach the dataSourceId with the objectId; while user import saved objects from some cluster with multiple data source feature enabled, then the objectId would already contains data source info, for this case, we want to replace the dataSourceId. The id schema are exactly same as when user add sample data and dev tools with multiple data source enabled.

Challenges

There are two modes to import saved objects, one is create new copy, the other is check conflict and override. We need to support both modes and cannot break the existing behavior.
Current Import API support create new copy by using the exact objectId to construct the importIdMap, support check conflict and override by checking if there is anything error and only push overrides to construct final import saved object request.
For these two modes, we need to resolve carefully, below are the detailed approach to support multiple data source when import saved objects.

Describe the solution you'd like

Mode - create new copy

When choosing create new copy, after user select one data source, attach the dataSourceId to the saved objects. The _ is between dataSourceId and objectId.

if (dataSourceId) {
      return acc.set(`${object.type}:${object.id}`, {
        id: `${dataSourceId}_${uuidv4()}`,
        omitOriginId: true,
      });
    }
    return acc.set(`${object.type}:${object.id}`, { id: uuidv4(), omitOriginId: true });

Mode - check conflict and override

Under this mode, if there is no conflict, import API would not add any object to importIdMap, while when multiple data source enabled and import saved objects, we need to first check if there is dataSourceId existing in the current objectId, we need to:

  1. attach if there is no dataSourceId in the objectId

// Sample code
if (noConflictError && dataSourceId) {
        importIdMap.set(`${type}:${id}`, { id: `${dataSourceId}_${object.id}`,id});
      } else {
        importIdMap.set(`${type}:${id}`, { id: uuidv4(),id });
      }

  2. replace the dataSourceId info when user specify one data source

// Sample code
if (dataSourceIdExited){
        const subId = id.split("_")[1];
        const newId = `${dataSourceId}_${subId}`;
        const newTypeArray = type.split("_");
        const newType = newTypeArray[newTypeArray.length - 1];
        importIdMap.set(`${newType}:${subId}`, { id: `${newId}`, omitOriginId });
 }

 // if conflict exist, replace the dataSourceId in the objectId and reconstruct the pendingOverwrites item
 const idArrayWithoutUnderscore = id.split("_");
 const newId = idArrayWithoutUnderscore.length >= 2? `${dataSourceId}_${idArrayWithoutUnderscore[idArrayWithoutUnderscore.length - 1]}` : id;
 const newTypeNoUnderScore = type.split("_");
 const newType = newTypeNoUnderScore.length >= 1 ? newTypeNoUnderScore[newTypeNoUnderScore.length - 1] : type; 
// extract the original type from the imported saved object
 pendingOverwrites.add(`${newType}:${newId}`); 

Additional context

"saved_objects": [
        {
            "id": "3ecd6420-ae62-11ee-a922-ffcd95c5e76a_37cc8650-b882-11e8-a6d9-e546fe2bba5f",
            "type": "visualization",
            "namespaces": [
                "default"
            ],
            "updated_at": "2024-01-09T01:51:16.078Z",
            "version": 14,
            "attributes": {
                "description": "",
                "uiStateJSON": "{}",
                "title": "[eCommerce] Sales by Category_emma-1-3",
                "version": 1,
                "kibanaSavedObjectMeta": {
                    "searchSourceJSON": "{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"searchSourceJSON.index\"}"
                },
            "references": [
                {
                    "name": "searchSourceJSON.index",
                    "type": "index-pattern",
                    "id": "3ecd6420-ae62-11ee-a922-ffcd95c5e76a_ff959d40-b880-11e8-a6d9-e546fe2bba5f"
                }
            ]
        },

[BionIT]

There are a few edges cases I can think of and make sure we handle them gracefully:

1. What is the behavior if datasource id already present in the prefix of the objects, and it either match or do not match the selected datasource, what does making a new copy look like and what does check conflict look like
2. Does the change affect the import API/UI if data source plugin is not enabled
3. If user export the objects from the dashboard that enabled data source plugin, all objects contain the datasource id in the prefix, but they are all different from each other, should user select or unselect data source, etc

[BionIT]

@kgcreative
Thanks for going over the UX and walking through the scenario for import saved objects UI! I tried the flow again, and since the uploaded file is only loaded when the import button is clicked, thus there can be 2 options:

1. have the source of data picker before the import button is clicked, while local cluster is selected by default, when click on Remote data source, then there is a option to choose from the datasources.
2. after import button is clicked, then the backend will check for the content of file and if any of them doesn't have datasource in the id, then show up error message and show the Source of data section.

Both are achievable, option 2 adds more complexity and only allow selecting data source after file is read, and by looking at index pattern management page when multiple data source is enabled, adding a new index pattern requires configure the data source first, thus wanted to hear your suggestion! The Source of data section only shows up when multiple data source plugin is enabled

The following are screenshots of the new section to pick data source, and it was made to match the style of the import options section, and want to hear your thoughts as well

• Local cluster is selected by default
  
• If Remote data source is selected, user can select connected data source
  

[BionIT]

Hi @kgcreative, if no concern, we would go with this layout with option 1.

[BionIT]

This is the older version of the layout and benefit is we will reuse the existing picker component

Video

Screen.Recording.2024-01-30.at.14.19.40.mov

[BionIT]

As discussed, this is the layout we will go with when multiple data source is enabled

[kgcreative]

LGTM -- if we can make the flyout a couple pixels wider so the last "IDs" text isn't all lonely in a second line, it would be ideal, but don't spend too much time on it.

[BionIT]

Thanks @kgcreative ! As discussed, the behavior and layout of the import saved object flyout when multiple data source is disabled would follow the same as above where 1) random ID will be changed to unique ID, and 2) create new objects with unique IDs will be the default option, 3) select a file to import will be changed to Select file

[kgcreative]

LGTM, thank you!

Cc: @dagneyb