Policy fields disappearing during upgrade process with conflict resolution

[endorama]

Kibana version: 8.1.0

Elasticsearch version: 8.1.0

Server OS version: NA

Browser version: Firefox 99

Browser OS version: NA

Original install method (e.g. download page, yum, from source, etc.): elastic-package stack up

Describe the bug:

Some variables "disappear" from UI when using the conflict resolution UI to upgrade an Agent policy to a new version.

The same variables are not present anymore in the Agent policy.
The same variables are not shown anymore in the Agent policy editor.

Steps to reproduce:

1. Add a policy using GCP integration version v1.5.0.
2. Ensure GCP integration v2.0.0 (attached to this issue) is available.
3. Upgrade the integration; Note: even when selecting "Upgrade integration policies" checkbox the policy is not updated
4. Go to "Integration policies" and click "Upgrade"
5. The conflict resolution UI is shown. Note that Project ID, Credentials File and Credentials JSON variables have disappeared and are not present, they should be at package level.
6. Save
7. Edit the policy. Note that Project ID, Credentials File and Credentials JSON variables are not present anymore (this is confirmed also when looking at the policy YAML: project_id is null, credentials_file and credentials_json are not present)

Expected behavior: Fields are shown in conflict resolution UI and in policy editor.

Screenshots (if relevant):

Any additional context:
This bug was identified in elastic/integrations#2987, with integrations developer oriented reproduction steps.
Issue #112272 is linked, as some discussion around this issue happened there.

Video showcasing the bug: https://user-images.githubusercontent.com/526307/165972437-b5c514bf-756f-4319-8ab0-fe5ba71c2b59.mp4
Agent Policy before upgrade: Agent policy before upgrade.yml.txt
Agent Policy after upgrade: Agent policy after upgrade.yml.txt
GCP v2.0.0 (unreleased) package: gcp-v2.0.0.zip

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

@endorama - I'm taking a look at this today, and I'm having trouble upgrading from GCP 1.6.1 to 2.0.0 in the Integrations UI because of some kind of saved objects error.

[2022-05-06T10:02:13.568-04:00][WARN ][plugins.fleet] Failure to install package [gcp]: [Error: Encountered 19 errors creating saved objects: [{"type":"dashboard","id":"gcp-6576c480-73a2-11ea-a345-f985c61fe654","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [dashboard.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'dashboard:gcp-6576c480-73a2-11ea-a345-f985c61fe654'. Preview of field's value: '{filter=[], query={query=data_stream.dataset:gcp.audit, language=kuery}}'","statusCode":500,"type":"unknown"}},{"type":"dashboard","id":"gcp-76c9e920-e890-11ea-bf8c-d13ebf358a78","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [dashboard.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'dashboard:gcp-76c9e920-e890-11ea-bf8c-d13ebf358a78'. Preview of field's value: '{filter=[], query={query=, language=kuery}}'","statusCode":500,"type":"unknown"}},{"type":"dashboard","id":"gcp-f40ee870-5e4a-11ea-a4f6-717338406083","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [dashboard.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'dashboard:gcp-f40ee870-5e4a-11ea-a4f6-717338406083'. Preview of field's value: '{filter=[], query={query=, language=kuery}}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-33ee1320-73a5-11ea-a345-f985c61fe654","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-33ee1320-73a5-11ea-a345-f985c61fe654'. Preview of field's value: '{filter=[], query={query=, language=kuery}}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-3f472ea0-5e47-11ea-a4f6-717338406083","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-3f472ea0-5e47-11ea-a4f6-717338406083'. Preview of field's value: '{}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-43f45ba0-5e4a-11ea-a4f6-717338406083","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-43f45ba0-5e4a-11ea-a4f6-717338406083'. Preview of field's value: '{}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-4627efa0-73a2-11ea-a345-f985c61fe654","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-4627efa0-73a2-11ea-a345-f985c61fe654'. Preview of field's value: '{filter=[], query={query=, language=kuery}}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-5d2f9160-e88e-11ea-bf8c-d13ebf358a78","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-5d2f9160-e88e-11ea-bf8c-d13ebf358a78'. Preview of field's value: '{filter=[], query={query=, language=kuery}}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-6d90d320-73a4-11ea-a345-f985c61fe654","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-6d90d320-73a4-11ea-a345-f985c61fe654'. Preview of field's value: '{filter=[], query={query=, language=kuery}}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-6f795e70-5e49-11ea-a4f6-717338406083","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-6f795e70-5e49-11ea-a4f6-717338406083'. Preview of field's value: '{}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-89513bc0-5e48-11ea-a4f6-717338406083","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-89513bc0-5e48-11ea-a4f6-717338406083'. Preview of field's value: '{}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-95e1f050-5e48-11ea-a4f6-717338406083","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-95e1f050-5e48-11ea-a4f6-717338406083'. Preview of field's value: '{}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-9d919d00-5e4d-11ea-a4f6-717338406083","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-9d919d00-5e4d-11ea-a4f6-717338406083'. Preview of field's value: '{}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-a8e40240-73a3-11ea-a345-f985c61fe654","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-a8e40240-73a3-11ea-a345-f985c61fe654'. Preview of field's value: '{filter=[], query={query=, language=kuery}}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-da5bc460-63e1-11ea-b0ac-95d4ecb1fecd","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-da5bc460-63e1-11ea-b0ac-95d4ecb1fecd'. Preview of field's value: '{}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-eb5bf570-73a2-11ea-a345-f985c61fe654","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-eb5bf570-73a2-11ea-a345-f985c61fe654'. Preview of field's value: '{filter=[], query={query=, language=kuery}}'","statusCode":500,"type":"unknown"}},{"type":"visualization","id":"gcp-ef1508c0-5e4c-11ea-a4f6-717338406083","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [visualization.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'visualization:gcp-ef1508c0-5e4c-11ea-a4f6-717338406083'. Preview of field's value: '{}'","statusCode":500,"type":"unknown"}},{"type":"search","id":"gcp-d88364c0-73a1-11ea-a345-f985c61fe654","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [search.kibanaSavedObjectMeta.searchSourceJSON] of type [text] in document with id 'search:gcp-d88364c0-73a1-11ea-a345-f985c61fe654'. Preview of field's value: '{filter=[{$state={store=appState}, meta={indexRefName=kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index, negate=false, alias=null, disabled=false, params={query=gcp.audit}, type=phrase, key=data_stream.dataset}, query={match_phrase={data_stream={dataset=gcp.audit}}}}], indexRefName=kibanaSavedObjectMeta.searchSourceJSON.index, query={query=, language=kuery}, highlightAll=true, version=true}'","statusCode":500,"type":"unknown"}},{"type":"map","id":"gcp-a97de660-73a5-11ea-a345-f985c61fe654","error":{"error":"Internal Server Error","message":"Unexpected bulk response [400] mapper_parsing_exception: failed to parse field [map.layerListJSON] of type [text] in document with id 'map:gcp-a97de660-73a5-11ea-a345-f985c61fe654'. Preview of field's value: '{visible=true, maxZoom=24, alpha=1, minZoom=0, sourceDescriptor={isAutoSelect=true, type=EMS_TMS}, style={}, id=866b5ce1-6ca0-47db-a6f2-54c5e0dcd2f0, label=null, type=VECTOR_TILE}'","statusCode":500,"type":"unknown"}}]]

I unzipped the attached gcp.zip file and placed its contents in a gcp/2.0.0 directory in my local package registry's package path. The package loads just fine in the Kibana UI, but I'm not able to upgrade to 2.0.0 or installed a fresh 2.0.0 policy.

Any idea what I might be doing wrong here?

[endorama]

We investigate it together and the error is related to a dashboard, Audit Source Locations [Logs GCP], that has been added in GCP 1.7.0 (elastic/integrations#3280, merged 6 days ago).
Given that the gcp-2.0.0.zip is older that dashboard does not exists so the migration breaks (this probably needs a separate issue, I would not expect it to break like this even if this migration path is unsupported)
Scratch this,

I tried with a fresh install and starting from GCP 1.5.0 I did not see the mentioned error. (I also updated the issue reproduction steps locking version to 1.5.0 to avoid this reproduction error in the future, thanks @kpollich for reporting the error, I should have specified the version).

Update: the dasbhoard has been added in 1.7.0 but a map named Audit Source Locations [Logs GCP] is present since GCP module creation (elastic/integrations#459): https://github.com/marc-gr/integrations/blob/b7efdc98f0b97676ac564b539b2e7074bf278bf6/packages/gcp/kibana/map/gcp-a97de660-73a5-11ea-a345-f985c61fe654.json

[kpollich]

I wound up needing to run the attached gcp-2.0.0.zip file contents through elastic-package build before placing them in local package registry service's data directory. I'll summarize my exact process that's working now below.

1. Extract gcp-2.0.0.zip file contents into $GOPATH/src/github.com/elastic/integrations/packages/gcp. I just did this through finder, but you can probably use unzip as well and handle this from the CLI if preferred.

You should see a git diff as such in your integrations repo now:

2. Run elastic-package build from integrations/packages/gcp

This will place the built version of the package in integrations/build/integrations/gcp/2.0.0

3. Copy the integrations/build/integrations/gcp/2.0.0 directory into the package-registry repo's build directory, e.g

# From the `integrations` repo
$ cp -r build/integrations/gcp/2.0.0/ ../package-registry/build/package-storage/packages/gcp/2.0.0

4. Run the package registry service via go run . in the package-registry repo

5. Set your registry URL in kibana.dev.yml

xpack.fleet.registryUrl: http://localhost:8080

6. Start up Elasticsearch and Kibana from the kibana repo

$ yarn es snapshot --license trial -E xpack.security.authc.api_key.enabled=true -E path.data=/tmp/es-data -E http.host=0.0.0.0 -E path.repo=/tmp/es-backups

# In another terminal session
$ yarn start

Should be able to install the GCP package as normal. I did only try this with @endorama's recommendation of installing 1.5.0 first to resolve the potential issue with the dashboard/map, but I was able to successfully install 2.0.0 after doing so. The map issue may not even be a persistent problem. The main issue was I just missed the build step when setting things up locally here 🙂

[kpollich]

Think I've got a fix for this (see below screen recording). Somewhat fortunately, I believe this is a UI-only issue - our backend merge/upgrade logic works as expected, we're just not displaying the variable fields properly in the UI.

Screen.Recording.2022-05-11.at.4.02.17.PM.mov

Getting a PR together now for the change.