[Fleet] Integration update fails when changing subobjects property

[jsoriano]

Same issue as reported in #183496 for custom integrations is happening also for integrations distributed in packages, as reported in elastic/integrations#10972.

As mentioned in the latter issue, this can be reproduced by trying to update from Okta 2.11.0 to 2.12.0.

On these cases a rollover should be performed by Fleet on upgrade.

Current workaround is to update the package using the install API with ignoreMappingUpdateErrors=true and rollover the data stream. For example for a local environment and the logs-okta.system-69312 data stream:

curl -X POST "https://localhost:5601/api/fleet/epm/packages/okta/2.12.0?ignoreMappingUpdateErrors=true" -H "Content-Type: application/json" -H "kbn-xsrf: true" -u elastic:changeme -k
curl -X POST "https://localhost:9200/logs-okta.system-69312/_rollover" -k -u elastic:changeme

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[terrancedejesus]

@jsoriano thanks for the workaround. Did the following in the dev console for the OOTB Okta integration and it worked.

POST kbn:/api/fleet/epm/packages/okta/2.13.0?ignoreMappingUpdateErrors=true

POST /logs-okta.system-default/_rollover

[aleksmaus]

@jsoriano @kpollich We stumbled on the same issue when changed the top level mapping to enabled: false for cisco_ise integration.

mapper_exception Root causes: mapper_exception the (enabled) parameter can't be upgraded for the object mapping [cisco_ise.log.cisco_av_pair].

The workaround for Okta https://support.elastic.dev/knowledge/view/1a2f83e7 mentions that this will be resolved in 8.17
Could you please confirm and add the ticket or PR that addresses this issue?
We are currently reverting this mapping change, but planning to put it back with package constraints on 8.17.

[jsoriano]

Could you please confirm and add the ticket or PR that addresses this issue?

From what I can tell, this issue hasn't been addressed, I can still reproduce it in 8.17 snapshots.

I am reproducing it with elastic-package, from the integrations repository:

$ git checkout ba948537e687b2a2501cee10f97dd33a0c77094f # okta 2.11.0
$ cd packages/okta
$ elastic-package stack up -v -d  --version 8.17.0-SNAPSHOT
$ elastic-package test system -v --setup --config-file ./data_stream/system/_dev/test/system/test-api-key-config.yml

And then trying to upgrade from the UI.

The workaround for Okta https://support.elastic.dev/knowledge/view/1a2f83e7 mentions that this will be resolved in 8.17

I think this issue was indeed originally planned for 8.17, but it seems to be in the backlog now 🤔

[juliaElastic]

I tried to reproduce the issue on local (latest main) by installing okta-2.11.0 and upgrading to latest 3.0.0 but couldn't yet.
I noticed the subobjects check is already in the code added by this pr: #186991.
Will check again on 8.17.0-SNAPSHOT to see if it can be reproduced.

Could reproduce now by installing version 2.11.0 and then installing 2.12.0.

POST kbn:/api/fleet/epm/packages/okta/2.11.0
{
  "force": true
}

 POST kbn:/api/fleet/epm/packages/okta/2.12.0
{
  "force": true
}

[2024-12-03T10:57:18.341+01:00][ERROR][plugins.fleet] ResponseError: mapper_exception
        Root causes:
                mapper_exception: the [subobjects] parameter can't be updated for the object mapping [okta.debug_context.debug_data]
 at KibanaTransport._request (/Users/juliabardi/kibana/node_modules/@elastic/transport/lib/Transport.js:543:27)
    at async /Users/juliabardi/kibana/node_modules/@elastic/transport/lib/Transport.js:641:32
    at async KibanaTransport.request (/Users/juliabardi/kibana/node_modules/@elastic/transport/lib/Transport.js:637:20)
    at installPackageWithStateMachine (/Users/juliabardi/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.ts:559:13)
    at installPackageWithStateMachine (/Users/juliabardi/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.ts:542:8)
    at _stateMachineInstallPackage (/Users/juliabardi/kibana/x-pack/plugins/fleet/server/services/epm/packages/install_state_machine/_state_machine_package_install.ts:123:43)
    at installPackageWithStateMachine (/Users/juliabardi/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.ts:523:81)
    at async installPackageFromRegistry (/Users/juliabardi/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.ts:399:12)
    at async installPackage (/Users/juliabardi/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.ts:748:22)

Interestingly when upgrading to 3.0.0 doesn't give an error and the rollover happens correctly.

[2024-12-03T11:02:51.802+01:00][INFO ][plugins.fleet] Mappings update for logs-okta.system-default failed due to ResponseError: illegal_argument_exception
        Root causes:
                illegal_argument_exception: can't merge a non object mapping [okta.target] with an object mapping
[2024-12-03T11:02:51.802+01:00][INFO ][plugins.fleet] Triggering a rollover for logs-okta.system-default