Issue Description Length Check (Sync Test - Ignore this issue and the description it contains!) (From Jira)

[petrklapka]

Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.

Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.
Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.
Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.
Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.
Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.
Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.

Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.
Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.
Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.
Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.
Update: to protect users data we have decided to prevent upgrades from succeeding if unknown saved object types are encountered.

Context (newest-oldest) on this discussion:

• Filter out documents of unknown types during migrations #104690
• SO migration v2: fail the migration early if unknown types are encountered #101351
• Fail v2 Migrations when unknown document types are found #101052

In order to improve the reliability of the Saved Object migration system, we'd need to eliminate situations that can cause data integrity issues. One of those situations is with how we currently handle objects of unknown types. The current behavior will retain these documents in the Kibana index, unmodified from their previous state. Whenever the type becomes known again (by re-enabling a plugin), Kibana will migrate this document to the most recent schema.

Supporting this is becoming ever more challenging as we expand our ability to do different types of migrations. For example, this behavior will result in corrupted state if there are any other objects that reference objects of a disabled type during the multi-namespace migration and then that type is enabled again later.

Data integrity is important for the long-term maintenance of a Kibana installation and is especially important to long-term, wide-scale users of Kibana. If we want to eliminate this type of data corruption, then we need to take some action here to prevent such scenarios.

The scenario where the migration system encounters objects of unknown types can occur in the following situations:

First-party plugins

• When a 1st party Elastic plugin is disabled after storing some data
  • We plan to stop allowing most plugins to be disabled starting in 8.0 By default, a plugin should not be disable-able #89584 which should dramatically mitigate, if not eliminate, this scenario going forward.
• When a Saved Object type is removed from usage from a 1st party plugin
  • This scenario has been eliminated by Add test that ensures that SO types are not removed #104418 which requires that all 1st party types that are removed are added to filter to be excluded from the next upgrade.

Third-party plugins

• When a 3rd party plugin is disabled or uninstalled after storing some data
  • This may be especially common for 3rd party plugins due to how we require plugins to be built specifically for each Kibana version. It's possible that a user may decide to upgrade their cluster without some custom plugins installed and then install them again later when they've been updated. In this case, I think users expect their data to be intact.
• When a Saved Object type is removed from usage from a 3rd party plugin
  • There's really not a way with the current architecture that we could easily detect this scenario distinctly from the scenario above, so the options are essentially the same.

Solutions

For users that only use 1st party plugins the solution seems quite straightforward. Given that we've already made progress on mitigating the ways this issue could happen, we either:

1. Refuse to start Kibana if any objects of unknown types exist in the index
2. Automatically filter for only for documents of known types and either leave the unknown documents in the previous index or move them to a special .kibana-orphaned index (for example).

(2) is preferable from a user perspective since it unblocks users from upgrading Kibana quickly, but has the drawback of 'silently' excluding documents.

The challenge is how can we handle the scenario with 3rd party plugins where they may not be installed during an upgrade. I propose we go with solution (2) and add a mechanism for importing & migrating documents from the .kibana-orphaned index that are detected as now being known by any plugin. This allows us to keep the default, happy path as safe as possible while giving more advanced users a way to recover their data in ways that may or may not be 100% integral. This mechanism could be exposed via either a config option or a UI prompt to import these objects after Kibana has started (which has some drawbacks but may work for most 3rd party plugins).

For customers that absolutely need 100% data integrity, we can recommend they ensure that all 3rd party plugins are installed during their upgrade rather than afterwards. For all others, we have an escape hatch that will probably work most of the time, but is not guaranteed and therefore not enabled by default.