fix: keep array entries not found in live state

[blakepettersson]

There's an issue with intersectMap not appending items for array properties where the live array is smaller than the desired array.

Hopefully fixes #9678.

Note on DCO:

If the DCO action in the integration test fails, one or more of your commits are not signed off. Please click on the Details link next to the DCO action for instructions on how to resolve this.

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this does not need to be in the release notes.
• The title of the PR states what changed and the related issues number (used for the release note).
• The title of the PR conforms to the Toolchain Guide
• I've included "Closes [ISSUE #]" or "Fixes [ISSUE #]" in the description to automatically close the associated issue.
• I've updated both the CLI and UI to expose my feature, or I plan to submit a second PR with them.
• Does this PR require documentation updates?
• I've updated documentation as required by this PR.
• Optional. My organization is added to USERS.md.
• I have signed off all my commits as required by DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My build is green (troubleshooting builds).
• My new feature complies with the feature status guidelines.
• I have added a brief description of why this PR is necessary and/or what this PR solves.

Please see Contribution FAQs if you have questions about your pull-request.

[crenshaw-dev]

This is probably a step in the right direction. It does fail to account for merge keys, though (e.g. in an env array, items should be merged on the name key).

[codecov]

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (129cf53) 50.01% compared to head (0b2c96a) 50.00%.
Report is 10 commits behind head on master.

❗ Current head 0b2c96a differs from pull request most recent head fb3d3ba. Consider uploading reports for the commit fb3d3ba to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #14602      +/-   ##
==========================================
- Coverage   50.01%   50.00%   -0.02%     
==========================================
  Files         266      262       -4     
  Lines       45971    45059     -912     
==========================================
- Hits        22992    22530     -462     
+ Misses      20731    20321     -410     
+ Partials     2248     2208      -40     

Files	Coverage Δ
controller/sync.go	63.43% <93.24%> (+7.51%)	⬆️

... and 54 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zswanson]

@blakepettersson anything you need help with to move this to the finish line?

[crenshaw-dev]

The new logic looks nicely done, and I appreciate all the unit tests. But I think we might have to address the underlying issue and see if we can implement strategic merge patching instead of plain json patching in ignoreDifferences.

[blakepettersson]

@zswanson time - it's something which sadly has been sorely lacking for me lately. When digging into this locally there are edge-cases upon edge-cases which I bump into, and this is sadly not as trivial as it seems in this PR (and it's already far from trivial).

The deeper I go into the rabbit hole, the more I think that this needs a rethink, and I agree with @crenshaw-dev that another approach is needed.

@crenshaw-dev when you say strategic merge patching, is that the same thing as Server-Side Diff (hence potentially addressed with #13663)?

[crenshaw-dev]

@blakepettersson not exactly related to server-side diff. It's been a while since I've looked into this code in depth, but iirc we're generating patches using a generic JSON patch library. To handle k8s resources, we really need to use k8s libraries to generate "strategic merge patches," i.e. patches that respect resources' merge keys.

[crenshaw-dev]

That response is vague, and it's because I don't remember this code well enough to remember exactly how strategic merge patching would be used to solve the problem. :-P

[saumeya]

Hi @blakepettersson @crenshaw-dev any help needed to move this fix to completion? We also need a fix for this issue.

[blakepettersson]

Hi @saumeya @zswanson et al - due to life circumstances I haven't had the time to look at this for the past couple of months. I just changed jobs, and there's a reasonable chance that I can revisit this PR in a not too long time frame from now. Before iterating on this PR further though, I would like to see if the actual issue could be better addressed by instead making use of Server-Side Apply / Server-Side Diff.

[blakepettersson]

Closing this PR since I don't think this is a viable way to go, see comment.