VCH Upgrade and Revert not refreshing cVM status

[malikkal]

Steps to reproduce.

1. Upgrade VCH to newer build.
2. Delete cVMs using the upgraded VCH.
3. Revert snapshot; i.e. rollback using vic-machine.
4. Docker ps will continue to display previously deleted cVMs.
5. State wouldn't be refreshed regardless of wait time.

[hickeng]

@emlin Will the vic-machine rollback enforce a power cycle or is it reverting to an online snapshot?
If online #5356 will likely address this
If offline then the personality may well be failing to validate tag entries against actual inventory on reload. @cgtexmex @jzt ?

I would expect this to potentially occur from any drop in the vSphere event stream and an out-of-band deletion, even without the rollback. The testing added for #5343 should be a good starting place for asserting correct behaviour here (see tests/test-cases/Group0-Bugs/5343.robot)

@caglar10ur please check if this presents with out-of-band deletion of cVMs when doing the connection drop/session deletion followup.

[mdubya66]

marking high for now until we get a triage.

[cgtexmex]

@hickeng containerVMs are loaded from vSphere on a VCH power cycle, so I'd suspect it's reverting to the snapshot.

[jzt]

@hickeng am I understanding this correctly? The cache is returned to its previous state after the snapshot revert? If that's the case, there's no other way around this than doing a periodic refresh, or perhaps forcing a refresh/sync after a write.

[cgtexmex]

@jzt if it's a snapshot...I'd assume so...it's like pausing and resuming a vm -- vm state is saved in the snapshot.

[emlin]

@hickeng right, vic-machine rollback does not enforce a power cycle, that should be the reason to have this dated cache.
So you're right, #5356 should address this issue. Or we could easily add the power cycle into rollback as well.

[emlin]

@jzt @cgtexmex you're right. The cache and process status are all cached in the snapshot, and after vm is resumed, the old data in memory is used.

[stuclem]

An attempt at a release note:

---

• Rolling back upgrade on a VCH reverts container VMs to their pre-upgrade state. #5754
  If you upgrade a VCH, perform some operations, for example vic-machine delete, on container VMs, then use vic-machine upgrade --rollback to revert the VCH to the previous version, the container VMs revert to their previous state. For example, deleted container VMs reappear after the rollback.

---

@emlin @cgtexmex @jzt is this accurate? Thanks!

[emlin]

@stuclem The problem is that after VCH is reverted, the cache information in VCH is reverted, but the deleted container VM cannot be reverted, so if user run docker ps, it will continue to show the deleted container VM, but that's incorrect.
The workaround of this problem is to restart VCH VM from VC UI.

[stuclem]

Thanks @emlin. Updated to:

---

• Container VMs deleted between upgrading and rolling back an upgrade on a VCH show up in docker ps . #5754
  If you upgrade a VCH, perform vic-machine delete on container VMs, then use vic-machine upgrade --rollback to revert the VCH to the previous version, the cache information in the VCH is reverted. However, the deleted container VM cannot be reverted, so if you run docker ps, the deleted container VMs are listed.

  Workaround: Restart the VCH endpoint VM in the vSphere Client.

---

[stuclem]

Thanks @jzt

[hickeng]

@zjs this will be dealt with by simple removal of the memory snapshot as part of the API work (hopefully). Noting here to put it on your radar as a test variant.