Bug: On restart, agbot doesn't update a node even though a new service was added and a deployment policy update occurred

[dlarson04]

Describe the bug.

On a recent agbot restart, the agbot received a policy change event

I0927 10:23:09.364242       8 worker.go:311] CommandDispatcher: AgBot received command (*agreementbot.PolicyChangedCommand): {event: CHANGED_POLICY, file: , name: ISV_qatoolbox-pablo_andres_fuero/com-ibm-mesh-agent, org: ISV_qatoolbox-pablo_andres_fuero, policy: {"header":{"name":"ISV_qatoolbox-pablo_andres_fuero/com-ibm-mesh-agent","version":"2.0"},"agreementProtocols":[{"name":"Basic","protocolVersion":1}],"workloads":[{"priority":{"priority_value":1,"retries":3,"retry_durations":60},"workloadUrl":"com.ibm.mesh.skupper.agent","organization":"IBM","version":"4.0.0","arch":"*"},{"priority":{"priority_value":2,"retries":3,"retry_durations":60},"workloadUrl":"com.ibm.mesh.skupper.agent","organization":"IBM","version":"3.0.0","arch":"*"}],"valueExchange":{},"dataVerification":{"metering":{}},"proposalRejection":{},"constraints":["com.ibm.mesh.skupper-site == true"],"nodeHealth":{},"userInput":[{"serviceOrgid":"IBM","serviceUrl":"com.ibm.mesh.skupper.agent","serviceVersionRange":"[0.0.0,INFINITY]","inputs":[]}],"secretBinding":[{"serviceOrgid":"IBM","serviceUrl":"com.ibm.mesh.skupper.agent","serviceArch":"*","serviceVersionRange":"[0.0.0,INFINITY]","enableNodeLevelSecrets":true,"secrets":[{"rabbitmq":"rabbitmq"},{"observability":"observability"}]}]}}

But it wasn't until later that the agbot took over a partition and found agreements

I0927 10:24:35.710803       8 partition.go:126] AgreementBot {794fc7b8-7095-44ab-bdd6-6f4c5ad17eb9 true} claimed partition 149
I0927 10:24:35.838316       8 partition.go:281] AgreementBot 794fc7b8-7095-44ab-bdd6-6f4c5ad17eb9 moved agreements, workload usage and secrets from partition 149 to 150
I0927 10:24:35.838375       8 agreementbot.go:1040] AgreementBotWorker beginning sync up.
I0927 10:24:35.888812       8 agreementbot.go:1119] AgreementBotWorker added agreement 007ccae4f6e201ac413acb66b1d81b54091403e8629581bbc44fd6b31a62344c to policy agreement counter.
I0927 10:24:35.898275       8 agreementbot.go:1119] AgreementBotWorker added agreement c11501a4beaabfb85f46cb60510c00b6cef5553d74a3d3751b33d4875740eae2 to policy agreement counter.
I0927 10:24:35.906608       8 agreementbot.go:1119] AgreementBotWorker added agreement 728ce9f525a20cd001e7bf19ecf2b5c65f35d6a047f70d75f29314ff1b82a25f to policy agreement counter.
I0927 10:24:35.915272       8 agreementbot.go:1119] AgreementBotWorker added agreement 4b055fa9f0113e96ea045cf1b3adf9e556e40b91b495dd240da5226bb2db3e8d to policy agreement counter.
I0927 10:24:35.923540       8 agreementbot.go:1119] AgreementBotWorker added agreement 24630e0dd48e253635047213873f6993e859a0e8cb9177bfc0c2993c2bd06c06 to policy agreement counter.
I0927 10:24:35.931729       8 agreementbot.go:1119] AgreementBotWorker added agreement 5e17f5f7b4e9593371d306261b8c40aa0abc4451eeae626586351e698878ccfb to policy agreement counter.
I0927 10:24:35.939318       8 agreementbot.go:1119] AgreementBotWorker added agreement bb4e507ae0eefd47afe8269def0ab4d1fd8d46d0bfbe67f0a599cc7f68913c47 to policy agreement counter.
I0927 10:24:35.939362       8 agreementbot.go:1136] AgreementBotWorker discovered BC instances in DB map[:map[:map[:true]]]

but it did not update those agreements to the new service level

Describe the steps to reproduce the behavior.

No response

Expected behavior.

Agbot should have processed the policy change event and updated the agents that had agreements for the old service

Screenshots.

No response

Operating Environment

Linux

Additional Information

No response