(shadow-manager): Stops syncing to cloud after restore from connectivity outtage

[crispinboylan-int]

Describe the bug
We get reports that sensors are not being update in the cloud, although greengrass is connected. When we investigate around the time of last update we see some kind of connectivity outtage in the logs, including Following the reconnection the shadow manager service never updates the cloud shadows again.

To Reproduce
Seems intermittent.

Expected behavior
Shadow manager reconnects.

Actual behavior
Shadow manager never tries to sync again.

Environment

• OS: Ubuntu 22.04
• JDK version: 17
• Nucleus version: 2.7.0
• Shadow Manager version: 2.2.4

Additional context
Logs from shadow manager around connectivity outtage:

2023-01-22T01:33:57.558Z [INFO] (Thread-3) com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy: sync. Stop syncing. {}
2023-01-22T01:33:57.558Z [INFO] (Thread-3) com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy: sync. Cancel 1 sync thread(s). {}
2023-01-22T01:33:57.558Z [INFO] (Thread-3) com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy: sync. Waiting for in-flight sync requests to finish. {}
2023-01-22T01:34:02.499Z [INFO] (Thread-3) com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy: sync. Finished waiting for sync requests to finish. {}

on restoration of connection we then see:

2023-01-22T01:38:17.817Z [INFO] (Thread-3) com.aws.greengrass.mqttclient.AwsIotMqttClient: Connection resumed. {clientId=home-XXXX#2, sessionPresent=true}
2023-01-22T01:38:17.823Z [INFO] (Thread-3) com.aws.greengrass.status.FleetStatusService: fss-status-update-published. Status update published to FSS. {trigger=RECONNECT, serviceName=FleetStatusService, currentState=RUNNING}
2023-01-22T01:38:43.084Z [INFO] (Thread-3) com.aws.greengrass.mqttclient.AwsIotMqttClient: Connection resumed. {clientId=home-XXXX, sessionPresent=true}
2023-01-22T01:38:43.106Z [INFO] (Thread-3) com.aws.greengrass.deployment.IotJobsHelper: No deployment job found. {ThingName=home-XXXX}

but from that point on we only ever see:

2023-01-22T01:38:59.940Z [INFO] (Thread-6) com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler: Successfully updated shadow. {service-name=co.mgh.ShadowPortal, thing name=home-XXX, shadow name=AirQuality, local-version=854642}
2023-01-22T01:39:12.905Z [INFO] (Thread-6) com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler: Successfully updated shadow. {service-name=co.mgh.ShadowPortal, thing name=home-XXX, shadow name=AirQuality, local-version=854643}

from shadow manager, we never see another sync in the logs with the cloud shadows.

log-events-viewer-result(2).csv

[crispinboylan-int]

can anyone comment on this? Just wondering if it could be something resolved in a later nucleus? I cant see anything relevant in the notes though.

[nikkhilmuthye]

Can you retry using Nucleus 2.9.3? Also, has this happened more than once?

[crispinboylan-int]

yes this has happened a couple of times before and has now just occurred on a system running 2.9.3

[jcosentino11]

Looking at the logs, I'm not seeing evidence that syncing starts back up. We're going to reproduce the issue and investigate on our side, we'll update this thread as we know more.

Please do try the very latest versions of Nucleus and Shadow manager as we've continue to make some improvements which may address you issue.

[MikeDombo]

I will close this issue soon if we don't hear anything. Is this still a problem when using the very latest versions of Nucleus and Shadow Manager?