ConditionHandling stops working after some time

[Haeavar]

Describe the bug
We configure the ConditionHandling using following options.

        "ConditionHandling": {
          "updateInterval": 5,
          "snapshotInterval": 60
        }

This works after a fresh start of the OpcPublisher.
After some hours it stop sending ua-condtion messages even if there are retained events.

To Reproduce
Currently its not clear what causes the stop but it happens always after some hours.

Additional context
It seems, that the OpcPublisher 2.8.1 still worked without issues. At least with version 2.9.8 we see these issues. But it could be introduced in earlier versions.

[Haeavar]

Is it possible to use the DefaultMonitoredItemWatchdogCondition to monitor the ua-condition?
It would help, if there are no ua-condition since 1h, the publisher restarts. This would help us until the bug is fixed.

[marcschier]

Question: is the stop related to a reconnect? If you look through the logs to the point it stops, a reconnect should be apparent.

[Haeavar]

@marcschier you are right. I see a reconnect in the log:

2024-07-13T03:10:12.310Z [24-07-13 03:10:12.3101] info: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaSubscription[0] 2024-07-13T03:10:12.310Z Subscription a3cb91608fcab78fb8f1d76037c568ba155dea1d_0:727814108 STOPPED! 2024-07-13T03:10:14.309Z [24-07-13 03:10:14.3094] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0] 2024-07-13T03:10:14.309Z Publish #28290, Reconnecting=False, Error: BadSecureChannelClosed

Since it is kind of a critical bug for us do you have an idea for a workaround until its fixed?

[Haeavar]

After some investigation it turns out, that this happens only for subscriptions to an aggregation OPC UA server. The events comes from an aggregated OPC UA server. If the aggregation server is disconnected (shutdown, restart or connection interrupt) and the OpcPublisher reconnected, all subscriptions are reestablished but the conditions are missing.
Only after removing the writer and recreate them, the conditions are send again.

[marcschier]

Looks like the issue is due to the condition timer staying disabled after reconnect, and the item in the subscription is in a state where it can not be reenabled again on the server, therefore the condition timer staying disabled, and even when the subscription is periodically resynchronized with the server (which should happen if there is one item not fully applied to the subscription). Right now I have no ready idea/fix here, nor good idea how to reproduce. The only thing that comes to mind is to recognize this in the watchdog condition and trigger the watchdog, a feature I would need to implement.

When you go through the logs, and you find where the items are "added", can you pinpoint any errors that apply to this condition item?

I would be happy to go through the logs, too, if you care to share them. You can share them through a gist you give me access to, or through MSFT support, or add a subset to the issue here.

[Haeavar]

You were right. There is indeed an bad monitored node:

2024-07-16T07:38:01.407Z [24-07-16 07:38:01.4071] info: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaSubscription[0]
2024-07-16T07:38:01.407Z 1e7f323b0fe01c49335c92d90fe6a8af1e4eb7ed_0:727814210 - Now monitoring 1 nodes:
2024-07-16T07:38:01.407Z # Good/Bad: 0/1
2024-07-16T07:38:01.407Z # Reporting: 1
2024-07-16T07:38:01.407Z # Sampling: 0
2024-07-16T07:38:01.407Z # Disabled: 0
2024-07-16T07:38:01.407Z # Not applied: 0
2024-07-16T07:38:01.407Z # Removed: 2
2024-07-16T07:38:01.407Z [24-07-16 07:38:01.4071] info: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaSubscription[0]
2024-07-16T07:38:01.407Z Issuing ConditionRefresh on subscription 1e7f323b0fe01c49335c92d90fe6a8af1e4eb7ed_0:727814210

This is the part of the log where the connection is lost and reconnected.
Logs_sfh-229_app-62_opcpublisher (4).txt