Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OPC Publisher cyclic read best practices / out of memory issues #2160

Closed
b-enigma-con opened this issue Jan 24, 2024 · 3 comments
Closed

OPC Publisher cyclic read best practices / out of memory issues #2160

b-enigma-con opened this issue Jan 24, 2024 · 3 comments
Labels
bug Something isn't working
Milestone
2.9.4

Comments

@b-enigma-con
Copy link

b-enigma-con commented Jan 24, 2024 •
edited
Loading

We are experiencing memory issues in the way we apply the cyclic read of the OPC Publisher. We are running version 2.9.2 on an IoT Edge module with 16GB RAM, 4 vCPU's. The (dedicated) OPC Server has 16GB of RAM as well. Both VM's are not even close to using 100% RAM. We are doing cyclic reads only for 1000 nodes, every 5 seconds. Our publishednodes.json looks like this (with 1000 nodes):

[
{
"EndpointUrl": "opc.tcp://xxxx",
"UseSecurity": true,
"OpcNodes": [
{
"Id": "NODE ID 1",
"UseCyclicRead": true,
"OpcSamplingInterval": 5000
},
{
"Id": "NODE ID 2",
"UseCyclicRead": true,
"OpcSamplingInterval": 5000
},
{
"Id": "NODE ID 3",
"UseCyclicRead": true,
"OpcSamplingInterval": 5000
},
....................
}

After a while, we're seeing the following logs from our iot edge module:

[14:02:18.0101] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
Publish #122, Reconnecting=False, Error: BadOutOfMemory
[14:02:18.0103] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
Publish #121, Reconnecting=False, Error: BadOutOfMemory
[14:02:18.1131] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
PUBLISH #121 - Unhandled error BadOutOfMemory during Publish.
Opc.Ua.ServiceResultException: BadOutOfMemory
at Opc.Ua.ClientBase.ValidateResponse(ResponseHeader header)
at Opc.Ua.SessionClient.EndPublish(IAsyncResult result, UInt32& subscriptionId, UInt32Collection& availableSequenceNumbers, Boolean& moreNotifications, NotificationMessage& notificationMessage, StatusCodeCollection& results, DiagnosticInfoCollection& diagnosticInfos)
at Opc.Ua.Client.Session.OnPublishComplete(IAsyncResult result)
[14:02:18.1134] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
PUBLISH #122 - Unhandled error BadOutOfMemory during Publish.
Opc.Ua.ServiceResultException: BadOutOfMemory
at Opc.Ua.ClientBase.ValidateResponse(ResponseHeader header)
at Opc.Ua.SessionClient.EndPublish(IAsyncResult result, UInt32& subscriptionId, UInt32Collection& availableSequenceNumbers, Boolean& moreNotifications, NotificationMessage& notificationMessage, StatusCodeCollection& results, DiagnosticInfoCollection& diagnosticInfos)
at Opc.Ua.Client.Session.OnPublishComplete(IAsyncResult result)
[14:02:29.8912] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
Publish #125, Reconnecting=False, Error: BadOutOfMemory
[14:02:29.8915] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
Publish #126, Reconnecting=False, Error: BadOutOfMemory
[14:02:29.9914] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
PUBLISH #125 - Unhandled error BadOutOfMemory during Publish.
Opc.Ua.ServiceResultException: BadOutOfMemory
at Opc.Ua.ClientBase.ValidateResponse(ResponseHeader header)
at Opc.Ua.SessionClient.EndPublish(IAsyncResult result, UInt32& subscriptionId, UInt32Collection& availableSequenceNumbers, Boolean& moreNotifications, NotificationMessage& notificationMessage, StatusCodeCollection& results, DiagnosticInfoCollection& diagnosticInfos)
at Opc.Ua.Client.Session.OnPublishComplete(IAsyncResult result)
[14:02:29.9917] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
PUBLISH #126 - Unhandled error BadOutOfMemory during Publish.
Opc.Ua.ServiceResultException: BadOutOfMemory
at Opc.Ua.ClientBase.ValidateResponse(ResponseHeader header)
at Opc.Ua.SessionClient.EndPublish(IAsyncResult result, UInt32& subscriptionId, UInt32Collection& availableSequenceNumbers, Boolean& moreNotifications, NotificationMessage& notificationMessage, StatusCodeCollection& results, DiagnosticInfoCollection& diagnosticInfos)
at Opc.Ua.Client.Session.OnPublishComplete(IAsyncResult result)
[14:02:29.9925] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
Publish #127, Reconnecting=False, Error: BadOutOfMemory
[14:02:29.9935] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
Publish #128, Reconnecting=False, Error: BadOutOfMemory
[14:02:30.1665] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
PUBLISH #127 - Unhandled error BadOutOfMemory during Publish.
Opc.Ua.ServiceResultException: BadOutOfMemory
at Opc.Ua.ClientBase.ValidateResponse(ResponseHeader header)
at Opc.Ua.SessionClient.EndPublish(IAsyncResult result, UInt32& subscriptionId, UInt32Collection& availableSequenceNumbers, Boolean& moreNotifications, NotificationMessage& notificationMessage, StatusCodeCollection& results, DiagnosticInfoCollection& diagnosticInfos)
at Opc.Ua.Client.Session.OnPublishComplete(IAsyncResult result)
[14:02:30.1697] fail: Azure.IIoT.OpcUa.Publisher.Stack.Services.OpcUaStack[0]
PUBLISH #128 - Unhandled error BadOutOfMemory during Publish.
Opc.Ua.ServiceResultException: BadOutOfMemory
at Opc.Ua.ClientBase.ValidateResponse(ResponseHeader header)
at Opc.Ua.SessionClient.EndPublish(IAsyncResult result, UInt32& subscriptionId, UInt32Collection& availableSequenceNumbers, Boolean& moreNotifications, NotificationMessage& notificationMessage, StatusCodeCollection& results, DiagnosticInfoCollection& diagnosticInfos)
at Opc.Ua.Client.Session.OnPublishComplete(IAsyncResult result)

  • Is this a result of the OPC UA server running out of memory or is this the IoT Edge device?
  • Is a cyclic read for 1000 nodes for every 5 seconds doable? Especially because the read is batched, I suppose this should be no issue?

Another strange thing is that when I configure 1000 nodes with only cyclic reading, the log says:
Monitored Opc nodes succeeded count: 7 <- I expect this to say 1000?
Monitored Opc nodes failed count : 0
@marcschier marcschier added the bug Something isn't working label Jan 25, 2024
@marcschier
Copy link
Collaborator

This appears to come from the server. What server is used? To double check that it is not related to a reconnect issue in 2.9.2, could you try using 2.9.4-preview3 tag of OPC publisher container?

@marcschier marcschier added need more information Needs more information and removed bug Something isn't working labels Jan 26, 2024
@b-enigma-con
Copy link
Author

b-enigma-con commented Jan 29, 2024 •
edited
Loading

Kepware KEPServerEX 6.15 is used, latest version.

I did deploy v2.9.4-preview3 of the OPC publisher (2.9.4.4-preview+52859473eb (.NET 8.0.1/linux-x64/OPC Stack 1.4.372.116)), and I'm not running out of memory anymore. But I notice that not 1000 values are beint sent every 5 seconds (see publishednodes file above), but approx 376. It seems that the amount of nodes that are used for cyclic reading is somehow capped at this amount?

OPC Publisher settings:
BatchSize: 500
BatchTriggerInterval: 1000

DIAGNOSTICS INFORMATION for : <>
OPC Publisher Version (Runtime) : 2.9.4.4-preview+52859473eb (.NET 8.0.1/linux-x64/OPC Stack 1.4.372.116)
Time : 2024-01-29T14:34:55.1899921Z
Ingestion duration : 00:00:00:52 (dd:hh:mm:ss)
Endpoints connected/disconnected : 1 | 0 (Connected)
Connection retries : 0
Monitored Opc nodes succeeded count: 7
Monitored Opc nodes failed count : 0
Subscriptions count : 2
Queued/Minimum request count : 3 | 3
Good/Bad Publish request count : 3 | 0
Ingress value changes : 0
Ingress events : 0
Ingress values/events unassignable : 0
Received Data Change Notifications : 0
Received Event Notifications : 0
Received Keep Alive Notifications : 0
Generated Cyclic read Notifications: 4,192
Generated Heartbeat Notifications : 0
Notification batch buffer size : 6
Encoder input/output buffer size : 0 | 0
Encoder Notif. processed/dropped : 4,186 | 0
Encoder Network Messages produced : 31
Encoder avg Notifications/Message : 135
Encoder worst Message split ratio : 0
Encoder avg Message body size : 62,509
Encoder avg Chunk (4 KB) usage : 15.3
Estimated Chunks (4 KB) per day : 0
Egress Messages queued/dropped : 0 | 0
Egress Messages successfully sent : 31 (0.53/s)

DIAGNOSTICS INFORMATION for : <>
OPC Publisher Version (Runtime) : 2.9.4.4-preview+52859473eb (.NET 8.0.1/linux-x64/OPC Stack 1.4.372.116)
Time : 2024-01-29T14:35:55.1889370Z
Ingestion duration : 00:00:01:52 (dd:hh:mm:ss)
Endpoints connected/disconnected : 1 | 0 (Connected)
Connection retries : 0
Monitored Opc nodes succeeded count: 7
Monitored Opc nodes failed count : 0
Subscriptions count : 2
Queued/Minimum request count : 3 | 3
Good/Bad Publish request count : 3 | 0
Ingress value changes : 0
Ingress events : 0
Ingress values/events unassignable : 0
Received Data Change Notifications : 0
Received Event Notifications : 0
Received Keep Alive Notifications : 0
Generated Cyclic read Notifications: 8,778
Generated Heartbeat Notifications : 0
Notification batch buffer size : 0
Encoder input/output buffer size : 0 | 0
Encoder Notif. processed/dropped : 8,778 | 0
Encoder Network Messages produced : 65
Encoder avg Notifications/Message : 135
Encoder worst Message split ratio : 0
Encoder avg Message body size : 62,543
Encoder avg Chunk (4 KB) usage : 15.3
Estimated Chunks (4 KB) per day : 0
Egress Messages queued/dropped : 0 | 0
Egress Messages successfully sent : 65 (0.55/s)

How can it be that I only seem to be receiving approx 380 datapoints every 5 seconds, while I expect 1000?

*Edit: did some additional testing and registered the findings in #2165.

@marcschier marcschier added this to the 2.9.4 milestone Jan 30, 2024
@marcschier marcschier added bug Something isn't working and removed need more information Needs more information labels Jan 30, 2024
@marcschier marcschier mentioned this issue Feb 1, 2024
Closed
5 tasks
@marcschier
Copy link
Collaborator

@b-enigma-con, I have published 2.9.4-preview4 tag to MCR for testing. Please let me know if this fixes the issues you are seeing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
2.9.4
Development

No branches or pull requests
2 participants
@marcschier @b-enigma-con