Inconsistent message rate

[arcarmona]

Hello,

I have currently set up an emqx (community edition) cluster in kubernetes using emqx-operator , I have around 300 clients and an expected message income of ~6k/s (message size ~1Kb). When running I get the expected connections, however when the message rate seems to be setting to its expected number, it drops (emqx dashboards also lags a bit before the drop).

I understand there can be small oscillations in the message income however for my use case such a big drop shouldnt be happening and the message rate per second should stay consistent around 6k. I wanted to ask if I might be missing some configuration parameter that might help solve this problem. Attached below you may find an screenshot of the behavior and my emqx yaml.

Thank you.

apiVersion: apps.emqx.io/v2beta1
kind: EMQX
metadata:
  name: emqx
spec:
  image: emqx:5.4.1
  imagePullPolicy: Always
  updateStrategy:
    evacuationStrategy:
      connEvictRate: 1000
      sessEvictRate: 1000
      waitTakeover: 10
    initialDelaySeconds: 10
    type: Recreate
  config:
    data: |
      listeners.tcp.default {
        proxy_protocol = true,
        max_connections = 10000,
        acceptors = 64,
        messages_rate = "10000/s",
        max_conn_rate = "10000/s"
      }
      listeners.ssl.default {
        enable = false
      }
      mqtt {
        max_topic_levels = 20,
        peer_cert_as_username = "cn",
        shared_subscription_strategy = "random",
        ignore_loop_deliver = true,
        max_inflight = 1000
      }
      node {
        dist_buffer_size = 2097150,
      }
      force_shutdown {
        max_heap_size = "64MB"
      }
      retainer {
        msg_expiry_interval = "10s",
        msg_clear_interval = "10s"
      }
  coreTemplate:
    spec:
      resources:
        limits:
          cpu: "6"
          memory: 6Gi
        requests:
          cpu: "6"
          memory: 6Gi
      ports:
        - name: dashboard
          containerPort: 18083
      replicas: 2
      volumeClaimTemplates:
        storageClassName: standard
        resources:
          requests:
            storage: 10Gi
        accessModes:
        - ReadWriteOnce
  dashboardServiceTemplate:
    spec:
      type: LoadBalancer
  listenersServiceTemplate:
    spec:
      clusterIP: None 

[zhongwencool]

        messages_rate = "10000/s",
        max_conn_rate = "10000/s"

Try to remove those rate limit to see what happen.