[Azure] [OpenAI] Malformed FunctionAppLogs documents found in azure_openai.logs dataset

[lucian-ioan]

In my analysis of Azure OpenAI logs, 21/91 documents contain malformed JSONs, breaking the processing of the fields and displaying the same error.message:

The documents appear unrelated to Azure OpenAI (from FunctionAppLogs instead):

The Azure OpenAI metrics contain 1153 documents, all without the error.message field populated. This is likely because this issue is eventhub-specific and metrics use Azure Monitor.

Elasticsearch Query

GET ${exampleVariable1} // _search
{
  "size": 0,
  "query": {
    "bool": {
      "should": [
        {
          "term": {
            "data_stream.dataset": "azure.open_ai"
          }
        },
        {
          "term": {
            "data_stream.dataset": "azure_openai.logs"
          }
        }
      ]
    }
  },
  "aggs": {
    "datasets": {
      "terms": {
        "field": "data_stream.dataset",
        "size": 2
      },
      "aggs": {
        "error_message_exists": {
          "filter": {
            "exists": {
              "field": "error.message"
            }
          }
        },
        "error_message_not_exists": {
          "filter": {
            "bool": {
              "must_not": {
                "exists": {
                  "field": "error.message"
                }
              }
            }
          }
        }
      }
    }
  }
}

Possible fix

This issue can likely be fixed by enabling sanitization in the logs data stream.

Right now it's missing from the manifest.yml file. I will test it and open a PR to add and enable it by default.

Question

Why do the malformed JSONs that break the processing default to data_stream.dataset: azure_openai.logs in this case?

[muthu-mps]

@lucian-ioan - This logs are from the Azure function app. These logs are also getting pushed to the event-hub earlier for testing. I don't think we need to sanitise these unstructured logs which is not relevant to OpenAI service.

[lucian-ioan]

@muthu-mps the issue is the event.dataset and data_stream.type are the exact same as for normal OpenAI logs in these Function App Logs:

[muthu-mps]

@muthu-mps the issue is the event.dataset and data_stream.type are the exact same as for normal OpenAI logs in these Function App Logs:

Seems like these logs are not getting dropped in the pipeline because of the errors. We can enable sanitisation and verify there is no malformed error logs.