fix analytics with new format

[AntoineJac]

Summary

This PR is a fixed of #12583
We need to refactor as many observability providers do not simplify ingest array. Here is a new nested format:

Current model is:

  "ai": {
    "mistral": {
      "request_total_tokens": 648,
      "number_of_instances": 1,
      "instances": [
        {
          "meta": {
            "request_model": "mistral-tiny",
            "plugin_id": "1c783122-0223-4a03-9f1d-c71d2baab01e",
            "response_model": "mistral-tiny",
            "provider_name": "mistral"
          },
          "usage": {
            "prompt_token": 392,
            "total_tokens": 648,
            "completion_token": 256
          }
        }
      ],
      "request_prompt_tokens": 392,
      "request_completion_tokens": 256
    },
    "azure": {
      "request_total_tokens": 145,
      "number_of_instances": 1,
      "instances": [
        {
          "meta": {
            "request_model": "gpt-35-turbo",
            "plugin_id": "5df193be-47a3-4f1b-8c37-37e31af0568b",
            "response_model": "gpt-35-turbo",
            "provider_name": "azure"
          },
          "usage": {
            "prompt_token": 89,
            "total_tokens": 145,
            "completion_token": 56
          }
        }
      ],
      "request_prompt_tokens": 89,
      "request_completion_tokens": 56
    },
    "cohere": {
      "request_total_tokens": 149,
      "number_of_instances": 1,
      "instances": [
        {
          "meta": {
            "request_model": "command",
            "plugin_id": "30a63ab9-03c6-447a-9b25-195eb0acaeb2",
            "response_model": "command",
            "provider_name": "cohere"
          },
          "usage": {
            "prompt_token": 78,
            "total_tokens": 149,
            "completion_token": 71
          }
        }
      ],
      "request_prompt_tokens": 78,
      "request_completion_tokens": 71
    }
  }

New model if using several plugins:

  "ai": {
    "ai-request-transformer": {
      "meta": {
        "request_model": "gpt-35-turbo",
        "provider_name": "azure",
        "response_model": "gpt-35-turbo",
        "plugin_id": "5df193be-47a3-4f1b-8c37-37e31af0568b"
      },
      "payload": {},
      "usage": {
        "prompt_token": 89,
        "completion_token": 56,
        "total_tokens": 145
      }
    },
    "ai-response-transformer": {
      "meta": {
        "request_model": "mistral-tiny",
        "provider_name": "mistral",
        "response_model": "mistral-tiny",
        "plugin_id": "1c783122-0223-4a03-9f1d-c71d2baab01e"
      },
      "payload": {},
      "usage": {
        "prompt_token": 168,
        "completion_token": 180,
        "total_tokens": 348
      }
    },
    "ai-proxy": {
      "meta": {
        "request_model": "gpt-35-turbo",
        "provider_name": "azure",
        "response_model": "gpt-35-turbo",
        "plugin_id": "546c3856-24b3-469a-bd6c-f6083babd2cd"
      },
      "payload": {},
      "usage": {
        "prompt_token": 28,
        "completion_token": 14,
        "total_tokens": 42
      }
    }
  },

Checklist

• The Pull Request has tests
• A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

Fix #[issue number]

KAG-3759
KAG-4124

EDIT: format JSON snippets

[gruceo]

looks good, can you change the commit description according to the convention?

[AntoineJac]

@gruceo can you please merge to Kong and Kong-ee if it looks ok? Thanks

[tysoekong]

I can handle the Kong-EE cherry-picking and stuff, I have been set on to this to get everything done before next week.

Just need this merging to allow proceeding the other remaining PRs.

[tysoekong]

Set rebase when merging master back in; my fix conflicts now...

[tysoekong]

Tests should pass now

[tysoekong]

Wait I have pulled in commits now in this PR

[ttyS0e]

@gruceo ready!

[gruceo]

LGTM, wait for @brentos to approve then we can merge.

[gruceo]

@tysoekong @ttyS0e there are 14 commits in this PR, so we should squash.

[tysoekong]

@brentos Do you approve also?

[brentos]

LGTM!

[tysoekong]

@gruceo Will it let you push squash and merge?

[team-gateway-bot]

Successfully created cherry-pick PR for master:

• https://github.com/kong/kong-ee/pull/8905