fix(core/worker_events): overwrite the post method of worker_events

[liverpool8056]

Summary

There is a limit for the size of payload allowed to be send in events lib. In some cases, Kong needs to send big payload. For example DAO:crud rely on events to invalidate cache. In these cases, the size of the payload may exceeds the limit due a large dao entity.
In this PR, the post method of the worker_events is overwritten to take a retry action when we get the 'payload too big' error message, and the retry action is just to re-post the event with a truncated payload so that the subsequent logic that relies on the event could have a chance to be informed rather than unaware at all. Based on this purpose, the retry action is taken and only taken once when the error happens.

The limit for the size of payload is defined inside the overwritten post method as a conventional value.

Checklist

• The Pull Request has tests
• There's an entry in the CHANGELOG
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Full changelog

• [Implement ...]

Issue reference

FTI-4963
KAG-1523

[liverpool8056]

This one is waiting for a co-related PR Kong/lua-resty-events#28 to be merged and bumped

[chronolaw]

lua-resty-events has bumped to 0.1.5. See: #10883

[chronolaw]

Should we send this too-big truncated payload?
Since it is truncated, could it be string.sub(data, 1, 1024) or something else?

[liverpool8056]

It's a valuable thinking for the length. In my feel, any length to be truncated is not convincing enough except that we try our best and keep the content as enough as we can or we can just tell the fact that the payload is truncated without any information about the original payload. And I think try our best would be better.
So that's my opinion, WDYT?

[chronolaw]

Could the event receiver do something with this truncated data? if not, I think it may not make sense.

[liverpool8056]

I think whether the receiver could do something or not, it depends on the receiver. The sender just try his best, and the senders don't know the ability and behavior of the receiver.
And since we have such a limit, in my feel like it should be always ok to send a payload like that size anytime.

[bungle]

In a real world, where do we have this issue happening?

The description says:

DAO:crud rely on events to invalidate cache. In these cases, the size of the payload may exceeds the limit due a large dao entity.

But where exactly, and why DAO crud event can be so big? Should we make it a bit more intelligent? E.g. truncate the fields or something? Or fix it at the source of event?

[liverpool8056]

In a real world, where do we have this issue happening?

For some plugin objects whose fields are not size-limited may be too large. As FTI-4963 suggested, the config field of request-validator plugin is too large for worker_event lib to send events. So large events are basically caused by size-unlimited fields.

Should we make it a bit more intelligent? E.g. truncate the fields or something?

I'm afraid it would be a bit flaky to try to truncate large fields of an entity, fields to be truncated might vary with the specific entities.

Or fix it at the source of event?

Fixing it at the source of event means we have to limit the size of fields that are unlimited initially, this would introduce a breaking change which might not such necessary at this moment.

[liverpool8056]

This PR could be closed as we had another approach for this, referring to Kong/lua-resty-events#37