Summary

Infrastructure "Refactor/optimize monster pathfinding."

Purpose of change

Monster pathfinding is the single most expensive thing in high traffic areas. While optimizing this in a previous PR, it quickly became clear that the code path was difficult to work with due to pathfinding logic being spread across multiple different parts of the codebase. Bugs in one area would cover for bugs in others, and due to this it wasn't entirely clear what behavior was originally intended.

Describe the solution

Rewrite creature pathfinding to have all pathfinding logic (i.e. can I move here, how much does it cost to move here, find path between x/y, etc) in a single location, such that no logic needs to be duplicated in order to answer questions regarding pathfinding.

This is done by making the pathfinding implementation generic, and having the pathfinding logic take place at the intersection of two different caches: the pathfinding settings, and the map pathfinding cache.

Pathfinding settings is a per-creature structure that encapsulates pathfinding logic specific to that creature, e.g. if it flies, if it can open doors, what fields it is immune to, etc. Caching this logic beforehand allows for fast rejection of bad tiles using a bitfield, which is important when thousands of tiles are visited.

The map pathfinding cache encapsulates pathfinding logic specific to that tile, e.g. if there is an obstacle there, the cost to move on the tile, the existence of traps/fields, etc. This creates a bitfield for the specific tile that allows for fast rejection of a tile when combined with the pathfinding settings.

Note that in the tested location (TCL) the vast majority of paths end up not being found because monsters are stuck behind glass/bars they cannot bash down, or there is a destination in the next room they can hear, but not move to. This is handled by using a bidirectional A* implementation that can fail fast if either point is in a closed in room, and adding exponential backoff to trying to path while stuck.

Finally, as now the pathfinding logic is unified, it means monsters can act much more intelligently. e.g. flying monsters can actually use the ability to fly when finding paths to a destination. To take advantage of this new ability (and to do integration testing), a new flying variant of mi-go was added: the mi-go shrike. It doesn't spawn anywhere yet however, that will be done in a future PR.

Describe alternatives you've considered

As the vast majority of the CPU usage left is due to paths that cannot be found, it would make sense to preprocess the map into a set of islands so these can be quickly rejected. The problem is that it isn't possible in general to do this as each monster type has different bashing thresholds, door avoidance, field avoidance, etc that could result in the creation of an island for one type of monster but not another. This would require per-monster(type) island generation that could quickly take up a lot of memory and processing power. I think it is worth visiting this as a separate step in a future PR: i.e. detect when a path cannot be found, and detect if the monster is stuck on an island it cannot escape from. This should remove what is left of the pathfinding CPU usage, but would make this PR too complicated.

I also briefly experimented with using a weighted variant of Jump Point Search to further optimize this. While in theory this should result in an order of magnitude speedup, the added complexity wasn't worth it for this PR. It is something to try doing in the future. I would tackle this before trying to tackle islands, as if this really does achieve an order of magnitude speedup it would be more generally useful and less error prone.

In the end though, this was largely solved by just using exponential backoff. Sometimes simple is best.

Caching the pathfinding settings is also an option that wasn't done. I'm not comfortable with properly handling the invalidation of that in the same PR. It would be too prone to breaking in weird ways, and didn't want the added complexity.

Testing

Added extensive unit and integration testing for the new pathfinding logic. The old monster movement behavior wasn't tested at all, so I did my best to look at how monsters already behave with the current pathfinding and try to capture that in the tests.

Additional context

Before:

After: