Entity checking algorithm isn't 100% accurate.

[StefenSharkey]

The entity checking algorithm occasionally grabs the wrong entity if another is too close to the player. Sometimes it will grab an entity behind the player, sometimes it'll grab an entity to the side, etc.

Another issue is that if there's a smaller entity (I.E. squid) in front of a larger entity (I.E. skeleton), the algorithm will return neither and will cause the preceding command to fail.

The algorithm is as follows.

public static Entity getEntityInCrosshairs(Player player) {
    int range = 10;
    Block[] blocks = player.getLineOfSight(null, range).toArray(new Block[0]);
    List<Entity> near = player.getNearbyEntities(range, range, range);

    for (Block block : blocks) {
      for (Entity entity : near) {
        if (entity.getLocation().distance(block.getLocation()) < 2) {
          return entity;
        }
      }
    }

    return null;
  }

It can also be found here.