Estimate and validate regular expression complexities

[anthony-chang]

Addresses #4061
Relevant cuDF PR: rapidsai/cudf#10808

This attempts to do a worst-case estimation for the memory footprint for a regular expression operation. We can evaluate regular expressions by converting them into a deterministic finite automaton (DFA) and counting the number states in this state machine. Since each state is a possibility in the evaluation, we need to store at worst all these states.

This estimation is done in the planning stage on the driver node, so we only have access to the regex pattern and not the column data so we can only do a rough estimation for the row count using the target batch size and the default string type size.

[andygrove]

This is looking good, but I would like to see some tests demonstrating the costs of different regular expressions.

[NVnavkumar]

Can we also add integration tests to simulate the fallback condition?

[anthony-chang]

This is looking good, but I would like to see some tests demonstrating the costs of different regular expressions.

@NVnavkumar @andygrove I added some python tests for both the fall back and no fallback scenarios but they rely on knowing the default string size in order to set the target batch size so that the row count is 1. I'm not sure if this is a good approach to testing it but I can't think of another way without exposing the countStates implementation.

Do you think these tests are sufficient?

[NVnavkumar]

LGTM. I think the basic mechanics are at least tested here. The default value for this will be very high at first anyways, so it should not affect most users, as it is just a fallback mechanism in case something cannot be handled on the GPU.

[anthony-chang]

build

[andygrove]

LGTM. Thanks @anthony-chang

[anthony-chang]

build