Add Substring_by_char

[HaoYang670]

Which issue does this PR close?

Closes #1768.

Rationale for this change

Support substring by char

What changes are included in this PR?

1. new pub function substring_by_char
2. tests
3. benchmark
4. update docs

Performance

substring utf8 (start = 0, length = None)                                                                            
                        time:   [20.035 ms 20.056 ms 20.081 ms]
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) high mild
  7 (7.00%) high severe

substring utf8 (start = 1, length = str_len - 1)                                                                            
                        time:   [21.334 ms 21.358 ms 21.388 ms]
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe

substring utf8 by char  time:   [117.18 ms 117.29 ms 117.41 ms]                                   
Found 10 outliers among 100 measurements (10.00%)
  4 (4.00%) high mild
  6 (6.00%) high severe

1/6 of the speed of substring(by byte)

Are there any user-facing changes?

Yes.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.42%. Comparing base (1a64677) to head (fcc0ee3).
Report is 2384 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1784      +/-   ##
==========================================
+ Coverage   83.39%   83.42%   +0.02%     
==========================================
  Files         198      198              
  Lines       56142    56303     +161     
==========================================
+ Hits        46822    46969     +147     
- Misses       9320     9334      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tustvold]

I think this looks good, left some comments on how it could be improved - but happy for this to be a follow up PR

[tustvold]

Allocating a string temporary, only to copy out of it, is likely a significant portion of the slow-down. That combined with the null handling.

This could definitely be handled as a separate PR, but you might want to consider doing something like (not tested).

let nulls = // align bitmap to 0 offset, copying if already aligned
let mut vals = BufferBuilder::<u8>::new(array.value_data().len());
let mut indexes = BufferBuilder::<OffsetSize>::new(array.len() + 1);
indexes.append(0);

for val in array.iter() {
  let char_count = val.chars().count();
  let start = if start >= 0 {
      start.to_usize().unwrap().min(char_count)
  } else {
      char_count - (-start).to_usize().unwrap().min(char_count)
  };
  let length = length.map_or(char_count - start, |length| {
      length.to_usize().unwrap().min(char_count - start)
  });

  let mut start_byte = 0;
  let mut end_byte = val.len();
  for ((idx, (byte_idx, _)) in val.char_indices().enumerate() {
    if idx == start {
      start_byte = byte_idx;
    } else if idx == start + length {
      end_byte = byte_idx;
      break
    }
  }
  // Could even be unchecked
  vals.append_slice(&val[start_byte..end_byte]);
  indexes.append(vals.len() as _);
}

let data = ArrayDataBuilder::new(array.data_type()).len(array.len()).add_buffer(vals.finish())
  .add_buffer(indexes.finish()).add_buffer(vals.finish());

Ok(GenericStringArray::<OffsetSize>::from(unsafe {data.build_unchecked()}))

[HaoYang670]

Tracked by #1800

[tustvold]

Perhaps we could extract this as a function instead of duplicating it?

[HaoYang670]

Tracked by #1801

[tustvold]

I think it would be easier to follow to just construct the regular array, and then call array.slice(1, 2)

[HaoYang670]

Tracked by #1801

[tustvold]

Loving the test coverage 👍