fread parse error

[pradkrish]

Closes #2680

At src/core/read/fread/fread_thread_context.cc:210, If j<ncols and if *tch reaches the end of line with the current quote rule, the while loop exits ending up in IOError.

With the code added in this PR, one way to fix this is to decrement tch(tch--) to the previous sep and try parsing again with the next quote rule. If the number of columns in the current row is only one, decrementing tch will never hit the sep in the current row and end up running into the previous row, hence the if logic inside the while loop to prevent this.

[pradkrish]

@st-pasha I am unable to see the log of the failing builds, will you be able give a snippet of one of them? Thanks.

[st-pasha]

Both test suits segfaulted when reading the file https://h2o-public-test-data.s3.amazonaws.com/fread/bad.txt
Unfortunately, there are no core files, so we'll have to reproduce it locally.

[st-pasha]

The test itself is tests/fread/test-fread-large.py::test_fread_all[/tmp/pydatatable_large_data/h2o-3/fread/bad.txt]. Notably, it's the first test in the "test_fread_all" sequence, so the problem may not be with the file per se (in the sense that any other file would have seg.faulted there too), just that "bad.txt" happened to be the first on the list.

[pradkrish]

I checked out this branch and ran it on my wife's macOS.

Versions
Python (venv): 3.7.9
Clang: 12.0.0

All tests passed and reading the file bad.txt gave a 408x3 table.

   | ####################  ##############                       ###############################################
--- + --------------------  -----------------------------------  -----------------------------------------------
  0 | #############         ##############                                                                     0
  1 | #############         ###########                                                                        0
  2 | #############         ##############                                                                     0
  3 | #############         #############                                                                      0
  4 | #############         #########                                                                          0
  5 | #############         ################                                                                   0
  6 | #############         ###############                                                                    0
  7 | #############         #################################                                                  0
  8 | #############         ########################                                                           0
  9 | #############         ####################                                                               0
 10 | #############         #############                                                                      0
 11 | #############         ############                                                                       0
 12 | #############         ###########                                                                        0
 13 | #############         ########                                                                           0
 14 | #############         ###                                                                                0
  … | …                     …                                                                                  …
403 | ###########           ##############                                                                     0
404 | ###########           #################                                                                  0
405 | ###########           ###############                                                                    0
406 | ###########           ###################################                                                0
407 | #########             ###############                                                                    0

any idea how to narrow down the bug? 🤔

[pradkrish]

@oleksiyskononenko You have told me before that you have a mac. My test suite is unfortunately failing on macOS in the cloud and I have no access to the logs either. As described above, I tested it locally on my wife's macbook and it passes. Whenever you have time, will you be able to build this branch and confirm whether the tests pass on your mac? I am kinda clueless how to proceed, any suggestions are useful. Thanks.

[oleksiyskononenko]

@pradkrish I just checked it on my mac and it didn't segfault on bad.txt for python 3.6.8. I've also re-triggered the build for this PR to see if this issue is reproducible.

[pradkrish]

@pradkrish I just checked it on my mac and it didn't segfault on bad.txt for python 3.6.8. I've also re-triggered the build for this PR to see if this issue is reproducible.

Thanks @oleksiyskononenko, it seems to pass now. 😃

[Darel13712]

Hey. It's been a year. Any plans to merge this? @st-pasha

[st-pasha]

@Darel13712 Need to fix the tests still

[Darel13712]

@st-pasha seems all checks pass now?