-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Poor performance when validating foreign key relationship on large dataset #805
Comments
@Stephen-Gates again would you mind please adding priority and milestone. This is another high one for me. Would you also be able to test on your machine to see if you are getting the same result? |
Here is another csv file that is causing this feature to stall Data Curator: https://data.qld.gov.au/dataset/crash-data-from-queensland-roads/resource/177dc50c-0cf7-46ba-8a69-99695aeaa46a I created foreign keys on gender and severity columns, referencing new reference tables I created using a subset of unique values from these columns. |
Repeated behaviour with tide data :-( |
Explore upgrading to the latest version of HandsOnTable |
There is some delay due to number of rows to iterate and errors to send back, but we can use a loader here. However the frictionless iterations and Data Curator callbacks are all working fine. The stalling is due to the handsontable comments plugin not handling the large amount of feedback. |
Thanks for the update @mattRedBox I remember you said upgrading handsontable was significant #678 |
Have upgraded to 3.0.0 and tests are running correctly.
Unfortunately, from what I can see in the hot (Handsontable) library, even the latest version does not allow more than a single cell to register a hot comment (which we use to put those little marks in each cell to show validation errors) - which is a pity as they have changed the 'selection' code to enable multiple cell selections and reselections. With all the other parts in place (streaming errors as they are received to cell so user has immediate feedback - just need to add a loading blockout - should be straightforward), I'll look at pinching the css from handsontable and implementing ourselves so that we can do 'batch' updates of cells, rather than 1 at a time - if successful, this should provide significant improvement in the wait times. |
Ok think I found a compromise within handsontable, similar to what I did in Find/Replace, where I can update array of cell objects, this time adding cell property change - then just need to render the entire hot, bypassing the specific setComment plugin call for an individual cell altogether. |
I repeated the first example in this issue. The validation completed and speed was acceptable. Although the errors show in all cells using the triangle, some data doesn't show until you start scrolling. The data being displayed in the cells is not correct. Only the first 1044 rows are for ablepoint. |
Hi @Stephen-Gates |
Current Behaviour (for problems)
Expected Behaviour
Steps to Reproduce
Your Environment
Data Curator version: 0.17.0
Operating System and version: Windows 10 Pro
AND
Data Curator version: 0.17.0
MacOS Sierra 10.12.6
The text was updated successfully, but these errors were encountered: