listAllDataObjectsUnderPath returns duplicate files for big folders with replica's

[ramonvansparrentak]

We discovered that listAllDataObjectsUnderPath sometimes returns a "file" twice when there are over 5000 dataobjects in the path.

The cause seems to be in listDataObjectsUnderPathViaGenQuery, this method filters out duplicate entries by checking if the previous path equals the current path

if (currentPath.equals(lastPath)) {
  continue;
}

When the method listAllDataObjectsUnderPath calls listDataObjectsUnderPathViaGenQuery multiple times in the while loop, the first result is never filtered. Although it could be the same as the result set of the previous call.

https://github.com/DICE-UNC/jargon/blob/master/jargon-core/src/main/java/org/irods/jargon/core/pub/CollectionListingUtils.java#L834

[ramonvansparrentak]

#438

Note, I haven't tested it yet.

[korydraughn]

Although it could be the same as the result set of the previous call.

I found this statement a little confusing due to the word, result set. Are you talking about a specific row or the entire set of rows?

Are you finding duplicates at page boundaries only or, do you see duplicates in the middle of a page too?

[trel]

i interpret the description and the fix as only affecting boundaries. the fix gives the current call 'visibility' into the last element of a 'previous' call/page - so it can be ignored if it is a duplicate.

[ramonvansparrentak]

The bug and suggested fix are only for the boundaries.

For example, for files a, b, c, d, with 2 replica's each. The first result set could look like ['a', 'a', 'b', 'b', 'c'] and the second set would look like ['c', 'd', 'd']. The current code fixes the duplicates in both sets and thus returns ['a', 'b', 'c'] and ['c', 'd']. But, the total result contains two c, at the boundaries of the result sets.

The suggest fix keeps track of c to filter it in the next set.

Hope that makes it clear.

[korydraughn]

I have not been able to reproduce this issue.

What version of Jargon are you using?
What version of iRODS did you see this happen against?

Do you have Jargon code which captures the issue?

[korydraughn]

It has been pointed out to me that the issue appears when there are multiple replicas.

Will adjust my test and give it another try.

[korydraughn]

I now have a test which reproduces the issue.

All that's left is to confirm PR #438 resolves it.