Entries disappearing while zooming ScatterChart

[Paladinko]

I'm using a ScatterChart with cca. 10k xVals, and about 200 Entries. When I zoom, all entries on the graph disappear except the ones on the far right. When I drag to the far right, they suddenly appear (regardless of the zoom amount).

[taltstidl]

I can confirm this issue. It seems to happen when there are fewer entries than x-values. (e.g. 500 x-values but only 25 entries)

[taltstidl]

Related issue in ios-charts: ChartsOrg/Charts#130

[cbadhrinath]

I am facing the same problem with the line chart, is there any fix for this....

[danielgindi]

@PhilJay , @gjpc just found the source of this problem. All the people having this problem seem to be having one thing in common: They are giving y arrays which are not ordered.

In many places we are checking "is the value out of the viewport already?" -> "we can stop looping. break".

Now if the values are not ordered, we are stopping prematurely.

So we have to decide... Either:

1. Remove those checks, and just keep doing bound checks on all values. <-- May require performance comparisons on 10K points (we are applying the matrices + bounds checks on more points)
2. Just state that y value arrays must be ordered.

Actually when the viewport is zoomed in on the right edge of the chart, we already having the "performance penalty" of extra matrix multiplications, as we are not breaking out of the loop for the values on the left.
So there's a simple way to test the first case, without changing the code. Having a 10K points, and zooming in on the left, then playing a little. That will give the "normal" performance. Then go do the same on the other end of the chart. If it feels slower, then we have a problem.

[PhilJay]

Yes, that seems like the reason why it's not working for some people, if by "not ordered" you mean "not ordered by x-index" (ascending).

From what I have noticed, the matrix multiplications are not that time consuming. It's more the additional boundary checks and general runs through the drawing loops that decrease performance.

I will do some extensive testing in the following days and then report back.

If performance is too bad when the checks are removed, I suggest we stick with the option that the entries need to be ordered by x-index, what do you think Daniel?

[danielgindi]

I think that's a compromise that everyone will be willing to take. It's much faster to just sort the yValues array once by the user [of the library] instead of looping through 10K values on each render loop.

[PhilJay]

To be honest I think I already know that performance will be much worse if the checks are removed :-)
I will test it anyway, but I think its better to require the entries to be sorted.

[danielgindi]

I can also deduce that intuitively, but you know, you never know...

[gjpc]

The problem with sorting the x (or any) axis is that often the order the data is placed into an array is part of the dataset, let's call the order variable time instead of just an index. In a scatter plot tracking wilder beasts the X axis and Y axis are position and the index position of the coordinates is the time that the beast was at the point. The chart should be ready for multiple points where x,y are all equal, this happens when the wilder beast grazes. And yes, it always take longer to go through more data, but it is a necessary hit with some data sets.

Perhaps setting a flag in a DataSet object's first scan of the data to let the renderers know it aint so. It may be advisable to just to go ahead and set the flag when realtime data manipulation occurs.

[taltstidl]

@PhilJay Thanks for all the work you've put into this awesome library. If you decide to only allow sorted arrays for performance reasons, could you then make your Entry class implement Comparable? This would allow sorting the array to work out of the box.

[taltstidl]

For anyone also experiencing this issue you can fix it by doing the following:
Add a new class called EntryComparator:

public class EntryComparator implements Comparator<Entry> {
    @Override
    public int compare(Entry entry1, Entry entry2) {
        return entry1.getXIndex() - entry2.getXIndex();
    }
}

Use this class to sort the array containing the entries before passing it to the DataSet: Collections.sort(mEntries, new EntryComparator());
This solved the issue for my ScatterChart. Thanks again @PhilJay for this library and also to everyone else who has contributed. It saved me a lot of work. 😃

[gjpc]

As noted above, ordering the data will lose information from some data sets.

From Wikipedia:

A scatter plot is used when a variable exists that is under the control of the experimenter. If a parameter exists that is systematically incremented and/or decremented by the other, it is called the control parameter or independent variable and is customarily plotted along the horizontal axis. The measured or dependent variable is customarily plotted along the vertical axis. If no dependent variable exists, either type of variable can be plotted on either axis and a scatter plot will illustrate only the degree of correlation (not causation) between two variables.

[danielgindi]

I do not see how that description indicates that information will be lost.

On Wed, Jun 17, 2015 at 5:52 PM, Gerard J. Cerchio <[email protected]

wrote:

As noted above, ordering the data will lose information from some data
sets.

From Wikipedia:

A scatter plot is used when a variable exists that is under the control
of the experimenter. If a parameter exists that is systematically
incremented and/or decremented by the other, it is called the control
parameter or independent variable and is customarily plotted along the
horizontal axis. The measured or dependent variable is customarily plotted
along the vertical axis. If no dependent variable exists, either type of
variable can be plotted on either axis and a scatter plot will illustrate
only the degree of correlation (not causation) between two variables.

—
Reply to this email directly or view it on GitHub
#718 (comment)
.

[gjpc]

To go back to my example, a wilder beast will not only travel north and south over time but it will travel both east and west. The X axis is the east-west position of the animal on ground. if you order the east west data it will not reflect the actual travel of the animal, it will falsely show that the animal moves only in an easterly direction.

[danielgindi]

Nobody wanted you to changed neither your X values or Y values, only to put
them in the correct order in the array..............

On Wed, Jun 17, 2015 at 6:01 PM, Gerard J. Cerchio <[email protected]

wrote:

To go back to my example, a wilder beast will not only travel north and
south over time but it will travel both east and west. The X axis is the
east-west position of the animal on ground. if you order the east west data
it will not reflect the actual travel of the animal, it falsely show that
the animal moves only in an easterly direction.

—
Reply to this email directly or view it on GitHub
#718 (comment)
.

[gjpc]

moving the position of the data point on the x axis loses the time element, the sequence of the plot. I not only removed the two break statements to the scatter plot renderer to make it useful for tracking data, but I also added a flag to display the index as the value not the Y axis value. The index of my data is the relative time the x,y data point was acquired:

[gjpc]

[gjpc]

If I ordered my x data point 538 would be to the right of point 535 not to the left as it was in the data

[danielgindi]

We are going round and round. Nobody asked you to moved it on the x axis, but on the "i" axis. The index in the array. NOT the x-index, just index in the y-values array:

[
{ xIndex: 0, yValue: value... },
{ xIndex: 1, yValue: value... },
{ xIndex: 2, yValue: value... },
{ xIndex: 3, yValue: value... }
]

Instead of

[
{ xIndex: 0, yValue: value... },
{ xIndex: 3, yValue: value... },
{ xIndex: 1, yValue: value... },
{ xIndex: 2, yValue: value... }
]

Capiche?

[gjpc]

Non copiche, Daniel you are only showing the y value there where do I pump in the X value independent of the xIndex?

I have to already use the xVals array to set the size of the area under study

[danielgindi]

each entry has an x-index by itself, independent of its position inside the array

[gjpc]

And that X sets the size of the area that is under study, otherwise I can't control the area of the graph.

[gjpc]

ScatterChartData *data = [[ScatterChartData alloc] initWithXVals:xVals dataSets:dataSets];

Also there is only one set of xVals for multiple sets of yVals. In my data the yVals are not dependent on the xVals.

קאפיש

[danielgindi]

Man you are not listening.

[gjpc]

LOL, no, I am reading. Where in IOS-Charts do I plot multiple yVals each associated with its own unique xVals?

[danielgindi]

For each entry you pass an X index and a Y value. These are whatever you like, you do not need to change anything until now. The entries are in an array. The order of the entries in the array should be in relation to their x-indexes. So all you have to do is sort the array, nothing more. Do not touch the Entries.

[gjpc]

And sorting that array destroys the Sequence of when the data value was acquired. I am not only plotting Latitude (y) and Longitude (x) I am plotting time, time going in the only direction that time travels.

[danielgindi]

It shouldn't destroy anything. Rendering is based on x indexes, not on
array location .

[gjpc]

Connect the data points by a line between each successive data point. That series of line segments will look very different for the sorted and unsorted cases.

[taltstidl]

Let me chime in here:
What @gjpc has (as far as I understand it) is this: thinking of the chart as a map with locations denoted by the x and y values (int xIndex and float value) the beast takes a path on that map, say a circle. So the unsorted array that @gjpc stores has three pieces of information:

1. The longitude of a position of the beast (the xIndex stored in the Entry)
2. The latitude of a position of the beast (the value stored in the Entry)
3. The time/order in which the positions where collected (the index of the specific Entry in the ArrayList containing them)

Sorting the ArrayList will naturally cause the third information to be dropped, as the index of an Entry no longer correlates with the order in which data was collected.
@gjpc So what you can do is use the setData() method of an Entry or the corresponding constructor and store the order there (although that's admittedly a bit tedious)

[danielgindi]

I thought you are talking about scatter charts. The only case where the
order has any visual significance is when lines connect the dots...

‏

[taltstidl]

@danielgindi Hmm, I think @gjpc didn't really complain about the visual appearance. That naturally stays the same whether the array is sorted or not. But I think @gjpc still cares about that information for other purposes besides visual appearance.
@gjpc Wouldn't copying the array and sorting the copied array solve your issue? (Although it requires two instances the performance of the charts would still be improved)

[gjpc]

@danielgindi oh yes, and count on that line enhancement when I have time to add it to the scatter plot.

@TR4Android yes, you have encapsulated the problem. I'll look into setData Thanks!

[gjpc]

@TR4Android Copying the array removes the visual information. That is why I modified the ScatterPlotRenderer with an option to display the value of the i index not the Y value. With a flag valueIsIndex. You can see it at work in the plot that I posted above. Besides the simple NSNumberFormatter I would like to add

NSString *ComplexValueFormatter( int index, ChartDataSet *aDataSet )

to the base chart data set object. It hands over the index and data set so a more comprehensive value may be displayed in the value boxes on the chart; but that would take me some time because I have very little Swift under my belt.

Also performance is not much of a problem, my average sized data set renders at blinding speed on my iPad Air, and we bio folks are an unusually patient bunch, sitting around in the weeds all day. And I always set my X axis to be larger than any value in my data set.

[taltstidl]

@gjpc I have no experience on iOS whatsoever and thus cannot help you. On Android copying an array shouldn't remove the visual information as far as I know though, but that could be different in iOS.
Performance might not be the issue for the average developer, but there are some users who have a really large dataset and they wouldn't be too happy, so it should be optional.

[gjpc]

@danielgindi @TR4Android pushed the optional scatter lines to my branch https://github.com/gjpc/ios-charts