Add an averaging function to OpenCV marker detection

[brettfiedler]

From #20: Detection jitter around all-side-equal and all-angle-equal is a bit frenzied as it enters and leaves the tolerance interval.

We'd like a simple averaging function for a few data points to smooth out the detection jitter as a result of OpenCVs marker detection uncertainty.

We want to balance smoothness with delay in changing the vertex position.

@jessegreenberg mentioned he would look at tackling it this way: https://www.arduino.cc/en/Tutorial/BuiltInExamples/Smoothing

[jessegreenberg]

A basic smoothing function like https://www.arduino.cc/en/Tutorial/BuiltInExamples/Smoothing is working well, but the smoothing does...smooth. New positions take a little longer to be set in the sim.

I tried reducing the number of saved values for the average (currently it is 10), but anything less had a lot of jitter. Maybe we can weight the average to prefer the more recent values.

[jessegreenberg]

I added a weighted average. It is snappier.

Even without the weighted average I am still seeing the jitter around all-sides-equal and all-angle-equal.

I think the thing that will solve this is to actually apply a resolution either in the sim or in the TMQ code that does not update a vertex position unless it has moved enough.

Actually, I don't think that the vertex positions from the TMQ are constrained to the model grid (smallest keyboard step size).

[jessegreenberg]

The above commits have this working better, the all-angle-equal state seems pretty stable.

[jessegreenberg]

OK this has been added. @BLFiedler over to you to test. Let me know if we need to modify this more or consider a different approach if it is still too noisy.

[jessegreenberg]

I missed #116, I think that issue will help with some of the remaining jitter.

[brettfiedler]

I want to see what folks think about the delay this introduces. It is noticeable if you're moving with moderate speed. I'll get a quick instruction video made up to send to folks and start gathering thoughts.

[brettfiedler]

Posting an idea for discussion - what if we assume smooth movement and

1.) disallow any changes that seem "instantaneous" within a small range of data points
2.) if the new position persists, then accept it as the new vertex position and update.

I think this is akin to the input grid? Or is there another way to just restrict points to the input grid and see what happens there?

EDIT: We could make the input grid larger for the tangible. Maybe we try 2X and see how it feels?

EDIT EDIT: How much of an averaging function do we need if the points are constrained to the input grid and that results in no appreciable boundary crossing?

[brettfiedler]

Let me know if these should live in different issues, but from the last meeting (5/10), we should:

• Investigate reducing the aggregation of squares (smaller into bigger) in the masking stage of detection, which may reduce how much the center moves
• Make the input grid 2X (maybe bigger) for the tangible/OpenCV inputs
• Reduce the center point averaging to fewer samples to reduce input delay if the above methods help substantially.

[jessegreenberg]

Investigate reducing the aggregation of squares

The idea behind this was that we are using what OpenCV calls "morphological operations" to reduce the noise after the filter. (https://docs.opencv.org/4.x/d9/d61/tutorial_py_morphological_ops.html). It removes areas of white pixels that are smaller than a provided region size, then restores the remaining regions by the same amount. The current region size is 10x10. Without any morphology it looks like this:

And there is still a lot of noise/wobble.

Size of the morphology matrix is currently 10x10. If it is reduced to 5x5 the squares look like this:

and there is still a lot of noise in the positioning. However, making the grid larger when the device is connected seems to fix the noise that propagates to the sim. But we are still getting sound every animation frame while it is a parallelogram and I am not sure why yet.

[jessegreenberg]

But we are still getting sound every animation frame

This is because new Vertex positions are set as new Vector2 instances even if the values are the same, which ensures that the Property listeners are firing. We need a guard that makes sure that a Vector2 is only set if it is different from the current value.

[jessegreenberg]

The above commit prevents it from singing every frame.

[jessegreenberg]

OK this is behaving a bit better. I didn't remove the smoothing function yet because I still think it helps. I think it is likely fast enough that a user would not notice it. But we can certainly adjust/remove it.

@BLFiedler can you please test and let me know if anything else is needed or this issue? I recommend query parameters ?showModelValues&deviceShapeAngleToleranceInterval=0.05&deviceShapeLengthToleranceInterval=0.03&toleranceIntervalScaleFactor=6&deviceAngleToleranceInterval=0.05

[jessegreenberg]

The noise was bothersome during development today, probably because of the lighting conditions today. I tried a bluring function against the camera input but it didn't help, just noting here:

Index: js/SimulationNode.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/js/SimulationNode.js b/js/SimulationNode.js
--- a/js/SimulationNode.js	(revision b87c2d279cbf648e74415e47e12c5731286d8968)
+++ b/js/SimulationNode.js	(date 1652907275593)
@@ -12,7 +12,7 @@
       parentQueryString += '?';
     }
 
-    const defaultQueries = 'postMessageOnLoad&ea';
+    const defaultQueries = 'postMessageOnLoad&ea&deviceConnection';
 
     const fullQueryString = parentQueryString + defaultQueries;
 
Index: index.html
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/index.html b/index.html
--- a/index.html	(revision b87c2d279cbf648e74415e47e12c5731286d8968)
+++ b/index.html	(date 1652916294299)
@@ -483,18 +483,28 @@
       // input as an image
       let imageSource = cv.imread( transferCanvasNode.elementId );
 
+      let blurDst = new cv.Mat();
+      const kSize = 3;
+      let blurKern = cv.Mat.ones( kSize, kSize, cv.CV_32FC1 );
+      blurKern = blurKern.mul( blurKern, 1 / ( kSize * kSize ) ); // don't produce an image with more "energy"
+      let blurAnchor = new cv.Point( -1, -1 );
+      cv.filter2D( imageSource, blurDst, cv.CV_8U, blurKern, blurAnchor, 0, cv.BORDER_DEFAULT );
+      blurKern.delete();
+
       // flip according to settings
       if ( horizontalFlipProperty.value ) {
-        cv.flip( imageSource, imageSource, 1 ); // 1 indicates around the y-axis (horizontal flip)
+        cv.flip( blurDst, blurDst, 1 ); // 1 indicates around the y-axis (horizontal flip)
       }
       if ( verticalFlipProperty.value ) {
-        cv.flip( imageSource, imageSource, 0 ); // 0 indicates around the x-axis (vertical flip)
+        cv.flip( blurDst, blurDst, 0 ); // 0 indicates around the x-axis (vertical flip)
       }
-      cv.imshow( flippedOutputCanvasNode.elementId, imageSource );
+      cv.imshow( flippedOutputCanvasNode.elementId, blurDst );
 
       // convert to HSV for filters
       let hsvSource = new cv.Mat();
-      cv.cvtColor( imageSource, hsvSource, cv.COLOR_RGB2HSV );
+      cv.cvtColor( blurDst, hsvSource, cv.COLOR_RGB2HSV );
+
+      blurDst.delete();
 
       // get the values to apply to filter out the green of the device
       const lowFilterValues = getMinFilterValues();
@@ -510,7 +520,7 @@
       let M = cv.Mat.ones( 10, 10, cv.CV_8U );
       let anchor = new cv.Point( -1, -1 );
       cv.erode( filterOutputMat, erosionMat, M, anchor, 1, cv.BORDER_CONSTANT, cv.morphologyDefaultBorderValue() );
-      cv.dilate( erosionMat, erosionMat, M, anchor, 1, cv.BORDER_CONSTANT, cv.morphologyDefaultBorderValue() );
+      // cv.dilate( erosionMat, erosionMat, M, anchor, 1, cv.BORDER_CONSTANT, cv.morphologyDefaultBorderValue() );
 
       // displays contours AND puts data on a canvas to find contours
       cv.imshow( filterOutputCanvasNode.elementId, erosionMat );

[brettfiedler]

Currently there are controls for the smoothing in the OpenCV control panel

[brettfiedler]

@emily-phet, we plan to publish with the Bluetooth parameters for external devices. Do we want to investigate packaging any of the OpenCV work or leave this to an unpublished branch (or whatever the best option is for future research use)?

[emily-phet]

Let's leave the OpenCV effort in a state we can come back to later. From what I understand, it would take some effort to get this feature into a more easily shareable state, and considerable effort to get it into a state polished enough for publication. Let's defer that effort for now.