Filtering feature #36

kalwalt · 2020-01-27T17:49:18Z

This PR try to implement filtering of transpose matrix with ArFilterTransMat method. Probably with this code the interpolation routine that was implemented to smooth and remove the jittering of the model became un-necessary, i'm refering to this part of the code:

jsartoolkit5/examples/nft_improved_worker/threejs_worker.js

Lines 179 to 191 in 244b2b2

    
           } else { 
        
             sphere.visible = true; 
        
                   // interpolate matrix 
        
                   for (var i = 0; i < 16; i++) { 
        
                     trackedMatrix.delta[i] = world[i] - trackedMatrix.interpolated[i]; 
        
                     trackedMatrix.interpolated[i] = 
        
                       trackedMatrix.interpolated[i] + 
        
                       trackedMatrix.delta[i] / interpolationFactor; 
        
                   } 
        
                   // set matrix of 'root' by detected 'world' matrix 
        
                   setMatrix(root.matrix, trackedMatrix.interpolated); 
        
           }

with this PR simply became:

 } else {
          sphere.visible = true; 
                // set matrix of 'root' by detected 'world' matrix
                setMatrix(root.matrix, world);
        }

filterCutoffFrequency i set to AR_FILTER_TRANS_MAT_CUTOFF_FREQ_DEFAULT that means has a default values of 5.0, in the nftSimple.c example instead is set to 15.0

Build: build with emscripten Docker (latest)

kalwalt · 2020-01-27T18:04:20Z

emscripten/ARToolKitJS.cpp

+					transF[j][k] = trans[j][k];
+				}
+			}
+			if (arFilterTransMat(arc->ftmi, transF, 1) < 0) {


The third argument of arFilterTransMat should a value that is zero or 1 as explained in the doc http://www.artoolkitx.org/artoolkit5/doc/apiref/arFilterTransMat_h/index.html#//apple_ref/c/func/arFilterTransMat
this needs to be changed.

kalwalt · 2020-01-27T19:05:20Z

i added a arLog to retrieve the value of the filterCutoffFrequency and it give me 0 so i think the code must be changed. So default value is 0?

kalwalt · 2020-01-27T19:34:20Z

AR_FILTER_TRANS_MAT_SAMPLE_RATE_DEFAULT and AR_FILTER_TRANS_MAT_CUTOFF_FREQ_DEFAULT defined in arFilterTransMat.h:

https://github.com/artoolkitx/artoolkit5/blob/5bf0b671ff16ead527b9b892e6aeb1a2771f97be/include/AR/arFilterTransMat.h#L66-L81

as i wrote in the top comment. Maybe i was receiving that value because i haven't added them to ARBindEM.cpp

nicolocarpignoli · 2020-01-28T08:16:29Z

I will test it! @kalwalt does it work with threejs_gltf_worker example?

kalwalt · 2020-01-28T09:06:22Z

I will test it! @kalwalt does it work with threejs_gltf_worker example?

works with all the nft examples but i removed the interpolation routine (that in the javscript code) in a dedicated example main_threejs_filter_worker.html try this and if you want the others. But i think i still need to fix few things to make some comparisons.

kalwalt · 2020-01-30T12:45:16Z

@nicolocarpignoli @ThorstenBux I added a WIP example with gpu.js but i think that there are many variables in this approach. I may implement a kernel for the matrix linear interpolation. Probably in this case that is not very recommended and i will remove it. Another idea is to implement a more efficient routine (linear interpolation) inside the C++ code. Probably will result faster?

kalwalt · 2020-01-31T23:51:41Z

@nicolocarpignoli @ThorstenBux i think the version with filtering and linear interpolation in the C++ code is much better, I'd like to know what you think 😄
Testing example: examples/nft_improved_worker/main_threejs_filter_worker.html

kalwalt · 2020-02-01T18:20:40Z

@nicolocarpignoli yes with the model animation, seems worste. I need to undersand why. Also fps are very slow on my Mobile device. Tested with a Wiko, View model with Android 7.1.2 and Chrome browser 79.0.3945.136

kalwalt · 2020-02-01T18:22:17Z

I will also remoove my gpu.js attempt, i think i will try to optimize the code without it, even if it sound interesting.

emscripten/ARToolKitJS.cpp

kalwalt · 2020-02-06T16:57:56Z

emscripten/ARToolKitJS.cpp

@@ -112,6 +120,8 @@ extern "C" {

 		float trans[3][4];
 		ARdouble transF[3][4];
+		ARdouble transFLerp[3][4];
+		memset( transFLerp, 0, 3 * 4 * sizeof(ARdouble) );


maybe this is not needed but not sure of this.

It is needed, without it the interpolation give weird results.

kalwalt · 2020-02-06T16:58:46Z

emscripten/ARToolKitJS.cpp

@@ -160,6 +170,9 @@ extern "C" {
 			if (arFilterTransMat(arc->ftmi, transF, reset) < 0) {
 					ARLOGe("arFilterTransMat error with marker %d.\n", markerIndex);
 			}
+
+			matrixLerp(transF, transFLerp, 24.0);


try with another interpolationFactor, maybe 12.0 instead of 24.0?

yes we have to try different values

kalwalt · 2020-03-24T20:10:21Z

Testing example: https://kalwalt.github.io/kalwalt-interactivity-AR/nft/nft_threejs_filter_worker_gltf.html

kalwalt · 2020-03-29T19:06:20Z

I found in the BabylonAR repository this 'naive' implementation of the Kalman filter, I should try it, it looks very promising!

ThorstenBux · 2020-03-29T20:11:40Z

Tested,

duck is jittering like crazy

evaristoc · 2020-05-15T21:02:06Z

@kalwalt #36 (comment) === this is a good way to go.

The Kalmar becomes a linear function after running for the first time so it can be even implemented in JS for some few relevant points.

Again, finding the right parameters might change by image and perhaps by device. But that I don't know.

If that the case, finding the right parameters might require a search algorithm (I have done some statistics / DS, so that my reference). But then you need to clarify which functions to optimise. I haven't done that job yet. A target success/failure variable is required.

The search can be automated if the right functions are formulated. The search might be slow and data intensive.

There are few more things that can be done to improve jittering that I have been testing. I will make a video and share the link to show you the results when having time. The target identification (short distance only) and homography were very good using the artoolkit algo for NFT. I will check later which algo is being used, I am really curious now. Previous essays I made with a simpler algos were not as good in detection, especially if the target image was a bit blurred, etc.

I will study some C++ shortly...

evaristoc · 2020-05-15T21:17:06Z

@kalwalt #36 (comment)

Nice reference! Thanks!

evaristoc · 2020-05-15T21:23:05Z

#36 (comment)

Yeah... the same here.

kalwalt · 2020-05-15T22:48:40Z

Hi @evaristoc recently i added a combined lerp and filtering (with arFilterTransMat) to my JsartoolkitNFT (dev branch) webarkit/jsartoolkitNFT#35 see if this is better or not. Also take a look at this discussion in the artoolkitx/jsartoolkit5 issue tracker

evaristoc · 2020-05-17T08:44:09Z

@kalwalt I check and let you know? Same testing link, I guess...

evaristoc · 2020-05-17T08:47:50Z

@kalwalt still jittering. Specially from some positions of the marker 😞

evaristoc · 2020-05-17T08:50:18Z

@kalwalt what is exactly arFilterTransMat? It is almost obvious from the name that it refers to the transformation matrix, but what does exactly do? Do you know?

EDIT: first order low pass filter: https://www.electronics-tutorials.ws/filter/filter_2.html: "the low pass filter only allows low frequency signals from 0Hz to its cut-off frequency, ƒc point to pass while blocking those any higher."

It might be close to what it is done when practicing time series analysis in statistics. It looks like it is assuming a certain frequency (a sinusoidal signal) and implementing a linear model (first order) to clean it. It appears to work well for certain positions, likely because it can capture the signal.

kalwalt · 2020-05-17T09:02:53Z

@kalwalt still jittering. Specially from some positions of the marker 😞

Hi @evaristoc have you tested which code?
arFilterTransMat is method coming from Artoolkit http://www.artoolkitx.org/artoolkit5/doc/apiref/arFilterTransMat_h/index.html (reference by ARtoolkitX but it is the same) read:
High frequency noise ("jitter") can be an undesirable property in optical tracking systems like ARToolKit. These functions implement a first-order low-pass filter for both the position and orientation components of the pose estimate transformation matrices.

Probably i didn't applyed correctly,It seems that has no effect ( or very low).

evaristoc · 2020-05-17T09:03:57Z

Hi @kalwalt I was editing my comment above:
EDIT: first order low pass filter: https://www.electronics-tutorials.ws/filter/filter_2.html: "the low pass filter only allows low frequency signals from 0Hz to its cut-off frequency, ƒc point to pass while blocking those any higher."

It might be close to what it is done when practicing time series analysis in statistics. It looks like it is assuming a certain frequency (a sinusoidal signal) and implementing a linear model (first order) to clean it. It appears to work well for certain positions, likely because it can capture the signal.

evaristoc · 2020-05-17T09:05:10Z

You might need not one but two or more of those to complete the job...

kalwalt · 2020-05-17T09:06:01Z

Hi @kalwalt I was editing my comment above:
EDIT: first order low pass filter: https://www.electronics-tutorials.ws/filter/filter_2.html: "the low pass filter only allows low frequency signals from 0Hz to its cut-off frequency, ƒc point to pass while blocking those any higher."

It might be close to what it is done when practicing time series analysis in statistics. It looks like it is assuming a certain frequency (a sinusoidal signal) and implementing a linear model (first order) to clean it. It appears to work well for certain positions, likely because it can capture the signal.

So maybe you can help, to find what is not correct in my approach. I Think it should filter the noise, but of course it should be done with a correct approach/method.

evaristoc · 2020-05-17T09:10:25Z

@kalwalt let's see. I expect to find a roof after which we can not improve the method any more, either because affects on performance or because it will mean to redo some aspects of the project.

Right now we are post-processing it.

But of course I want to help!

evaristoc · 2020-05-17T09:14:47Z

I would first advice to create a table or something visible of the inputs and outputs, @kalwalt?
You want to smooth the signal, so you might need several passes through different filters.

We should evaluate other filters too, as you have been mentioning in other repos.

evaristoc · 2020-05-17T09:43:10Z

@kalwalt will later make a sort of list of all the fronts where you are dealing with this (they are a few).

Let me know where you want to deal with this in particular. Here?

Meanwhile if you can point me to the script where you implemented this particular filter, the low pass one? We can probably create a factory for using not one but different kind of filters.

I suspect filters will have to be implemented case by case. I expect the jittering to change for different target images. I even expect that to change between devices, etc. But let's see, I need to research this more.

kalwalt · 2020-05-17T09:56:58Z

You can open an issue in this repository and fill all the data there and link to this PR. I prefere here because much control than in artoolkitx/jsartoolkit5. Eventually if yu are interested we have created a new org https://github.com/webarkit. If you are interested to join us, let me know! 🙂

kalwalt · 2020-05-17T10:02:47Z

the arFilterTransMat is used in ARToolKitJS,cpp the code is wrapped in defines:

#if WITH_FILTERING
// code here
#endif

So you are not obliged to compile it, with filtering.

evaristoc · 2020-05-17T14:44:59Z

I will be having a look to different examples made with the same library. Example: http://jevois.org/

And keeping this one for further reading: https://answers.opencv.org/question/182075/convert-pose-from-artoolkit-into-opencv-screen-coordinates/

It seems to be some kind of overlap between the two. I have been doing some opencv recently too.

evaristoc · 2020-05-17T14:52:34Z

@kalwalt apparently this is what ARfilterTransMat does

The filter tries to correct orientation and position. For orientation it tries to keep the changes within a frequency. I think it is here where the actual low pass is done (EDIT: it is implemented in both, for orientation and for position, with the same calculated alpha, only that for a different equation (trig for orientation, linear for position)). For position is a simple lerp. It uses an alpha value for BOTH corrections and that for my perspective is a hmmm.....

That means you are probably implementing a second lerp over a lerp when going this route. (EDIT: I was referring here to the use of a custom LERP in other parts of the code you had been implementing.)

alpha is the corrector already explained in the material I passed to you.

It looks quite simple, IMHO.

I haven't checked at what point of the whole process flow it is implemented, but it is likely a post-processing, as we are all trying to do.

evaristoc · 2020-05-17T15:08:01Z

You can open an issue in this repository and fill all the data there and link to this PR. I prefere here because much control than in artoolkitx/jsartoolkit5. Eventually if yu are interested we have created a new org https://github.com/webarkit. If you are interested to join us, let me know!

Yes, nice thanks!

evaristoc · 2020-05-18T10:05:42Z

@kalwalt

Some additional observations about the ARfilterTransMat and jitter smoothing:

In the ARfilterTransMat the proposed sample rate is a fixed value of no particular use, IMO. I might be wrong, haven't seen all the implementation yet.
The resulting alpha value that will be used in the function as weighting is a constant. That is not necessarily a bad thing and it is the simplest implementation. However, variability of the noise signal might change from different positions, actual conditions (eg. lighting, device, position of the target image in front of the camera, etc)...
I have seen the ARfilterTransMat is being implemented in serious projects. That doesn't mean it is good though, but maybe convenient.
There is an interpolation used to correct positioning in the function. This is what the simple interpolation will do. Eg.

y0 = 10; //previous position
y1 = 20; //new position
alpha = 0.1; //I think it usually gives a low value in the function

y1Entry = alpha*y1 + (1-alpha)*y0 = 2 + 9 = 11 //interpolation

Similarly:

y0 = 10;
y1 = 0;
alpha = 0.1;

y1Entry = alpha*y1 + (1-alpha)*y0 = 0 + 9 = 9 //interpolation

In both cases, y1Entry will be used as y0 for the next frame.

In principle: you need to find a cut-off value that makes alpha small/big enough to smooth between y1 and y0. For what I have seen in NFT coordinates, the values of those coords are usually big, so you might have to try big numbers of cut-off value to see some effect.
Something about interpolation that might be useful: http://paulbourke.net/miscellaneous/interpolation/. A sort of interpolation could be a weighted moving average of two or more values, if I recall correctly.
The effect on the orientation smoothing by the ARfilterTransMat function is still to be checked.
I implemented a more discrete form of a pass filter here together with here. WARNING: Not without artifacts or performance costs!. I think my implementation looks more like a band pass filter, not a low pass filter, and mine is sample-based. You can read it as follow:

if new value changes over a previous value +/- cut-off, make new entry as previous value +/- cut-off, otherwise make new entry equal new value.
In other words (I think...):
y1Entry = min(y0 + cutoff, max(y0 - cutoff, y1))

kalwalt · 2020-05-18T11:22:47Z

Thank you @evaristoc for the long deep comment, i will think about it. 😄

Carnaux · 2021-04-13T21:37:26Z

@kalwalt how is the progress on this feature? Is it the same we talked about in the WebArkit slack?

kalwalt · 2021-04-13T21:47:02Z

@Carnaux haven't done some progress lately... Yes, this is in part what we talked in the WebARKit slack channel.

kalwalt added 2 commits January 25, 2020 18:45

new FilterNFT files, not tested

8071d19

filtering working, seems better than interpolating with js code.

2aee370

kalwalt added enhancement New feature or request C/C++ coding with C and C++ language code design NFT nft markerless technology labels Jan 27, 2020

kalwalt self-assigned this Jan 27, 2020

kalwalt mentioned this pull request Jan 27, 2020

Jttering of the model or mesh when tracking a NFT marker #30

Open

kalwalt commented Jan 27, 2020

View reviewed changes

removing FilterNFT files

0c76369

alfilterCutoffFreq always gives values = zero

c971a84

kalwalt added 4 commits January 28, 2020 11:30

enhanced settings filterSampleRate=50.0 filterCutoffFrequency=25.0

6f4446c

added reset option

0d04071

adding gpujs with a WIP example

298ed62

simplifying things, but receive an error

546ae82

new linear interpolation in C++ code much more efficient

d97349b

nicolocarpignoli and others added 2 commits February 1, 2020 15:24

for filter, adding only an example with gltf

d7f738a

updated index for nft examples

a835960

removing gpu.js code and libs

a987c47

kalwalt commented Feb 6, 2020

View reviewed changes

emscripten/ARToolKitJS.cpp Show resolved Hide resolved

kalwalt commented Feb 6, 2020

View reviewed changes

adding NaiveFilter.h

e7e7f24

kalwalt mentioned this pull request Apr 13, 2020

Q's re hosting and NFT without AR webarkit/jsartoolkitNFT#9

Closed

kalwalt added 2 commits April 28, 2020 20:24

another lerp routine

42347a8

rebuilding libs

86fb0a9

	} else {
	sphere.visible = true;
	// interpolate matrix
	for (var i = 0; i < 16; i++) {
	trackedMatrix.delta[i] = world[i] - trackedMatrix.interpolated[i];
	trackedMatrix.interpolated[i] =
	trackedMatrix.interpolated[i] +
	trackedMatrix.delta[i] / interpolationFactor;
	}

	// set matrix of 'root' by detected 'world' matrix
	setMatrix(root.matrix, trackedMatrix.interpolated);
	}

Filtering feature #36

Are you sure you want to change the base?

Filtering feature #36

Conversation

kalwalt commented Jan 27, 2020 • edited Loading

kalwalt Jan 27, 2020 • edited Loading

Choose a reason for hiding this comment

kalwalt commented Jan 27, 2020 • edited Loading

kalwalt commented Jan 27, 2020

nicolocarpignoli commented Jan 28, 2020

kalwalt commented Jan 28, 2020

kalwalt commented Jan 30, 2020

kalwalt commented Jan 31, 2020

kalwalt commented Feb 1, 2020

kalwalt commented Feb 1, 2020

kalwalt Feb 6, 2020

Choose a reason for hiding this comment

kalwalt Mar 24, 2020

Choose a reason for hiding this comment

kalwalt Feb 6, 2020

Choose a reason for hiding this comment

nicolocarpignoli Feb 7, 2020

Choose a reason for hiding this comment

kalwalt commented Mar 24, 2020

kalwalt commented Mar 29, 2020

ThorstenBux commented Mar 29, 2020

evaristoc commented May 15, 2020 • edited Loading

evaristoc commented May 15, 2020

evaristoc commented May 15, 2020

kalwalt commented May 15, 2020

evaristoc commented May 17, 2020

evaristoc commented May 17, 2020 • edited Loading

evaristoc commented May 17, 2020 • edited Loading

kalwalt commented May 17, 2020

evaristoc commented May 17, 2020

evaristoc commented May 17, 2020

kalwalt commented May 17, 2020

evaristoc commented May 17, 2020

evaristoc commented May 17, 2020 • edited Loading

evaristoc commented May 17, 2020

kalwalt commented May 17, 2020

kalwalt commented May 17, 2020

evaristoc commented May 17, 2020 • edited Loading

evaristoc commented May 17, 2020 • edited Loading

evaristoc commented May 17, 2020

evaristoc commented May 18, 2020 • edited Loading

kalwalt commented May 18, 2020

Carnaux commented Apr 13, 2021

kalwalt commented Apr 13, 2021

kalwalt commented Jan 27, 2020 •

edited

Loading

kalwalt Jan 27, 2020 •

edited

Loading

kalwalt commented Jan 27, 2020 •

edited

Loading

evaristoc commented May 15, 2020 •

edited

Loading

evaristoc commented May 17, 2020 •

edited

Loading

evaristoc commented May 17, 2020 •

edited

Loading

evaristoc commented May 17, 2020 •

edited

Loading

evaristoc commented May 17, 2020 •

edited

Loading

evaristoc commented May 17, 2020 •

edited

Loading

evaristoc commented May 18, 2020 •

edited

Loading