Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Calibrate HSV measurements #116

Open
laurensvalk opened this issue Aug 25, 2020 · 18 comments
Open

[Feature] Calibrate HSV measurements #116

laurensvalk opened this issue Aug 25, 2020 · 18 comments
Labels
enhancement New feature or request platform: Powered Up Issues related to LEGO Powered Up software: pybricks-micropython Issues with Pybricks MicroPython firmware (or EV3 runtime) topic: sensors Issues involving sensors

Comments

@laurensvalk
Copy link
Member

laurensvalk commented Aug 25, 2020

Is your feature request related to a problem? Please describe.
The hues reported by the SPIKE Prime Color sensor, as well as the ones we calculate for the BOOST ColorDistanceSensor aren't really great in the 0--120 degrees region.

For example, yellow is reported at a hue of about 30, while in reality it should be about 50--60.

This is currently not a problem since the matching value is set to 30 in the color map. However, it would be a problem for displaying hsv colors on the status light, because then it would look orange.

Describe the solution you'd like
Pass the RGB and/or HSV values through a mapping that corrects this. This way, measuring a color (e.g. yellow) and showing it on the status light should produce the correct result.

In the case of SPIKE Prime, it means we should no longer use its HSV mode, but compute HSV ourselves from the RGB values. This seems to be what the sensor is doing anyway, so we don't seem to lose any information with this approach.

First steps towards solution
I've been starting to collect some measurents of scanning LEGO bricks. Each object is a stack of at least 3 bricks of the same color, and I was scanning the flat side of the stack. I placed them at such a distance from the sensor such that the V value was constant across all measurements.

This table reports the values we currently read, compared to what they should be according to the Bricklink color guide.

Bricklink Hue SPIKE Hue SPIKE RGB (V=70) BOOST Hue BOOST RGB (V=30) BOOST RGB V=50 NXT Hue
Red 358 350 360, 116, 155 358 132, 15, 21 227, 29, 38 358
Orange 27 3 359, 158, 146 1 132, 22, 16 221, 35, 32 1
Yellow 53 29 364, 276, 188 29 138, 78, 23 220, 133, 39 30
Green 147 155 180, 360, 288 132 43, 134, 62 ?, ?, ? 120
Spike Blue 199 197 150, 300, 356 190 35, 116, 136 55, 189, 220 190
Blue 209 215 117, 216, 361 219 22, 60, 137 35, 103, 220 220
Spike Magenta 342 339 362, 153, 228 347 135, 23, 51 222, 42, 83 345

Notes

  • The RGB and V values are not currently comparable between SPIKE/BOOST. In the SPIKE case, this is currently the V value obtained from the sensor, which is scaled differently. SPIKE RGB saturates at about 1023, while BOOST RGB peaks at about 440 or 500.
@laurensvalk laurensvalk added enhancement New feature or request platform: Powered Up Issues related to LEGO Powered Up topic: sensors Issues involving sensors software: pybricks-micropython Issues with Pybricks MicroPython firmware (or EV3 runtime) labels Aug 25, 2020
@laurensvalk
Copy link
Member Author

laurensvalk commented Aug 25, 2020

I'd like to tag @tomalphin who looks to be quite the expert on LEGO colors.

The colors in your diagrams seem to be slightly different than the Bricklink colors. If not Bricklink, which reference would you recommend as the standard to pick the right hue for LEGO colors?

EDIT: Ah, I see this comment on your blog:

I extracted colors from a photograph that I took, which was then calibrated to neutral gray using a gray card. At the time, I did not have a more advanced color profiling system, but I now own an x-rite Color Checker Passport, and at some point in the future I plan to photograph the same selection of bricks using multiple cameras, all corrected using the gray card and custom profiles, and then finally averaging the RGB / HSL values.

In case you are interested to be involved, it would be awesome to have accurate reference values for the primary/secondary LEGO colors. It would be pretty cool to calibrate the sensor as a super accurate color scanner which goes at just $15 on the official shop.

@laurensvalk
Copy link
Member Author

A patch for this is applied here. More accurate HSV representation would be nice, but it is less critical now that the user can do their own calibration.

@laurensvalk
Copy link
Member Author

A patch for this is applied here. More accurate HSV representation would be nice, but it is less critical now that the user can do their own calibration.

This is apparently not good enough, so let's reopen it.

Related: pybricks/pybricks-micropython#93. @Novakasa

@Novakasa
Copy link

Novakasa commented Apr 26, 2022

I could take a look at this, at least I can extract some raw RGB data and look at the deviation for some LEGO colors.

Should we then incorporate a mapping already in raw_to_rgb(...) for the specific sensors? I'm guessing that would enable us to catch any sensor dependence at the root. But, depending on the mapping required it might be difficult to do it in RGB space.

@Novakasa
Copy link

In theory, there could be a distance-dependent mapping to attempt to compensate for a given distance of the surface. This probably should be provided by the user.

@laurensvalk
Copy link
Member Author

Instead of fixing/equalizing the RGB readings, we could perhaps have some mapping, e. g. like the inverse of https://github.com/pybricks/pybricks-micropython/blob/f69bd694ba481563c980e09fae41b60c0ecea4fc/lib/pbio/src/color/conversion.c#L85

And use sensor type specific parameters if needed to ensure that the rgb-to-hsv conversion produces consistent hsv values.

I don't think we need to match LEGO colors at this level per se, but they are still useful as a reference since all of us can reproduce the same results then.

@laurensvalk
Copy link
Member Author

Would only v be distance dependent in theory?

If we can get H to be consistent across sensors, I think that's a good start.

@Novakasa
Copy link

Would only v be distance dependent in theory?

That was my impression when I last did a very superficial test, but I'll look at it more systematically.

@Novakasa
Copy link

Novakasa commented Apr 30, 2022

Today I played around with this setup:
image
The train car has a sensor pointed to the side and some colored bricks are posted in line. I made the "scanning" somewhat consistent and changed the distance of the colored bricks to the sensor surface in increments of 1 lego plate from 1 to 11 plates. I also did the whole thing with lights on and lights off (admittedly not an extreme change in ambient lighting).

It's still early, but here is a result that kind of shows that in large part only the value changes with the distance, with the sweetspot being 2 plates distance.
image

In my case the effect of ambient light conditions is rather negligible.

I've pushed the related files to this repo: https://github.com/Novakasa/pybricks-color

@Novakasa
Copy link

Also, the largest RGB value I encountered is 473, exceeding the 440 that is used when mapping to 0-255 in raw_to_rgb(...).
I am using PUPDevice to measure the rgb values, but I haven't checked if that would amount the the same data as in raw_to_rgb.

@laurensvalk
Copy link
Member Author

Nice, that is good to know. Which mapping was used to produce this? The default or your PR?

If I recall correctly, this also hold qualitatively for the SPIKE sensor, but the value for saturation and value were a bit different under the same conditions. Perhaps the sensor-type specific mapping from rgb-to-hsv can take care of making them more similar.

Also, the largest RGB value I encountered is 473, exceeding the 440 that is used when mapping to 0-255 in raw_to_rgb(...).

This might well be the case. I think there was an (unfortuate) hard cap of 440 on the SPIKE sensor RGB values coming out of the sensor. As a result, hue calculations break when the brick is too close.

I am using PUPDevice to measure the rgb values, but I haven't checked if that would amount the the same data as in raw_to_rgb.

Yes, these are the RGB values we use.

@Novakasa
Copy link

Novakasa commented May 2, 2022

Which mapping was used to produce this? The default or your PR?

I only recorded the RGB values from PUPDevice, the mapping to hsv was done after the fact using the wikipedia algorithm. So, it should be the same as the current mapping in master.

I think there was an (unfortuate) hard cap of 440 on the SPIKE sensor RGB values coming out of the sensor. As a result, hue calculations break when the brick is too close.

So the ColorDistanceSensor uses the same mapping to make its RGB values consistent with the SPIKE sensor? If I see it correctly, we get an int overflow in raw_to_rgb with values above 440.

@laurensvalk
Copy link
Member Author

I only recorded the RGB values from PUPDevice, the mapping to hsv was done after the fact using the wikipedia algorithm. So, it should be the same as the current mapping in master.

Except for the hack that skews some of the values. :) So when posting graphs (which are awesome, keep making them!), maybe add one line directly underneath to say how it was made.

So the ColorDistanceSensor uses the same mapping to make its RGB values consistent with the SPIKE sensor?

It uses the same mapping right now, but that does not result in consistent HSV values. The idea is to make one rgb-to-hsv mapping for the SPIKE sensor, and another for the Boost Color Distance Sensor, such that the end-user HSV values are comparable.

@Novakasa
Copy link

Novakasa commented May 2, 2022

Except for the hack that skews some of the values.

Oh yeah, kind of essential for this topic :P

It uses the same mapping right now, but that does not result in consistent HSV values

Hmm, I think I still don't quite get it. If I look at the mapping from raw to rgb both seem to use different approaches. SPIKE vs Color-Distance-Sensor. The ColorDistanceSensor maps from 0-440 to 0-255, while SPIKE maps from 0-1023 to 0-255. As I encountered raw values above 440 using the ColorDistanceSensor, shouldn't we adapt the threshold there?

BTW, In my plot above the hue is in radians, so it is kind of hard to see it really, but the hue didn't really change much with distance.

@laurensvalk
Copy link
Member Author

If I look at the mapping from raw to rgb both seem to use different approaches.

You're right 👍

shouldn't we adapt the threshold there?

We should probably drop every scaling and skewing operation we have, including the one you found (and I forgot about 😄), and replace it with a sensor-type-dependent raw-rgb-to-hsv conversion.

@Novakasa
Copy link

Novakasa commented May 3, 2022

image
For this plot, I took the colors from rebrickable and plotted their RGB values. Then I overlaid my measured dataset at distance=4 (chosen to be the closest match to the rebrickable colors, which is kind of arbitrary because the RGB data is not normalized).

The order of LEGO colors is yellow, green, red, black, white, blue

This plot shows that a mapping in RGB space would have to be nonlocal for each channel (seeing e.g. that the green channel matches for the last color, but doesn't at the first one). Next I will investigate whether a mapping in HSV is more straightforward.

@Novakasa
Copy link

Novakasa commented May 4, 2022

I tried a linear mapping in RGB space (a 3x3 matrix) and used scipy.optimize.least_squares to try to fit the "center color" of each peak to its corresponding rebrickable-rgb-value. This is the same plot as above, but with the linear mapping applied:

image

We can see that it improves in some cases, but can't fit everything simultaneously. I think this shows that at least in RGB, the mapping has to be nonlinear.

The resulting matrix looks like this:

array([[ 1.39618714, -0.04407636, -0.18775358],
       [ 0.02527175,  1.59905428, -0.37466398],
       [-0.21313028,  0.23361669,  1.20627808]]

Since I haven't put any absolute normalization on the data, this matrix can be multiplied by an arbitrary scalar factor to compensate for RGB normalization ( = value calibration)

@Novakasa
Copy link

In pybricks/pybricks-micropython#104, I mentioned this:

There is an issue when measuring a white lego brick at the optimal distance of 2 plates, however that seems to be a bug somewhere in the conversion from raw to hsv, as I see the raw RGB values at correct values whereas the hsv values have some suspicious artifacts (saturation jumps from ~0 to ~100 with little change of the raw values).

It just came to me that that is probably due to what I talked about in the above comment #116 (comment). White at distance of 2 plates probably clips the range of 0-440, so the mapping produces values above 255, leading to an int overflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request platform: Powered Up Issues related to LEGO Powered Up software: pybricks-micropython Issues with Pybricks MicroPython firmware (or EV3 runtime) topic: sensors Issues involving sensors
Projects
None yet
Development

No branches or pull requests

2 participants