allow setting of legacy pattern for charuco

[mssc89]

App doesnt seem to detect any points on my charuco board. Board wasn't created by this app but follows 4x4_250 charuco dictionary with A0 dimensions and 10x14 grid. I've tried with two different cameras, just to make sure that video file or colors weren't the issue. No points appear on the frames, clicking "add grid" button results in "Not enough points....grid not added..." error in the console. Autocalibration throws "Waiting for sufficient calibration boards to become populated at port 1" for each frame. I've verified that the board and video files are fine by writing my own simple test program, corners and markers are detected without any issues. What might be the issue here?

[mprib]

Check the config.toml where the charuco board is defined. By default the dictionary is the 4x4_50. Hopefully if you update that parameter in the charuco definition things should flow through. Please let me know if this resolves the issue!

[charuco]
columns = 4
rows = 5
board_height = 11.0
board_width = 8.5
dictionary = "DICT_4X4_50"
units = "inch"
aruco_scale = 0.75
square_size_overide_cm = 5.4
inverted = true

[mssc89]

I have already modified the config appropriately. This is its full contents:

camera_count = 1
creation_date = 2024-10-20T22:19:10.125729
fps_sync_stream_processing = 100
save_tracked_points_video = true

[cam_1]
port = 1
rotation_count = 3
error = "null"
matrix = "null"
distortions = "null"
translation = "null"
rotation = "null"
exposure = "null"
grid_count = "null"
size = [4608, 2592]

[charuco]
columns = 14
rows = 10
board_height = 84.1
board_width = 99.99
dictionary = "DICT_4X4_250"
units = "cm"
aruco_scale = 0.75
square_size_overide_cm = 8.0
inverted = false

With another problem I've encountered, board_width is capped to 99.99 - when launching app with config value exceeding this number, it gets capped anyway (not enough for 1189mm of a0 format). Dimensions shouldn't be a problem when it comes to just detecting points, should they?

[mprib]

Thank you for bringing the value cap issue to my attention. I've updated a separate issue related to that input box so that I can address this at the same time: #641

My recollection is that the board dimensions should not be influencing the point detection. That is there primarily to aid with printing. The square size override is there to help with the world scale.

Back to your Issue....

One other thought is that it is the aruco_scale may be throwing things off. In the raw charuco board definition, the scale is used to set the aruco_length parameter based on the square_length (inferred from the board dimensions or the override):

caliscope/caliscope/calibration/charuco.py

Line 82 in 9748eab

def board(self):

I hope to have more time tomorrow/later this week to look at this. If you are comfortable sharing one of your calibration video files it would be helpful for my troubleshooting/iterating.

[mprib]

@mssc89,

I just merged in some changes and published to PyPI (v0.5.2). I have made some updates that should hopefully resolve the board size restrictions you encountered and make the GUI run a little more smoothly.

I continue to suspect that the aruco_scale parameter is the thing that needs to be modified in the config to get the board recognized. This value is just the aruco_length/square_length from the cv2.aruco.CharucoBoard call that was used to create the board you are using.

Long ago I had played around with changing the relative size of the aruco marker relative to the solid squares and settled on a 0.75 ratio as a sensible default that seemed to have good recognition. I'm curious to learn what your board is using.

Please let me know if this resolves your issue. If not, I'm still open to troubleshooting it, though it would be easier even with just a single frame or screenshot of the charuco you are using.

[mssc89]

Thanks for quick reply. When it comes to aruco_scale, as my aruco markers are 60mm and squares are 80mm, 0.75 should work fine (another problem I've encountered, aruco_scale value set in config gets overwritten with 0.75 when launching the app, so I can't test other values). I've tried playing around with other config values but no luck.

I attach 9 seconds of my calibration video:

port_1.mp4

[mprib]

Thank you for the video file! When I use the configuration above (10 rows, 14 columns, DICT_4X4_250, aruco_scale = 0.75) this is the board generated by opencv that Caliscope is looking for (and doesn't match the board in the video):

Could you share the code that you used to define the board such that the points were successfully identified in your test script?

Just to provide some additional context, the board dictionary name in the config is ultimately used to map to a cv2 dictionary as shown in the snippet from charuco.py:

ARUCO_DICTIONARIES = {
    "DICT_4X4_50": cv2.aruco.DICT_4X4_50,
    "DICT_4X4_100": cv2.aruco.DICT_4X4_100,
    "DICT_4X4_250": cv2.aruco.DICT_4X4_250,
    "DICT_4X4_1000": cv2.aruco.DICT_4X4_1000,
    "DICT_5X5_50": cv2.aruco.DICT_5X5_50,
    "DICT_5X5_100": cv2.aruco.DICT_5X5_100,
    "DICT_5X5_250": cv2.aruco.DICT_5X5_250,
    "DICT_5X5_1000": cv2.aruco.DICT_5X5_1000,
    "DICT_6X6_50": cv2.aruco.DICT_6X6_50,
    "DICT_6X6_100": cv2.aruco.DICT_6X6_100,
    "DICT_6X6_250": cv2.aruco.DICT_6X6_250,
    "DICT_6X6_1000": cv2.aruco.DICT_6X6_1000,
    "DICT_7X7_50": cv2.aruco.DICT_7X7_50,
    "DICT_7X7_100": cv2.aruco.DICT_7X7_100,
    "DICT_7X7_250": cv2.aruco.DICT_7X7_250,
    "DICT_7X7_1000": cv2.aruco.DICT_7X7_1000,
    "DICT_ARUCO_ORIGINAL": cv2.aruco.DICT_ARUCO_ORIGINAL,
    "DICT_APRILTAG_16h5": cv2.aruco.DICT_APRILTAG_16h5,
    "DICT_APRILTAG_25h9": cv2.aruco.DICT_APRILTAG_25h9,
    "DICT_APRILTAG_36h10": cv2.aruco.DICT_APRILTAG_36h10,
    "DICT_APRILTAG_36h11": cv2.aruco.DICT_APRILTAG_36h11,
}

This is the dictionary that is ultimately used to create the charuco board in charuco.py:

        return cv2.aruco.CharucoBoard(size=(self.columns, self.rows),
                                      squareLength= square_length,
                                      markerLength= aruco_length,
                                      dictionary= self.dictionary_object,
        )

Thank you!

[mssc89]

I see, I've generated my board using calib.io, not opencv own functions. Seems like calib.io is using the "legacy" generator which starts board with a marker instead of a chessboard when row count is even, as opencv did until release 4.6.0. They hid this functionality behind setLegacyPattern() in newer releases
(opencv/opencv#23152). Can you see those boards being supported by caliscope?

[mprib]

Ah, excellent! Seems we've tracked down the heart of the problem and it doesn't look like a difficult fix.... I'm renaming this thread and going to see if I can make a quick fix.

Are you pulling your code from github or just downloading from PyPI?

[mssc89]

I'm using PyPI

[mprib]

Cool. I think I have it working. The latest version is now published to PyPI as 0.5.3, though with the way these things go, I probably broke something else while I was at it.

At the very least, your board is recognized:

Note that you need to add legacy_pattern=true to the charuco parameters in the config.toml. Also, note that this board is 10 columns and 14 rows (not 14 rows and 10 columns):

[charuco]
columns = 10
rows = 14
board_height = 60.1
board_width = 99.99
dictionary = "DICT_4X4_250"
units = "cm"
aruco_scale = 0.75
square_size_overide_cm = 8.0
inverted = false
legacy_pattern = true

Please let me know how it goes!

[mssc89]

Thank you, I’ve tested the new version and it works as expected. I have two additional questions for you, I didnt want to open additional issues for them:

1. What are your hands-on recommendations for calibration videos? I’ve followed scarce tutorials found on internet and my results are suboptimal (reprojection error is between 5-20 depending on the video).
2. Did you consider using deepcharuco instead of opencv implementation? From little testing I did, it shown to be more robust in difficult lighting and angle conditions.

[mprib]

Glad to hear it's working!

1. What are your hands-on recommendations for calibration videos? I’ve followed scarce tutorials found on internet and my results are suboptimal (reprojection error is between 5-20 depending on the video).

[ edit: reflecting on this I'm not really sure what you mean by error between 5-20. I got a RMSE of 0.582 from the video you sent me. Could you clarify what you mean?]

I'd have two recommendations after viewing your sample calibration video. First, have the board shown from multiple distances. Second, have the board shown from multiple angles. You want the foreshortening to be quite apparent.

You may also benefit from having the camera fixed and slowly moving the board in front of it (held at an angle and back and forth through a range of distances). Motion blur is your enemy.

2. Did you consider using deepcharuco instead of opencv implementation? From little testing I did, it shown to be more robust in difficult lighting and angle conditions.

I wasn't aware of this project and am glad that you've put me in the loop. I definitely see the value in this for allowing lower exposure/less blurred calibration images. If there was a simple plug and play solution for this I'd definitely try to integrate it in the near term. Is there a specific implementation you would recommend?

[mssc89]

The video I attached here was just a quick test recording, I've refined my methods since. Currently I have the camera fixed and try to move the board around with varying distance and angles - these "refined" videos result in pretty high RMSE/reprojection error, even for cameras without significant distortions. I will try to re-record them with your tips in mind, do you by any chance have a good calibration video at hand so I can follow more closely?

When it comes to deepcharuco, I've briefly tested this implementation as it seems to be the only publicly available one. it also supports MPS on mac systems out of the box and has pretty good inference instruction straight in the readme. edit: the only problem with it, it's trained on particular board configuration, and only one model based on the original paper was pretrained.

[mprib]

I had edited my question so I'm not sure if you'd read that. I'm still not sure what is meant by a reprojection error "between 5-20".

I got 0.582 from your rough cut video so I'm hoping that you get better than that when the camera is stable and foreshortened views are available.

It's also important to consider that the reprojection is measured in pixels and you have a video resolution of 4608x2592. That's a lot of pixels....the spatial error of half a pixel at that resolution is going to be less than the spatial error associated with some 1280p image.

---

Thanks for the deepcharuco reference. It's a project that I will follow, though likely won't fold in over the near-term.

---

edit: I'm going to go ahead and mark this as closed. If you have any additional questions it would be better to raise it as a discussion just to keep the threads of conversation better sorted for posterity/future reference. Thanks for your feedback on this project.