How to calibrate your camera for in sync recording

This page describes how to test your camera is suitable to be used for WAVE audio testing. For §8 audio-only tests and §9 combined audio and video tests the camera is required to record audio and video jointly and the recording must be in sync. There should not be a large offset in the joint recording as this might cause false negative results. Please follow these instructions to test your camera before running the test suite, and it is important to repeat this test if camera settings/set-up are changed.

Pre-requisites

• Ensure you have followed the audio set-up instructions here.
• To test that the camera can capture audio and video in-sync you will need:
  • A/V sync test media with “flashes” and “beeps” available to download from: test media
  • The following tools installed on you PC:
    • ffmpeg – to process the video to individual frames.
    • MediaInfo – to check the precise frame rate of the recording.
    • ffplay – for playing the test media. This is part of ffmepg package, installing ffmpeg will be default install ffplay.
    • Sonic Visualizer – audio processing tool to detect the time at which the beeps occur.
• Ensure that no other programs are running on the PC while playing the test media. This helps prevent any synchronization issues that might be caused by the media player or the PC's performance.
• Set the audio volume of the PC to a high level to ensure clear recording through its wired output.
• Connect the camera to a power supply or ensure it is fully charged (if running on battery).

Record test media for calibration

• Play the A/V sync test media on the PC using ffplay and record it with the camera.
• Check the recording using ffplay and MediaInfo to ensure what has been captured is correct i.e.:
  • Contains audio recorded jointly with video.
  • Video is clearly captured at around 120fps and 1080p Full HD.
  • Audio is 48kHz and 16-bit.

⚠️ You must use the same method of audio connection from the PC playing the A/V sync test media to the camera as you will use for recording the actual WAVE tests from the Device Under Test (DUT) to the camera e.g. if you use a 3.5mm audio cable to record the A/V sync test, you must use a 3.5mm cable to record audio from the DUT to the camera.

Automated calibration

Running calibration script

camera_calibration_helper.py is a script that automates the manual camera calibration process. It takes a test media recording as an input and generates calibration information such as PASS|WARNING|ERROR, and the average offset in milliseconds. Run the script as follows:

cd device-observation-framework
python .\camera_calibration_helper.py --calibration <file>

e.g.:

python .\camera_calibration_helper.py --calibration D:\device-observation-framework-main\calibration_recording_file_name.mp4

Configuration

If there are issues detecting flashes and beeps correctly, the thresholds can be adjusted for better results. The calibration thresholds can be adjusted in the config.ini file [CALIBRATION] section.

e.g.:

# Flash threshold (adjust if need)
flash_threshold = 150
# audio beep threshold (adjust if need)
beep_threshold = 0.3

Manual calibration

Manual checks are useful to debug when camera_calibration_helper.py is unable to detect flashes and beeps correctly.

Process video

Follow the following steps to process video to check where the flashes are captured and calculate time in recording. Ignore the flash at the beginning of the recording (at 0 second in test media), because some players might have issue to playback the first flash correctly.

• Use ffmpeg to extract the frames in the recording to images (png format) with the following command:

    ffmpeg -i .\<file_name>.MP4 -vsync 0 %04d.png

• Open the file explore that contains extracted images and locate to the images where "flash" is captured. When the "flash" is captured, the image changes from black to grey to bright white then back to grey to black.
• Find the first image with grey and note the file name (frame number) e.g. 234 in the capture below.
• Repeat for at least 3 flashes in consecutive order (ensuring you do not miss any), each time noting the starting "grey" frame number.
• Calculate "flash" start time in recording
  • Find the exact frame rate of the recording to be able to make correct calculation (as some cameras might not capture at exactly 120 fps (e.g. 119 fps, 119.8 fps…).
    • record_frame_duration_in_ms = 1000 / frame_rate
    • flash_time = (frame_num – 1) x record_frame_duration_in_ms
  • In the example recording, the camera frame rate is 119.880 fps (120000/1001 fps) and the frame number when the flash starts is 234:
    • record_frame_duration_in_ms = 1000 / (120000/1001) = 1001 / 120
    • flash_time = (234-1) X 1001 / 120 = 1944ms

NOTE: Perform this calculation for each of the flashes identified in step 1.

Process audio

Follow the following steps to process audio to check where the beeps are captured and calculate time in recording. Ignore the beep at the beginning (at 0 second in test media), because some players might have issue to playback first beep correctly at the beginning.

• Open the recording file in Sonic Visualizer.
• Process only the left channel which is in the top half of the Sonic Visualizer window (see next slide).
• Find the start of the audio wave for the "beep" and note down the time.

NOTE: You can zoom in to see the exact time.

• Repeat for at least 3 beeps in consecutive order (ensuring you do not miss any), each time noting the start time.

Calculate offset

• Finally, calculate the offset between the beep time (from step 3) and the flash time (from step 2.b.).
  • offset (ms) = beep_time - flash_time
• The offset should be within the tolerance [-120,40]ms.
• Repeat it for at least 2 more flash and beep pairs in consecutive order.
• In the example:
  • Offset_1 = 1944 – 1971 = -27ms (within tolerance)
  • Offset_2 = 2911 – 2961 = -50ms (within tolerance)
  • Offset_3 = 3912 – 3951 = -39ms (within tolerance)
  • … …

Pass or Fail?

• Pass or Fail is determined based on the assumption that the player and device are not introducing any offset.
• Automated calibration will give detailed information of PASS|ERROR|WARNING with an explanatory message.
• The first and last flash/beep pairs can be ignored, as some players or PCs may have issues with the beginning and end of playback.
• The average offset must be within the maximum tolerance of ±200ms. If the offset exceeds this tolerance, the camera cannot be used.
• The average offset should be within a small tolerance of ±20ms for the camera to be considered good.
• An average offset between ±20ms and ±200ms is acceptable for the camera, as long as the offset is applied to the observation framework.
• If not all flash/beep pairs are captured together (e.g., missing flashes or beeps), the camera can still be used, but it may cause issues.
• If your camera fails:
  • Play the recording back to check if there are any system sounds (notifications from emails etc.) recorded at the point of failure.
  • We suggest to repeat the calibration test multiple times before deeming the camera unsuitable for running the WAVE tests. Playing the A/V sync test content on another PC/device (a TV can be used) or with another player will help to rule out the device/player as the cause of offset failure.
  • Check the camera status, such as ensuring the battery is not low, and adjust settings as needed, for example turn off any unnecessary features.
  • If the same camera fails multiple attempts, then it cannot be used as it will cause false results from the Observation Framework.

IMPORTANT

Once the camera is calibrated, save the recording and pass it to the Observation Framework by adding --calibration <file> when running the process. This will apply the calculated offset to the framework.

Ensure that the camera settings/set-up do not change while using it to capture the WAVE tests. If any changes do occur, re-calibration is required.