10 bit colors to numpy

[johan-sightic]

Hi!
I'm reading frames from a mp4 HEVC video in yuv422p10le format, i.e. 10bit color depth, to numpy.
I get a ndarray with dtype uint16 which I expected but I thought there would only be at most 2^10=1024 different unique values in the frame, but I get about 60000 different values. Is this expected? What am I missing?
This is my code:

import av
import numpy as np

# Read hevc video in yuv422p10le format
with av.open("video.mp4") as container:
    frame = next(container.decode(video=0))
    np_frame = frame.to_ndarray(format="rgb48")
    print(container.streams.video[0].pix_fmt)  # >>> yuv422p10le
    print(np_frame.dtype)  # >>> uint16
    print(np.unique(np_frame))  # >>> [    0     1     2 ... 65498 65527 65535]
    print(len(np.unique(np_frame)))  # >>> 58702 (expected <= 1024 differnte values for yuv422p10le)

Thanks!

[WyattBlue]

The issue here is that you're converting the YUV data to RGB format with to_ndarray(format="rgb48"). This conversion process is what's causing you to see more unique values than expected.

Your original video is in YUV422p10le format, where each component indeed has 10-bit depth (0-1023 possible values)
When you convert to "rgb48" format:

• A color space conversion from YUV to RGB occurs
• Each YUV value gets transformed using the YUV->RGB conversion matrix
• These calculations create new values through interpolation
• The resulting RGB values use the full 16-bit range

To see the original 10-bit values, you should read the frame in its native YUV format, which I think is:

    np_frame = frame.to_ndarray()

[johan-sightic]

I see! Thank you!
I did not know you could get a ndarray in YUV format. Is there a list of supported formats for to_ndarray()?

yuv422p10le does not seem to be supported

frame.to_ndarray()
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "av/video/frame.pyx", line 359, in av.video.frame.VideoFrame.to_ndarray
ValueError: Conversion to numpy array with format `yuv422p10le` is not yet supported

[robinechuca]

It is not yet implemented in PyAV but you can convert the frame in yuv444p16le, then cast it as a numpy array.

import av
import numpy as np
# read in yuv422p10le format
with av.open("video.mp4") as container:
    frame_10 = next(container.decode(video=0))
    frame_16 = frame.reformat(format="yuv444p16le")
    frame_16_np = np.frombuffer(frame_16.planes[0]).view(np.uint16)  # only y componant
    print(f"there is {len(set(frame_16_np.tolist()))} differents values in the frame (<2**10)")

[johan-sightic]

Even better!

[robinechuca]

Update since the version 14.1.0, the type "yuv444p16le" is now supported.
In libav, the conversion of yuv format is just a bit shift.

import av
import numpy as np
# read in yuv422p10le format
with av.open("video.mp4") as container:
    frame_10 = next(container.decode(video=0))
    frame_16_np = frame.to_ndarray(format="yuv444p16le")
    frame_10_np = frame_16_np >> 16 - 10   # reverse bit shift