Pi4 MMAL decode of h264 stream makes noise on Headphone playback

[marsohod4you]

On Pi4 with Raspberry Buster OS with Kernel 5.4 not possible to decode h264 stream with MMAL HW without noise on Headphones during audio playback.

Reproduce is very easy.
Take https://github.com/raspberrypi/userland.git and build with script ./buildme
In output directory there are several MMAL examples, simplest is build/bin/mmalplay
Starting this app with parameter like file test.h264 BigBuckBunny plays fragment of video fullscreen.
Ok, then in separate terminal start
aplay -D hw:2,0 some-audio-8bit-22050hz.wav
Audio plays on Headphone. Then start from other terminal build/bin/mmalplay test.h264 for video playback.
When video starts then audio begins crackling and noising.

Other userland mmal examples do not show playback on screen but simply h264 MMAL decode cause noise on Headphones.
For this test use applications build/bin/mmal_example_basic_1 and build/bin/mmal_example_basic_2
issue is exactly same.

Interesting is that this issue does not happen on older raspberry kernel like 4.19
Other interesting fact is that noise happens only on headphones. Audio on any HDMI port plays good.

There are several similar complains in internet:
https://forum.kodi.tv/showthread.php?tid=349778
https://forum.libreelec.tv/thread/20854-raspberry-pi-4-audio-problems-playing-ts-streams/
But no solution found.

[pelwell]

Kernel releases have moved on to 5.10 now - does that also crackle?

Also, what config.txt settings are active? Report the output of grep -v -E "^(#.*)?$" /boot/config.txt (or perhaps grep -v -E "^(#.*)?$" /flash/config.txt).

[6by9]

When creating an issue you should have been presented with a template that requested a load of basic system information, ideally the output from raspinfo. You haven't provided any information requested, and specifically the exact kernel (uname -a) and firmware (vcgencmd version) versions being used.

https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=195178
The headphone output uses a sigma delta noise shaping algorithm running as part of the firwmare.
Video decoder also runs within the firmware.
If there is contention on the hardware (normally only from deinterlacing or copying pixel buffers), then the audio will drop frames and crackle.

There was a change in that area at the end of October. Without knowing versions there is no point in continuing the discussion as we don't know if you are testing before or after that change.

[marsohod4you]

hi.
I have just upgraded my Raspberry Pi4 from kernel 5.4 to latest.
Now my version is
uname -a
Linux raspberrypi 5.10.11-v7l+ #1399 SMP Thu Jan 28 12:09:48 GMT 2021 armv7l GNU/Linux

Other:

vcgencmd version
Jan 27 2021 22:19:57
Copyright (c) 2012 Broadcom
version 99d9a48302e4553cff3688692bb7e9ac760a03fa (clean) (release) (start)

Attached file is output of raspinfo
raspinfo.txt

Issue still remains after apt update/upgrade to 5.10.
Just want to add some comments.

1. I am writing my own app for decoding video with MMAL HW and faced this issue
2. to be sure that this is not my bug now I use for tests only pure apps from https://github.com/raspberrypi/userland.git and I see exactly this same issue with crackling audio so I decide this is not my bug
3. I run independent processes in different terminals for audio playback aplay -D hw:2,0 sound.wav and for video decode userland/build/bin/mmalplay test.h264
   This demonstrates issue 100% reproducable. Stream test.h264 is also taken from userland directories and this stream itself is pure h264 it does not have any audio inside.
4. I was thinking issue is video rendering on screen but it is not. Other userland examples like mmal_example_basic_1 or mmal_example_basic_2 demonstrate same crackling sound on headphones but those apps do not render video on screen

[6by9]

Converting the buffer from the internal format / MMAL_ENCODING_OPAQUE to an externally defined pixel format will be the main hit on the VPU, and the likely cause of interaction.

Both mmal_example_basic examples convert to the default encoding, which is likely to be MMAL_ENCODING_I420 so will have issues.
mmalplay has a format override, so mmalplay -f opaque -fr opaque foo.h264 should force it to opaque, and that shouldn't have any issue (no conversions required).

[marsohod4you]

heh..
Probably You know what You say and You are right, but this does not give me understanding what to do with that next.
I need raw I420 frames from h264 stream for my further processing in my app, but I cannot get without audio issue.

[P33M]

Correct me if I'm wrong, but aren't mmalplay/mmal_example just throwing frames at the decoder as fast as possible? BBB is playing "fast". If you're maxing out the VPU then all bets are off, unless there's a specific regression from firmware version Hexxeh/rpi-firmware@e8ddac7 and onwards.

[marsohod4you]

Yes, mmalplay plays very fast. Probably as fast as it can. And this is what I need really in my own app - decode as ast as possible.
When I use command mmalplay -f opaque foo.h264 then NO audio distortion. But I do not know what this gives me. Do I can access to raw pixels? What is format of those pixel planes?

[6by9]

mmalplay -f opaque foo.h264 sets decoder->output[0]->format to MMAL_ENCODING_OPAQUE..
The buffer contents are a GPU-only handle to an underlying buffer. It is not possible to access the pixel data from the ARM.

Without knowing some more about your use case it is impossible to advise further on how to balance the load.

[6by9]

The other approach is you add audio_pwm_mode=1 in /boot/config.txt to disable the SDM algorithm. The audio quality will be lower.

[marsohod4you]

Is it possible to render to screen or dispmanx surface only some specific rectangular part of decoded frame?
But each decoded frame may have dynamically different specific list of draw output rectangles.

[6by9]

Yes, you set the source cropping rectangle.

For MMAL you set it on the video render component.

      MMAL_DISPLAYREGION_T param = {0};
      param.hdr.id = MMAL_PARAMETER_DISPLAYREGION;
      param.hdr.size = sizeof(MMAL_DISPLAYREGION_T);

      param.set |= (MMAL_DISPLAY_SET_DEST_RECT;
      param.dest_rect.x = TOP_LEFT;
      param.dest_rect.y = TOP_RIGHT;
      param.dest_rect.width = WIDTH;
      param.dest_rect.height = HEIGHT;

      status = mmal_port_parameter_set(render->input[0], &param.hdr);
      if (status != MMAL_SUCCESS && status != MMAL_ENOSYS)
      {
         vcos_log_error("unable to set preview port parameters (%u)", status);
      }

Memory says it is as a 16.16 fixed point representation to allow fractional positioning.

DispmanX won't take MMAL_ENCODING_OPAQUE directly, but the same porperty is available for any resources it uses. In vc_dispmanx_element_add you have the prototype

VCHPRE_ DISPMANX_ELEMENT_HANDLE_T VCHPOST_ vc_dispmanx_element_add ( DISPMANX_UPDATE_HANDLE_T update, DISPMANX_DISPLAY_HANDLE_T display,
                                                                     int32_t layer, const VC_RECT_T *dest_rect, DISPMANX_RESOURCE_HANDLE_T src,
                                                                     const VC_RECT_T *src_rect, DISPMANX_PROTECTION_T protection, 
                                                                     VC_DISPMANX_ALPHA_T *alpha,
                                                                     DISPMANX_CLAMP_T *clamp, DISPMANX_TRANSFORM_T transform );

src_rect specifies the cropping of the source image.

[marsohod4you]

vc_dispmanx_element_add() works only once when element is added. So I cannot use src_rect at this moment.
I need to render from 1st frame one rectangle region, from second frame another rectangle region, from 3rd frame maybe several rectangles.
All rendering happens one over previous.
What You propose seem allows crop only once at init phase.
But I need different crop after every frames.

[6by9]

DispmanX requires raw pixel buffers - it can not work with MMAL_ENCODING_OPAQUE. That conversion is going to affect your audio.

You look a little further down the header file and note https://github.com/raspberrypi/userland/blob/master/interface/vmcs_host/vc_dispmanx.h#L108

VCHPRE_ int VCHPOST_ vc_dispmanx_element_change_attributes( DISPMANX_UPDATE_HANDLE_T update, 
                                                            DISPMANX_ELEMENT_HANDLE_T element,
                                                            uint32_t change_flags,
                                                            int32_t layer,
                                                            uint8_t opacity,
                                                            const VC_RECT_T *dest_rect,
                                                            const VC_RECT_T *src_rect, <<<<<<<<<<
                                                            DISPMANX_RESOURCE_HANDLE_T mask,
                                                            DISPMANX_TRANSFORM_T transform );

All rendering happens one over previous.

You're trying to create a video mosaic type effect?

DispmanX and MMAL expect to have all resources available for the render, and then render all the pixels in the output buffer - it's not a simple blit.

For offline rendering there's a possibility that you could include the output buffer as a source in the render list at 1:1 resolution, but I couldn't guarantee it will work as it depends on any prefetching working.
If you use the vc.ril.hvs component (which does offline rendering and is used by the VLC in RaspiOS for OSD and subtitles) then it allows you to compose up to 5 planes together into an output buffer, with whatever parameters (src_rect, dest_rect, layer, alpha, etc) you wish for on each plane. That's probably the easiest route to trying anything out.
Output is generated each time a buffer is fed into input[0] and an output buffer is available, so you want to use input[0] for your background/output layer.
Set up the parameters for your new video frame (which can be MMAL_ENCODING_OPAQUE) on input[1] and supply the new video frame to input[1]. Then pass the output buffer to be updated in to input[0] and it should produce the output frame.

You want to be using MMAL_PARAMETER_ZERO_COPY set on input[0] and output[0] otherwise each buffer submission involves a copy to/from gpu_mem, but it does mean you need to do a bit of messing with buffer pointers being the vc_handles from a vcsm allocated pool of buffers.

[marsohod4you]

Is it possible to get description of OPAQUE image format? How to acess it by CPU?
I believe it should be possible to use opaque image handle to mmap opaque surface content into user process space like it is possible to mmap dispmanx surface content to user space with mailslot API.
I understand that internal format can be complex but anyway I need it because no other way out.
Maybe I could use NEON instructions to convert Opaque surface pixels into I420 format?
This is my last chance because seem GPU conversion makes audio crackle issue on headphones on recent kernels.

I tried all possible implementations with connecting MMAL_COMPONENT_DEFAULT_VIDEO_DECODER to MMAL_COMPONENT_DEFAULT_VIDEO_RENDERER. Yes, when using opaque format between components then no audio crackle issues on headphones.
But video renderer is doing not what I need.
Video renderer always displays FULL frame over previous. But I need to take from each frame only part (rectangle or list of rectangles) of frame and I need to paint those parts over but not fully overlapping previous paintings.
video renderer does not allow to do this.

[6by9]

If you really think it helps, then the struct is

typedef struct MMAL_OPAQUE_IMAGE_REF_T
{
   uint32_t pool_handle;  /**< Handle to the pool the image is coming from */
   uint32_t image_handle; /**< Handle to the image within the pool */
   uint16_t crop_width;   /**< visible width in pixels */
   uint16_t crop_height;  /**< visible height in pixels */
   MMAL_OPAQUE_BUFFER_FRAME_TYPE_T frame_type;  /**< Decoded frame type */
   MMAL_OPAQUE_BUFFER_INTERLACE_T interlace;    /**< Interlace setting */
   uint32_t frame_interval;                 /**< Time since last frame. If top bit clear then ns. If set, then frames/sec as 30..16/15..0 */
   uint32_t aspect_x;
   uint32_t aspect_y;
   uint16_t crop_x_org;
   uint16_t crop_y_org;
   uint32_t mean_qp;
   uint32_t FirstCorruptMB; /**< Location of first corrupted macroblock */
   uint32_t TotalCorruptMBs;
} MMAL_OPAQUE_IMAGE_REFERENCE_T;

typedef struct MMAL_OPAQUE_IMAGE_T
{
   /**< Reference to the image */
   MMAL_OPAQUE_IMAGE_REFERENCE_T image;
   /** Reference to a low resolution version of the image */
   MMAL_OPAQUE_IMAGE_REFERENCE_T image_low;

   uint32_t stripe_offset; /**< offset into stripe buffer (lines) */
   uint32_t stripe_height; /**< height of this stripe (lines) */
   uint32_t release_lines; /**< total number of lines finished with by destination */
} MMAL_OPAQUE_IMAGE_T;

/** Layout of an opaque buffer */
typedef struct MMAL_OPAQUE_BUFFER_T
{
   uint32_t host_side_reserved;  /**< Reserved for use on the host side */
   uint32_t magic;    /**< Special magic value used to tag opaque buffers */
   uint16_t flags;    /**< Flags describing properties of an opaque buffer (see
                           \ref opaquebufferflags "Opaque buffer flags") */
   uint16_t sequence; /**< Sequence number. Only used for debug */

   MMAL_OPAQUE_BUFFER_TYPE_T type; /**< Type of this opaque buffer */

   /** Union of the different types */
   union
   {
      MMAL_OPAQUE_IMAGE_T image; /**< Image type */
   } u;

   /** True if this buffer has acquired a ref to the image(s). This is only ever
    * updated when ownership is acquired or released, so locking on it should not
    * be needed.
    */
   MMAL_BOOL_T owns_image;

} MMAL_OPAQUE_BUFFER_T;

(I've removed a couple of enums as they don't add anything)

pool_handle and image_handle are internal handles that are not exposed outside the GPU. There is no way for the ARM to look up the address for those images, therefore there is no way for it to access the pixel data. It is an opaque image type.

I'd already described one potential route to get the hardware to do your weird composition.
The other approach is to use vc.ril.isp or vc.ril.hvs to do the conversion from OPAQUE to I420 for you. That removes the load from the VPU that is also running the SDM audio algorithm.

[6by9]

Closing as the duplicate question has been asked on the forums, and I'm not chasing two identical threads.
https://www.raspberrypi.org/forums/viewtopic.php?f=38&t=305082