4G limit on an 8G board

[great9]

There seems to be a memory limit of 4G even though my board is 8G.
I've described it on nyanmisaka's ffmpeg-rockchip repo: nyanmisaka/ffmpeg-rockchip#51

[HermanChen]

The 32bit iova space is hardware limit. But we still can use some tick in driver to workaround this problem. We are trying.

[great9]

The 32bit iova space is hardware limit. But we still can use some tick in driver to workaround this problem. We are trying.

Thank you!

Lots of people have this issue. At one point I was considering of getting a secondary board with 16GB but then I found out more people had this problem.

[great9]

The 32bit iova space is hardware limit. But we still can use some tick in driver to workaround this problem. We are trying.

any updates?

[great9]

@HermanChen

[HermanChen]

It is not fixed yet.

[great9]

It is not fixed yet.

just pinging in the hopes of getting a message back from you saying that 4 devs are working on a fix. :)

from design perspective, why was this ever limited to 32 bit?
Vendors marketed the board as a multi-realtime-stream encoder/decoder. It's very easy to hit that limit.

[great9]

any updates @HermanChen ?

[chencl9410]

0001-DEBUG-video-rockchip-mpp-disable-CONFIG_DMABUF_CACHE.patch
To disable DMABUF_CACHE so that iova will be instantly unmaped after every encoder/decoder task finished.
Please apply this patch on kernel6.1 or kernel5.10 and recompile kernel boot and retry.
@great9

[great9]

0001-DEBUG-video-rockchip-mpp-disable-CONFIG_DMABUF_CACHE.patch To disable DMABUF_CACHE so that iova will be instantly unmaped after every encoder/decoder task finished. Please apply this patch on kernel6.1 or kernel5.10 and recompile kernel boot and retry. @great9

this patch doesn't apply with 5.10 rkr3.4 (aka the latest 5.10 from radxa)

[chencl9410]

0001-DEBUG-video-rockchip-mpp-disable-CONFIG_DMABUF_CACHE.patch To disable DMABUF_CACHE so that iova will be instantly unmaped after every encoder/decoder task finished. Please apply this patch on kernel6.1 or kernel5.10 and recompile kernel boot and retry. @great9

this patch doesn't apply with 5.10 rkr3.4 (aka the latest 5.10 from radxa)

It‘s okay to ignore the code conflict between different branch and commit.
you may just mannually copy the code of the patch to following two files.
drivers/dma-buf/Kconfig | 14 +++++++-------
drivers/video/rockchip/mpp/mpp_iommu.c | 6 +++---
then rebuild and try

[MarcA711]

Hey, did anyone successfully try out the patch above? I spent some time figuring out how to compile the kernel but I had no success.

@HermanChen @chencl9410 please prioritize this issue. It is known for a long time and not fixed yet. Please apply the fix to the BSP kernel so that it lands in all distributions. This issue is critical and the reason why I can't use my Rockchip board. I would have recommended Rockchip board to friends, but I can't recommend it with this issue. Please fix it fast.

[MarcA711]

0001-DEBUG-video-rockchip-mpp-disable-CONFIG_DMABUF_CACHE.patch To disable DMABUF_CACHE so that iova will be instantly unmaped after every encoder/decoder task finished. Please apply this patch on kernel6.1 or kernel5.10 and recompile kernel boot and retry. @great9

@HermanChen @chencl9410
the patch was tested and does not fix the issue, see Joshua-Riek/ubuntu-rockchip#1017 (comment)

[great9]

0001-DEBUG-video-rockchip-mpp-disable-CONFIG_DMABUF_CACHE.patch To disable DMABUF_CACHE so that iova will be instantly unmaped after every encoder/decoder task finished. Please apply this patch on kernel6.1 or kernel5.10 and recompile kernel boot and retry. @great9

@HermanChen @chencl9410 the patch was tested and does not fix the issue, see Joshua-Riek/ubuntu-rockchip#1017 (comment)

it would help in case of frigate...

[chencl9410]

@great9 @MarcA711
The 4GB memory limitation comes from the 32-bit iova space and 32-bit register configuration.
We’re trying another workaround solution that use IOMMU domains to isolate address spaces. The expectation is that, as long as the buffer bound to the domain is switched before the task is handed over to the hardware, this issue can be avoided. This way, we can map multiple 4GB address spaces by using multiple domains.
There are still some driver-level details that need to be debugged for this solution, and it will take some time. For example, switching domains in the driver while the hardware is running causes errors, so the process needs to be serialized.

[nyanmisaka]

The expectation is that, as long as the buffer bound to the domain is switched before the task is handed over to the hardware, this issue can be avoided. This way, we can map multiple 4GB address spaces by using multiple domains.

Thanks for confirming that, that sounds very promising. Hopefully we can work around this limitation someday.

[MarcA711]

@HermanChen @chencl9410 Sorry for asking again but some time has passed since last time.

Is there any progress? Do you know how long it will take to finally fix this awful bug?

I have been wanting to buy another SBC for some time now. I would like to buy another one with Rockchip SOC but not before this problem is finally fixed. My other Rockchip SBC crashes regularly and works unreliably because of this error. If I knew that this will be definitely fixed in the near future I can finally buy.

[chencl9410]

@MarcA711
We have confirmed the feasibility of the multi-domain solution. This solution allows each MPP session to support up to 4GB of IOVA space, instead of the current limitation where the total multi-channel setup only supports 4GB. Currently, we have preliminary support for single-core devices to switch between multiple domains and for dual-core devices to fix each core to a different domain. However, both approaches affect the maximum performance of the dual-core codec.
Our goal is to enable a dual-core collaborative multi-domain switching solution. In this process, we have encountered several asynchronous issues that need to be managed on the driver side.
For example, when a process exits, the buffer that needs to be automatically unmapped and released might not currently be bound to the MMU. In this case, it’s necessary to switch and bind the domain before unmapping. However, if the hardware is currently bound to another domain, directly switching domains could result in hardware read/write exceptions.

[MattiaGrigoli]

are there any news on this? any temporary fixes or any other thing that is a workaround for this issue?

[great9]

ping @chencl9410 @HermanChen

[chencl9410]

Workaround Solutions for MPP IOVA 4G Limitation for now:

1. Fixed Domain Solution for Dual-Core Codec:

• Assign one domain to each core, allowing up to 2x4G IOVA address space without switching domains during operation. This solution has been tested and is stable.

• Supports up to 4G for a single session and up to 8G for multiple sessions.

  patch：0001-WIP-use-seperate-domain-for-dual-core.patch

2. Domain Switching Solution for Single-Core Codec:

• Allocate 4 or 8 domains and distribute multiple MPP sessions across different domains. During task creation, map addresses from different domains. Both mapping and unmapping operations need to switch to the corresponding domain.
• Currently, due to the unmapping timing sometimes being independent of the codec task lifecycle, there are some issues in the driver management. Debugging is ongoing, and a patch cannot be provided for now. I will continue debugging this issue next week and update progress promptly.

3. Domain Switching Solution for Dual-Core Codec:

• A combination of the above solutions is being developed and processed.

4. Handling Potential Single-Session Requirements Exceeding 4G:

• Based on our evaluation, it is unlikely for general use cases to require more than 4G per session. However, certain scenarios, such as JPEG decoding of a large number of photos, may result in large buffers not being released. For such cases, we recommend opening multiple sessions in user space to achieve the desired functionality (after the multi-domain switching patch becomes available).

[nyanmisaka]

Workaround Solutions for MPP IOVA 4G Limitation for now:

Thanks for the update!

If I understand correctly, using FFmpeg as an example, each FFmpeg process corresponds to one MPP decoder session, is that correct?

If so, can we expect that multiple FFmpeg processes decoding multiple 1080p videos (e.g. > 16 times ffmpeg processes and exceeded 4GB but within 8GB in total) will not cause crashes, but will only slow down?

As far as I know the RGA hardware is also affected and it is used as post-processing (downscaling and pixel format conversion such as yuv2rgb) for the MPP decoder. RK3588 has dual-core RGA3, which doesn't require DMA32 flag. Can these improvements also be applied to the RGA3 driver?

Just in case you need an example to verify the MPP+RGA use case, this FFmpeg command can be used. And here's the Wiki page for it.

./ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga \
-i h264_1080p.avc -vf scale_rkrga=format=rgba -f null -

When the 4GB issue occurs, the return value is -12 (-ENOMEM). Tracing the RGA3 kernel driver revealed that this is also related to the IOMMU being unable to bind the required FD.

[chencl9410]

Workaround Solutions for MPP IOVA 4G Limitation for now:

Thanks for the update!

If I understand correctly, using FFmpeg as an example, each FFmpeg process corresponds to one MPP decoder session, is that correct?

If so, can we expect that multiple FFmpeg processes decoding multiple 1080p videos (e.g. > 16 times ffmpeg processes and exceeded 4GB but within 8GB in total) will not cause crashes, but will only slow down?

As far as I know the RGA hardware is also affected and it is used as post-processing (downscaling and pixel format conversion such as yuv2rgb) for the MPP decoder. RK3588 has dual-core RGA3, which doesn't require DMA32 flag. Can these improvements also be applied to the RGA3 driver?

Just in case you need an example to verify the MPP+RGA use case, this FFmpeg command can be used. And here's the Wiki page for it.

./ffmpeg -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga \
-i h264_1080p.avc -vf scale_rkrga=format=rgba -f null -

When the 4GB issue occurs, the return value is -12 (-ENOMEM). Tracing the RGA3 kernel driver revealed that this is also related to the IOMMU being unable to bind the required FD.

can we expect that multiple FFmpeg processes decoding multiple 1080p videos (e.g. > 16 times ffmpeg processes and exceeded 4GB but within 8GB in total) will not cause crashes, but will only slow down
yes. if the case is 16 times ffmpeg processes uses memory exceeded 4GB but within 8GB in total, which will causes crash before. after apply this patch, will no longer occurs.

And about slowing down performance：
The performance limitation introduced by this patch comes from restricting each session to use only a single core. As a result, the maximum frame rate for a single encoding or decoding task is no longer the combined maximum frame rate of both cores, but rather the maximum frame rate of a single core. Therefore, in scenarios such as 16-channel 1080p decoding, there will be almost no performance loss, as these tasks would originally be automatically distributed across two cores. The main impact will be on extreme cases, such as single-stream 8K encoding, where the maximum frame rate will be halved. 8K video decoding will no longer support 60fps, but instead will support 30fps (although simple video content might still be able to achieve 60fps).

As for RGA, it also requires the RGA driver to implement similar measures for handling IOVA mappings that exceed 4G. I will discuss this issue with my colleague responsible for the RGA driver. Considering that RGA is primarily used for image format conversion, it may not require the same frame buffer management as a codec, and there might be a more flexible solution for releasing buffers. Additionally, I will spend some time preparing the FFmpeg environment you mentioned for experimentation.

[nyanmisaka]

Thx again for the comment, that's very informative. Relaxing the limit to 8GB can satisfy 99% of use cases.

Additionally, I will spend some time preparing the FFmpeg environment you mentioned for experimentation.

FYI if you find building ffmpeg time consuming, you can get the pre-built portable ffmpeg binaries.

[chencl9410]

@nyanmisaka
According to feedback from the RGA developers, due to hardware limitations of the RGA2's IOMMU, it cannot use physical addresses larger than 32-bit, not just a limitation of virtual addresses. However, the RGA3 on the RK3588 supports access to physical addresses larger than 32-bit. You might want to refer to the librga developer guide and FAQ to see if related issues can be resolved there. If not, it's recommended to open an issue on the librga GitHub repository for further consultation.

[nyanmisaka]

@nyanmisaka According to feedback from the RGA developers, due to hardware limitations of the RGA2's IOMMU, it cannot use physical addresses larger than 32-bit, not just a limitation of virtual addresses. However, the RGA3 on the RK3588 supports access to physical addresses larger than 32-bit. You might want to refer to the librga developer guide and FAQ to see if related issues can be resolved there. If not, it's recommended to open an issue on the librga GitHub repository for further consultation.

I have read through the RGA documentation and know the hardware limitations of RGA2 so I disabled RGA2 and used only RGA3. But RGA3 is still affected by the IOVA 4G issue.

Here are some logs that I hope will be of interest to RGA developers. 8k(7680x4320) video is used to trigger the 4G limitation more easily. The pipeline is MPP decoder(nv12) -> RGA conversion (rgba).

 RgaBlit(1527) RGA_BLIT fail: Cannot allocate memory
 RgaBlit(1528) RGA_BLIT fail: Cannot allocate memory
handl-fd-vir-phy-hnd-format[0, 69, (nil), (nil), 0, 8192]
rect[0, 4, 7680, 4320, 7680, 4336, 8192, 0]
f-blend-size-rotation-col-log-mmu[8192, 0, 0, 0, 0, 0, 1]
handl-fd-vir-phy-hnd-format[0, 89, (nil), (nil), 0, 768]
rect[0, 0, 7680, 4320, 7680, 4320, 0, 0]
f-blend-size-rotation-col-log-mmu[768, 0, 0, 0, 0, 0, 1]
[Parsed_scale_rkrga_0 @ 0xffff80002150] RGA blit failed: -12
[vf#0:0 @ 0xaaaae5cab490] Error while filtering: Generic error in an external library
[vf#0:0 @ 0xaaaae5cab490] Task finished with error code: -542398533 (Generic error in an external library)
[vf#0:0 @ 0xaaaae5cab490] Terminating thread with return code -542398533 (Generic error in an external library)

[308978.666342] mpp_dma_import_fd:239: dma_buf_map_attachment fd 73 failed(-12)
[308978.666354] rk_vcodec: mpp_task_attach_fd:1760: can't import dma-buf 73
[308978.666356] rk_vcodec: mpp_translate_reg_address:1816: reg[130]: 0x00000049 fd 73 failed
[308978.666359] rk_vcodec: mpp_task_dump_mem_region:2003: --- dump task 0 mem region ---
[308978.666361] rk_vcodec: mpp_task_dump_mem_region:2008: reg[128]: 0x00000000e5c00000, size 200000
[308978.666364] rk_vcodec: mpp_task_dump_mem_region:2008: reg[129]: 0x00000000e5c00000, size 200000
[308978.666367] rk_vcodec: mpp_process_task_default:614: alloc_task failed.
[308978.672976] mpp_dma_import_fd:239: dma_buf_map_attachment fd 74 failed(-12)
[308978.672983] rk_vcodec: mpp_task_attach_fd:1760: can't import dma-buf 74
[308978.672985] rk_vcodec: mpp_translate_reg_address:1816: reg[130]: 0x0000004a fd 74 failed
[308978.672987] rk_vcodec: mpp_task_dump_mem_region:2003: --- dump task 0 mem region ---
[308978.672989] rk_vcodec: mpp_task_dump_mem_region:2008: reg[128]: 0x00000000ffc00000, size 200000
[308978.672992] rk_vcodec: mpp_task_dump_mem_region:2008: reg[129]: 0x00000000ffc00000, size 200000
[308978.672994] rk_vcodec: mpp_process_task_default:614: alloc_task failed.
[308978.673068] rga_dma_buf: Failed to map attachment, ret[-12]
[308978.673072] rga_mm: rga_mm_map_dma_buffer core[2] map dma buffer error!
[308978.673075] rga_mm: rga_mm_map_buffer map dma_buf error!
[308978.673076] rga_mm: job buffer map failed!
[308978.673078] rga_mm: src channel map job buffer failed!
[308978.673080] rga_mm: failed to map buffer
[308978.673082] rga_job: rga_job_commit: failed to map job info
[308978.673110] rga_job: request[2732892] task[0] job_commit failed.
[308978.673114] rga_job: rga request[2732892] commit failed!
[308978.673116] rga: request[2732892] submit failed!
[308978.697023] rk_vcodec: mpp_wait_result_default:902: session 279891:127 pending list is empty!
[308978.697028] rk_vcodec: mpp_msgs_wait:1612: session 127 wait result ret -5
[308978.697033] rk_vcodec: mpp_wait_result_default:902: session 279891:127 pending list is empty!
[308978.697035] rk_vcodec: mpp_msgs_wait:1612: session 127 wait result ret -5

[great9]

@nyanmisaka According to feedback from the RGA developers, due to hardware limitations of the RGA2's IOMMU, it cannot use physical addresses larger than 32-bit, not just a limitation of virtual addresses. However, the RGA3 on the RK3588 supports access to physical addresses larger than 32-bit. You might want to refer to the librga developer guide and FAQ to see if related issues can be resolved there. If not, it's recommended to open an issue on the librga GitHub repository for further consultation.

I have read through the RGA documentation and know the hardware limitations of RGA2 so I disabled RGA2 and used only RGA3. But RGA3 is still affected by the IOVA 4G issue.

Here are some logs that I hope will be of interest to RGA developers. 8k(7680x4320) video is used to trigger the 4G limitation more easily. The pipeline is MPP decoder(nv12) -> RGA conversion (rgba).

have you submitted this as an issue to librga devs on that repo?

[MattiaGrigoli]

0001-DEBUG-video-rockchip-mpp-disable-CONFIG_DMABUF_CACHE.patch To disable DMABUF_CACHE so that iova will be instantly unmaped after every encoder/decoder task finished. Please apply this patch on kernel6.1 or kernel5.10 and recompile kernel boot and retry. @great9

@HermanChen @chencl9410 the patch was tested and does not fix the issue, see Joshua-Riek/ubuntu-rockchip#1017 (comment)

i can confirm that on joshua's ubuntu the patch doesn't work, like it's not even there. here are some info:

• board = orange pi 5 plus
• ubuntu jammy (22.04)
• kernel = joshua 5.10 (with applied patch 0001-WIP-use-seperate-domain-for-dual-core.patch)
• 2 gstreamer streams using mppvideodec from one camera
• 2 inference on the streams (one for each stream, they use RGA for resizing the image)
  journalctl log:
  
  after this there's a kernel panic

is there really no fix for this?

[chencl9410]

0001-DEBUG-video-rockchip-mpp-disable-CONFIG_DMABUF_CACHE.patch To disable DMABUF_CACHE so that iova will be instantly unmaped after every encoder/decoder task finished. Please apply this patch on kernel6.1 or kernel5.10 and recompile kernel boot and retry. @great9

@HermanChen @chencl9410 the patch was tested and does not fix the issue, see Joshua-Riek/ubuntu-rockchip#1017 (comment)

i can confirm that on joshua's ubuntu the patch doesn't work, like it's not even there. here are some info:

• board = orange pi 5 plus
• ubuntu jammy (22.04)
• kernel = joshua 5.10 (with applied patch 0001-WIP-use-seperate-domain-for-dual-core.patch)
• 2 gstreamer streams using mppvideodec from one camera
• 2 inference on the streams (one for each stream, they use RGA for resizing the image)
  journalctl log:
  
  after this there's a kernel panic

is there really no fix for this?

These logs are prints from RGA mapping failures. I have already reported the trigger conditions for similar issues to the RGA developers. I was able to trigger the errors from both RGA and MPP by running the ffmpeg test case provided by Nyanmiska in multiple instances. After applying the patch, the MPP import fd error no longer occurs. The effectiveness of this patch can also be verified through the simplest mpi_dec_multi_test.c multi-instance test (using mpi_dec_multi_test -i /path/to/8Kvideo.h265 -s 20 -n -1) for pure decoder testing. It directly shows the effect of the patch on multi-session 8K decoding before and after the patch is applied.

[nyanmisaka]

0001-DEBUG-video-rockchip-mpp-disable-CONFIG_DMABUF_CACHE.patch To disable DMABUF_CACHE so that iova will be instantly unmaped after every encoder/decoder task finished. Please apply this patch on kernel6.1 or kernel5.10 and recompile kernel boot and retry. @great9

@HermanChen @chencl9410 the patch was tested and does not fix the issue, see Joshua-Riek/ubuntu-rockchip#1017 (comment)

i can confirm that on joshua's ubuntu the patch doesn't work, like it's not even there. here are some info:

• board = orange pi 5 plus
• ubuntu jammy (22.04)
• kernel = joshua 5.10 (with applied patch 0001-WIP-use-seperate-domain-for-dual-core.patch)
• 2 gstreamer streams using mppvideodec from one camera
• 2 inference on the streams (one for each stream, they use RGA for resizing the image)
  journalctl log:
  
  after this there's a kernel panic

is there really no fix for this?

These logs are prints from RGA mapping failures. I have already reported the trigger conditions for similar issues to the RGA developers. I was able to trigger the errors from both RGA and MPP by running the ffmpeg test case provided by Nyanmiska in multiple instances. After applying the patch, the MPP import fd error no longer occurs. The effectiveness of this patch can also be verified through the simplest mpi_dec_multi_test.c multi-instance test (using mpi_dec_multi_test -i /path/to/8Kvideo.h265 -s 20 -n -1) for pure decoder testing. It directly shows the effect of the patch on multi-session 8K decoding before and after the patch is applied.

The above is a completely different issue specific to gstreamer-rockchip, which fails because it does not allocate the buffer used by RGA2 with the DMA32 flag. This does not exist in ffmpeg-rockchip.

#define MPP_BUFFER_FLAGS_DMA32          0x00200000      //ROCKCHIP_BO_DMA32 << 16

[MattiaGrigoli]

0001-DEBUG-video-rockchip-mpp-disable-CONFIG_DMABUF_CACHE.patch To disable DMABUF_CACHE so that iova will be instantly unmaped after every encoder/decoder task finished. Please apply this patch on kernel6.1 or kernel5.10 and recompile kernel boot and retry. @great9

@HermanChen @chencl9410 the patch was tested and does not fix the issue, see Joshua-Riek/ubuntu-rockchip#1017 (comment)

i can confirm that on joshua's ubuntu the patch doesn't work, like it's not even there. here are some info:

• board = orange pi 5 plus
• ubuntu jammy (22.04)
• kernel = joshua 5.10 (with applied patch 0001-WIP-use-seperate-domain-for-dual-core.patch)
• 2 gstreamer streams using mppvideodec from one camera
• 2 inference on the streams (one for each stream, they use RGA for resizing the image)
  journalctl log:
  
  after this there's a kernel panic

is there really no fix for this?

These logs are prints from RGA mapping failures. I have already reported the trigger conditions for similar issues to the RGA developers. I was able to trigger the errors from both RGA and MPP by running the ffmpeg test case provided by Nyanmiska in multiple instances. After applying the patch, the MPP import fd error no longer occurs. The effectiveness of this patch can also be verified through the simplest mpi_dec_multi_test.c multi-instance test (using mpi_dec_multi_test -i /path/to/8Kvideo.h265 -s 20 -n -1) for pure decoder testing. It directly shows the effect of the patch on multi-session 8K decoding before and after the patch is applied.

The above is a completely different issue specific to gstreamer-rockchip, which fails because it does not allocate the buffer used by RGA2 with the DMA32 flag. This does not exist in ffmpeg-rockchip.

#define MPP_BUFFER_FLAGS_DMA32          0x00200000      //ROCKCHIP_BO_DMA32 << 16

thank you for pointing the right direction of my issue, how can i fix it with gstreamer?
where i have to apply this flag?
(where's the gstreamer-rockchip repo? it seems i can't find it)

[nyanmisaka]

Joshua-Riek/ubuntu-rockchip#1030 (comment)