QEMU - Increase TA memory size

[vunh91]

Please help!
I need to develop a TA with memory size up to 100 MB.
PLATFORM=vexpress-qemu_virt

I tried to increase the TA memory size as below:

1. Increased PGT_CACHE_SIZE in /optee_os/core/arch/arm/include/mm/pgt_cache.h
   #define PGT_CACHE_SIZE ROUNDUP(CFG_NUM_THREADS * 2, PGT_NUM_PGT_PER_PAGE)
   changed to:
   #define PGT_CACHE_SIZE ROUNDUP(CFG_NUM_THREADS * 130, PGT_NUM_PGT_PER_PAGE)
2. Decreased TTBCR_N_VALUE in in /optee_os/core/arch/arm/mm/core_mmu_v7.c to the smallest possible
   #define TTBCR_N_VALUE 7
   changed to:
   #define TTBCR_N_VALUE 5
3. Tried to increase SECRAM_SIZE in /optee_os/core/arch/arm/plat-vexpress/platform_config.h
   #define SECRAM_SIZE 0x01000000
   changed to:
   #define SECRAM_SIZE 0x02000000
4. Tried to change memory size in user_ta_header_defines.h to:
   #define TA_STACK_SIZE (2 * 1024 * 1024)
   #define TA_DATA_SIZE (10 * 1024 * 1024)
   Then it works fine. But when i changed the total of memory size greater than 12MB
   #define TA_STACK_SIZE (2 * 1024 * 1024)
   #define TA_DATA_SIZE (11 * 1024 * 1024)
   i got errors as below:

D/TC:0 abort_handler:669 [abort] abort in User mode (TA will panic)
E/TC:0
E/TC:0 User TA data-abort at address 0xc (translation fault)
E/TC:0 fsr 0x00000805 ttbr0 0x0e0c606a ttbr1 0x0e08006a cidr 0x1
E/TC:0 cpu #0 cpsr 0x60000030
E/TC:0 r0 0x00000000 r4 0x00400780 r8 0x00000000 r12 0x0e0cbed8
E/TC:0 r1 0x00000000 r5 0x00402bf7 r9 0x00000000 sp 0x00400708
E/TC:0 r2 0x00400780 r6 0x00000000 r10 0x00000000 lr 0x00402961
E/TC:0 r3 0x00000000 r7 0x00000000 r11 0x00000000 pc 0x00402960
E/TC:0 Status of TA a29a07bc-f12e-11e7-8c3f-9a214cf093ae (0xe07f910) (active)
E/TC:0 arch: arm load address: 0x401000 ctx-idr: 1
E/TC:0 stack: 0x100000 3147776
E/TC:0 region 0: va 0x100000 pa 0xed12000 size 0x301000 flags rw-
E/TC:0 region 1: va 0x401000 pa 0xe300000 size 0xc000 flags r-x
E/TC:0 region 2: va 0x40d000 pa 0xe30c000 size 0x3000 flags r--
E/TC:0 region 3: va 0x410000 pa 0xe30f000 size 0xa03000 flags rw-
E/TC:0 region 4: va 0 pa 0 size 0 flags ---
E/TC:0 region 5: va 0 pa 0 size 0 flags ---
E/TC:0 region 6: va 0 pa 0 size 0 flags ---
E/TC:0 region 7: va 0 pa 0 size 0 flags ---
E/TC:0 Call stack:
E/TC:0 0x00402960
E/TC:0
E/TC:0 Core data-abort at address 0x10 (translation fault)
E/TC:0 fsr 0x00000005 ttbr0 0x0e0c606a ttbr1 0x0e08006a cidr 0x1
E/TC:0 cpu #0 cpsr 0x200001ff
E/TC:0 r0 0x00000008 r4 0x0040e874 r8 0x00000000 r12 0x0e0cbed8
E/TC:0 r1 0x0040e698 r5 0x00402bf7 r9 0x00000000 sp 0x0e0c8670
E/TC:0 r2 0x00000014 r6 0x00000000 r10 0x00000000 lr 0x0e016cef
E/TC:0 r3 0x00000010 r7 0x0e0c8678 r11 0x00000000 pc 0x0e016d30
E/TC:0 Call stack:
E/TC:0 0x0e016d30
E/TC:0 0x0e016fdb
E/TC:0 0x0e0170ad
E/TC:0 0x0e017141
E/TC:0 0x0e013e15
E/TC:0 0x0e014147
E/TC:0 0x0e01418d
E/TC:0 0x0e0144e1
E/TC:0 0x0e001f50
E/TC:0 Panic '[abort] abort in abort handler (trap CPU)' at core/arch/arm/kernel/abort.c:610 <get_fault_type>
E/TC:0 Call stack:
E/TC:0 0x0e017151
E/TC:0 0x0e0285b3
E/TC:0 0x0e0021ef
E/TC:0 0x0e0144a5
E/TC:0 0x0e001f50

[jenswi-linaro]

There isn't enough SECRAM defined in QEMU to support TAs of 100MB. Even if there where enough memory it would still be a challenge. This is low level stuff that will break in mysterious ways, too hard to try to diagnose in an issue.

[vunh91]

Thank you very much for your help.
Could to you please tell me what is the maximum SECRAM defined in QEMU to support TAs?
I will try to optimize my app with smallest RAM possible.

[jenswi-linaro]

https://github.com/qemu/qemu/blob/8e5dc9ba49743b46d955ec7dacb04e42ae7ada7c/hw/arm/virt.c#L145 says that VIRT_SECURE_MEM is 0x01000000 bytes. This is shared between OP-TEE OS and the TAs. OP-TEE OS typically takes 1MiB and leaves the rest for TAs.

[vunh91]

Thank you for your help @jenswi-linaro

[ademyankov]

@vunh91 did you solve the issue w/ TA memory limit? I am having the same trouble. My TA grew over 3Mb in size and OPTEE fails to load it on v8arm qemu.

Those the parameters I tried to change but nothing seems to help:

+++ b/include/plat/arm/common/arm_def.h
@@ -71,7 +71,7 @@
-#define ARM_TZC_DRAM1_SIZE            ULL(0x01000000)
+#define ARM_TZC_DRAM1_SIZE            ULL(0x02000000) /* 32Mb */

+++ b/plat/qemu/include/platform_def.h
@@ -71,7 +71,7 @@
-#define SEC_DRAM_SIZE                 0x00f00000
+#define SEC_DRAM_SIZE                 0x01e00000 /* ~32Mb */

+++ b/core/arch/arm/plat-vexpress/platform_config.h
@@ -176,7 +176,7 @@
-#define SECRAM_SIZE            0x01000000
+#define SECRAM_SIZE            0x02000000 /* 32Mb */

@@ -211,7 +211,7 @@
-#define SECRAM_SIZE            0x01000000
+#define SECRAM_SIZE            0x02000000 /* 32Mb for emu */

+++ b/hw/arm/virt.c
@@ -142,7 +142,7 @@
-    [VIRT_SECURE_MEM] =         { 0x0e000000, 0x01000000 },
+    [VIRT_SECURE_MEM] =         { 0x0e000000, 0x02000000 }, /* 32Mb for Secure Memory */

Understandably, it's not a trivial task to change the memory layout but yet it could have been easier to do so, in my opinion.

@jenswi-linaro any tips/advice you can share?

[jenswi-linaro]

Loading a single TA of 3Mb into memory shouldn't be a problem. Perhaps you're loading more than one?

In OP-TEE it should be enough to change CFG_TZDRAM_SIZE in core/arch/arm/plat-vexpress/conf.mk at the relevant section. To understand how memory is partitioned check core/arch/arm/include/mm/generic_ram_layout.h

I don't think any changes are needed at all in TF-A.

When it comes to QEMU, your guesses are as good as mine.

[SimonWan]

@ademyankov do you by any chance increase the secure DRAM size?

[ademyankov]

@SimonWan no, I wasn't able to do that in optee_os 3.0 and 3.1... I switched to optee_os 3.4 and I had to increase the table size for some stuff in optee_os (#2845) to make it work.

[SimonWan]

@ademyankov you mean you can increase the SECURE DRAM_SIZE with optee_os 3.4? I update the values of SEC_DRAM_SIZE, SECRAM_SIZE,[VIRT_SECURE_MEM] and MAX_XLAT_TABLES as you mentioned in two posts but it looks like the SEC DRAM is still limited to 0x00f00000 as the default setting.

[ademyankov]

@SimonWan I haven't tried that cuz there was no need to. My TA is a bit over 3Mb and it didn't load by optee_os 3.1. but it has no issues w/ optee_os 3.4. and it loads and works fine. So, I just abandoned that idea for now.

[SimonWan]

@ademyankov got you, thanks.

[dreemkiller]

How anyone successfully been able to modify the QEMU support for TZDRAM to enable loading large (in my case, > 22MB) TAs?
I have modified the core/arch/arm/plat-vexpress/conf.mk with updated:
CFG_TZDRAM_SIZE ?= (128 * 1024 * 1024)
CFG_SHMEM_SIZE ?= (64 * 1024 * 1024)
CFG_MAX_XLAT_TABLES = 12

However, I am still not able to load a large TA binary.
Has anyone done this? If so, are there instructions, or even a publicly available fork of Optee?
It seems like this should be possible, but I am hitting a wall.

[SimonWan]

How anyone successfully been able to modify the QEMU support for TZDRAM to enable loading large (in my case, > 22MB) TAs?
I have modified the core/arch/arm/plat-vexpress/conf.mk with updated:
CFG_TZDRAM_SIZE ?= (128 * 1024 * 1024)
CFG_SHMEM_SIZE ?= (64 * 1024 * 1024)
CFG_MAX_XLAT_TABLES = 12

However, I am still not able to load a large TA binary.
Has anyone done this? If so, are there instructions, or even a publicly available fork of Optee?
It seems like this should be possible, but I am hitting a wall.

Not sure if this is the correct answer, while if you already correctly configured the DRAM size as explainted in the ram_layout, then probably you can give it a try with CFG_TEE_RAM_VA_SIZE to expand the VA size as well.

[dreemkiller]

as an update: I have used a larger CFG_TEE_RAM_VA_SIZE. However, I was still seeing memory allocation problems.
It turns out that the heap wasn't big enough.
Adding
CFG_CORE_HEAP_SIZE ?= 131072

to conf.mk solved the problem, and I was able to copy the full 22MB TA into TZ (still have a few problems, so this may not be the last word, but it's progress).