STM32 embedded debug binaries much larger with 0.14.0-dev.2851+b074fb7dd

[marnix]

Zig Version

0.14.0-dev.2851+b074fb7dd

Steps to Reproduce and Observed Behavior

Build an STM32 binary in Debug mode (for the STM32F3DISCOVERY board in this case), with 0.14.0-dev.2851+b074fb7dd results in a 0x868a binary size, an increase of almost 16K over Zig 0.13.0 (and too large to fit in this chip's flash...).

It looks like there is a __aeabi_unwind_cpp_pr0 + __aeabi_unwind_cpp_pr1 + a ~16K memmove that all weren't there with 0.13.0...

We suspect that this may have been introduced with #22513.

Reproduction scenario: With https://github.com/marnix/microzig/tree/zig-issue-22603-0.14.0-dev.2851%2Bb074fb7dd (and Zig 0.14.0-dev.2851+b074fb7dd), in examples/stmicro/stm32, run zig build and inspect the resulting zig-out/firmware/stm32f3discovery.elf file, e.g. using readelf -S which shows a 0x868a bytes .text segment.

Expected Behavior

Building the same code (modulo trivial Zig 0.14.0 syntax changes, and leaving out the microzig 0.14.0 changes from marnix/microzig@cebfb1b) with Zig 0.13.0, the binary size was 0x49a2.

To show the original behavior, with https://github.com/marnix/microzig/tree/zig-issue-22603-0.13.0 (and Zig 0.13.0), do the same. The .text segment size was 0x49a2 bytes.

[alexrp]

cc @dweiller and @andrewrk

[dweiller]

@marnix can you tell me a target triple that replicates the behavior?

[marnix]

@marnix can you tell me a target triple that replicates the behavior?

@dweiller Updated the initial description.

[dweiller]

Just taking a guess that the right target/cpu is -target arm-freestanding -mcpu cortex_m4, if your code doesn't actually need memmove and is only including it because that version of zig uses it for memcpy, you should be able to use a newer compiler version (something after 0.14.0-dev.2882+18fcb3b5e), otherwise we'll have to wait for a new memmove implementation to be merged (I've got one in a branch, so hopefully not too long).

[alexrp]

I will say that a 16K memcpy/memmove implementation seems kind of unacceptable for any embedded work. Seems like we should have a smaller implementation for ReleaseSmall or something?

[marnix]

@marnix can you tell me a target triple that replicates the behavior?

@dweiller Sorry I didn't answer this directly.

Just taking a guess that the right target/cpu is -target arm-freestanding -mcpu cortex_m4, if your code doesn't actually need memmove and is only including it because that version of zig uses it for memcpy, you should be able to use a newer compiler version (something after 0.14.0-dev.2882+18fcb3b5e), otherwise we'll have to wait for a new memmove implementation to be merged (I've got one in a branch, so hopefully not too long).

Almost, zig ... --verbose shows me -ODebug -target thumb-freestanding-eabihf -mcpu cortex_m4+vfp4d16sp command line arguments.

[marnix]

I will say that a 16K memcpy/memmove implementation seems kind of unacceptable for any embedded work. Seems like we should have a smaller implementation for ReleaseSmall or something?

I know nothing about compiler-rt nor about LLVM optimizations, but with the same Zig 0.14.0-dev.2851+b074fb7dd, here is the size of __aeabi_mem... + related code in my binary, per ...-objectdump -d, for various optimization modes.

mode	addresses	size
-Doptimize=Debug	0x08004624..0x0800868a	16,486 bytes
-Doptimize=ReleaseSafe	0x0800047c..0x080044d6	16,474 bytes
-Doptimize=ReleaseFast	0x080003e0..0x0800443a	16,474 bytes
-Doptimize=ReleaseSmall	0x080003ce..0x08002012	7,236 bytes

[alexrp]

For comparison, it would be nice to see the same table for 0.13.0.

[marnix]

For comparison, it would be nice to see the same table for 0.13.0.

Now with Zig 0.13.0, here is the size of __aeabi_mem... + related code in my binary, per ...-objectdump -d, for various optimization modes.

mode	addresses	size
-Doptimize=Debug	0x080048ac..0x080049a2	246 bytes
-Doptimize=ReleaseSafe	0x080003f0..0x080004d6	230 bytes
-Doptimize=ReleaseFast	0x080003c8..0x080004ae	230 bytes
-Doptimize=ReleaseSmall	0x080003c4..0x080003fa	54 bytes

---

Just for fun, here are those last 54 bytes:

080003c4 <__aeabi_memcpy>:
 80003c4:       f000 b804       b.w     80003d0 <memcpy>

080003c8 <__aeabi_memclr>:
 80003c8:       460a            mov     r2, r1
 80003ca:       2100            movs    r1, #0
 80003cc:       f000 b80e       b.w     80003ec <memset>

080003d0 <memcpy>:
 80003d0:       b15a            cbz     r2, 80003ea <memcpy+0x1a>
 80003d2:       b510            push    {r4, lr}
 80003d4:       3a01            subs    r2, #1
 80003d6:       4603            mov     r3, r0
 80003d8:       780c            ldrb    r4, [r1, #0]
 80003da:       701c            strb    r4, [r3, #0]
 80003dc:       b11a            cbz     r2, 80003e6 <memcpy+0x16>
 80003de:       3a01            subs    r2, #1
 80003e0:       3101            adds    r1, #1
 80003e2:       3301            adds    r3, #1
 80003e4:       e7f8            b.n     80003d8 <memcpy+0x8>
 80003e6:       e8bd 4010       ldmia.w sp!, {r4, lr}
 80003ea:       4770            bx      lr

080003ec <memset>:
 80003ec:       b122            cbz     r2, 80003f8 <memset+0xc>
 80003ee:       4603            mov     r3, r0
 80003f0:       f803 1b01       strb.w  r1, [r3], #1
 80003f4:       3a01            subs    r2, #1
 80003f6:       d1fb            bne.n   80003f0 <memset+0x4>
 80003f8:       4770            bx      lr

[marnix]

Just taking a guess that the right target/cpu is -target arm-freestanding -mcpu cortex_m4, if your code doesn't actually need memmove and is only including it because that version of zig uses it for memcpy, you should be able to use a newer compiler version (something after 0.14.0-dev.2882+18fcb3b5e), otherwise we'll have to wait for a new memmove implementation to be merged (I've got one in a branch, so hopefully not too long).

@dweiller Yes, I think that my minimalistic project only uses memcpy (and memclr), not memmove.

Ah, you're saying that your 18fcb3b for #18912 has improved the size for memcpy, but left memmove unchanged, and also that is being improved (perhaps in #22606, thanks in advance!).

I will test once build 2882 or later lands on the Download page.

Thanks!

[andrewrk]

This is an easy fix.

If the generic implementation under -OReleaseSmall does not generate the optimal machine code, then put a conditional compilation branch that checks the target and uses inline assembly.

Just to make sure I understand, however, is it the case that LLVM is lowering this

    for (0..len) |i| {
        dest.?[i] = src.?[i];
    }

to a 16K implementation? sounds like the stm32 backend of llvm is not great.

It looks like there is a __aeabi_unwind_cpp_pr0 + __aeabi_unwind_cpp_pr1

These symbols should be deleted by the linker if they are unused. Does the project use C++ (I sure hope not)?

---

Looks to me like -OReleaseSmall memcpy in this target is indeed very small:

https://zig.godbolt.org/z/337xxa9ze

here you can see the rules for selecting compiler_rt optimization mode:

zig/src/Compilation.zig

Lines 6794 to 6798 in 015a5e1

	switch (comp.root_mod.optimize_mode) {
	.Debug, .ReleaseSafe => return target_util.defaultCompilerRtOptimizeMode(target),
	.ReleaseFast => return .ReleaseFast,
	.ReleaseSmall => return .ReleaseSmall,
	}

Sounds like the project should be using -ORelaseSmall, not -OReleaseFast.

[ikskuh]

Just to make sure I understand, however, is it the case that LLVM is lowering this

for (0..len) |i| {
    dest.?[i] = src.?[i];
}

A minimal reproduction with output:

export fn _start(noalias dest: ?[*]u8, noalias src: ?[*]const u8, len: usize) callconv(.C) void {
    @memcpy(dest.?[0..len], src.?[0..len]);
}

compiled with

[felix@xqwork tmp]$ ./zig-linux-x86_64-0.14.0-dev.2851+b074fb7dd/zig build-exe -OReleaseSmall -target thumb-freestanding-eabihf -mcpu cortex_m4+vfp4d16sp memcpy.zig -fno-strip
[felix@xqwork tmp]$ llvm-objdump memcoy -hpt

memcoy: file format elf32-littlearm

Program Header:
    PHDR off    0x00000034 vaddr 0x00010034 paddr 0x00010034 align 2**2
         filesz 0x000000a0 memsz 0x000000a0 flags r--
    LOAD off    0x00000000 vaddr 0x00010000 paddr 0x00010000 align 2**16
         filesz 0x000000f4 memsz 0x000000f4 flags r--
    LOAD off    0x000000f4 vaddr 0x000200f4 paddr 0x000200f4 align 2**16
         filesz 0x00001c34 memsz 0x00001c34 flags r-x
   STACK off    0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**64
         filesz 0x00000000 memsz 0x01000000 flags rw-
 UNKNOWN off    0x000000d4 vaddr 0x000100d4 paddr 0x000100d4 align 2**2
         filesz 0x00000020 memsz 0x00000020 flags r--

Dynamic Section:

Sections:
Idx Name            Size     VMA      Type
  0                 00000000 00000000 
  1 .ARM.exidx      00000020 000100d4 
  2 .text           00001c34 000200f4 TEXT
  3 .debug_abbrev   000004b4 00000000 DEBUG
  4 .debug_info     0001b326 00000000 DEBUG
  5 .debug_str      00007d45 00000000 DEBUG
  6 .debug_pubnames 000045fd 00000000 DEBUG
  7 .debug_pubtypes 00000cdc 00000000 DEBUG
  8 .ARM.attributes 00000045 00000000 
  9 .debug_frame    00003fb4 00000000 DEBUG
 10 .debug_line     0000f26e 00000000 DEBUG
 11 .debug_loc      0002ae70 00000000 DEBUG
 12 .debug_ranges   00003c28 00000000 DEBUG
 13 .comment        00000067 00000000 
 14 .symtab         00000150 00000000 
 15 .shstrtab       000000bc 00000000 
 16 .strtab         000000d0 00000000 

SYMBOL TABLE:
00000000 l    df *ABS*  00000000 memcoy
000200f4 l       .text  00000000 $t
00000000 l    df *ABS*  00000000 compiler_rt
00021cf0 l     F .text  00000006 OUTLINED_FUNCTION_94
000200f8 l       .text  00000000 $t
000200fa l       .text  00000000 $t
000200fe l       .text  00000000 $t
00021cf6 l     F .text  00000012 OUTLINED_FUNCTION_145
00021d1a l     F .text  0000000e OUTLINED_FUNCTION_222
00021cde l     F .text  00000012 OUTLINED_FUNCTION_13
00021d08 l     F .text  00000012 OUTLINED_FUNCTION_155
00021cde l       .text  00000000 $t
00021cf0 l       .text  00000000 $t
00021cf6 l       .text  00000000 $t
00021d08 l       .text  00000000 $t
00021d1a l       .text  00000000 $t
000200f4 g     F .text  00000004 _start
000200fa  w    F .text  00000004 __aeabi_memcpy
000200f8  w    F .text  00000002 __aeabi_unwind_cpp_pr0
000200fe  w    F .text  00001be0 memmove
[felix@xqwork tmp]$ llvm-size memcpy
   text    data     bss     dec     hex filename
   7252       0       0    7252    1c54 memcpy
[felix@xqwork tmp]$ 

which yields a 7136 large memmove implementation with a 7252 byte large executable.

vs.

export fn _start(noalias dest: ?[*]u8, noalias src: ?[*]const u8, len: usize) callconv(.C) void {
    for (0..len) |i| {
        dest.?[i] = src.?[i];
    }

}

compiled with

[felix@xqwork tmp]$ ./zig-linux-x86_64-0.14.0-dev.2851+b074fb7dd/zig build-exe -OReleaseSmall -target thumb-freestanding-eabihf -mcpu cortex_m4+vfp4d16sp -fno-strip memcpy.zig
[felix@xqwork tmp]$ llvm-objdump memcpy -hpt

memcoy: file format elf32-littlearm

Program Header:
    PHDR off    0x00000034 vaddr 0x00010034 paddr 0x00010034 align 2**2
         filesz 0x000000a0 memsz 0x000000a0 flags r--
    LOAD off    0x00000000 vaddr 0x00010000 paddr 0x00010000 align 2**16
         filesz 0x000000e4 memsz 0x000000e4 flags r--
    LOAD off    0x000000e4 vaddr 0x000200e4 paddr 0x000200e4 align 2**16
         filesz 0x00000012 memsz 0x00000012 flags r-x
   STACK off    0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**64
         filesz 0x00000000 memsz 0x01000000 flags rw-
 UNKNOWN off    0x000000d4 vaddr 0x000100d4 paddr 0x000100d4 align 2**2
         filesz 0x00000010 memsz 0x00000010 flags r--

Dynamic Section:

Sections:
Idx Name            Size     VMA      Type
  0                 00000000 00000000 
  1 .ARM.exidx      00000010 000100d4 
  2 .text           00000012 000200e4 TEXT
  3 .debug_loc      0002aeb2 00000000 DEBUG
  4 .debug_abbrev   000004b4 00000000 DEBUG
  5 .debug_info     0001b32c 00000000 DEBUG
  6 .debug_str      00007d45 00000000 DEBUG
  7 .debug_pubnames 000045fd 00000000 DEBUG
  8 .debug_pubtypes 00000cdc 00000000 DEBUG
  9 .ARM.attributes 00000045 00000000 
 10 .debug_frame    00003fb4 00000000 DEBUG
 11 .debug_line     0000f288 00000000 DEBUG
 12 .debug_ranges   00003c28 00000000 DEBUG
 13 .comment        00000067 00000000 
 14 .symtab         00000070 00000000 
 15 .shstrtab       000000bc 00000000 
 16 .strtab         00000038 00000000 

SYMBOL TABLE:
00000000 l    df *ABS*  00000000 memcoy
000200e4 l       .text  00000000 $t
00000000 l    df *ABS*  00000000 compiler_rt
000200f4 l       .text  00000000 $t
000200e4 g     F .text  00000010 _start
000200f4  w    F .text  00000002 __aeabi_unwind_cpp_pr0
[felix@xqwork tmp]$ llvm-size memcpy
   text    data     bss     dec     hex filename
     34       0       0      34      22 memcpy
[felix@xqwork tmp]$ 

which does not force memmove or memcpy to exist, and is only 34 bytes large.

The generated code for memmove is horrible.

[ikskuh]

This issue should be a release blocker btw as it would make Zig completely unusable on embedded devices as soon as one uses @memcpy with the assumption it is doing efficient work.

My guess is that it's the auto-devectorization that fails considering all these byte loads and stores.

A quick test confirms this:

export fn _start(noalias dest: ?[*]align(64) u8, noalias src: ?[*]const align(64) u8, len: usize) callconv(.C) void {
    const vec_dst: *@Vector(32, u8) = @ptrCast(dest);
    const vec_src: *const @Vector(32, u8) = @ptrCast(src);

    vec_dst.* = vec_src.*;
    _ = len;
}

compiles down to

Disassembly of section .text:

000200ec <_start>:
   200ec: b5f0         	push	{r4, r5, r6, r7, lr}
   200ee: e891 10fc    	ldm.w	r1, {r2, r3, r4, r5, r6, r7, r12}
   200f2: 69c9         	ldr	r1, [r1, #0x1c]
   200f4: e880 10fc    	stm.w	r0, {r2, r3, r4, r5, r6, r7, r12}
   200f8: 61c1         	str	r1, [r0, #0x1c]
   200fa: bdf0         	pop	{r4, r5, r6, r7, pc}

000200fc <__aeabi_unwind_cpp_pr0>:
   200fc: 4770         	bx	lr

but wrapping it in a function explodes:

export fn _start(noalias dest: ?[*]align(64) u8, noalias src: ?[*]const align(64) u8, len: usize) callconv(.C) void {
    copyVector(@ptrCast(dest), @as(*const @Vector(32, u8), @ptrCast(src)).*);
    _=len;
}

fn copyVector(dst: *volatile @Vector(32, u8), src: @Vector(32, u8)) void {
    dst.* = src;
}

compiles into this bloat:

Disassembly of section .text:

000200ec <_start>:
   200ec: b510         	push	{r4, lr}
   200ee: 7fca         	ldrb	r2, [r1, #0x1f]
   200f0: 77c2         	strb	r2, [r0, #0x1f]
   200f2: 7f8a         	ldrb	r2, [r1, #0x1e]
   200f4: 7782         	strb	r2, [r0, #0x1e]
   200f6: 7f4a         	ldrb	r2, [r1, #0x1d]
   200f8: 7742         	strb	r2, [r0, #0x1d]
   200fa: 7f0a         	ldrb	r2, [r1, #0x1c]
   200fc: 7702         	strb	r2, [r0, #0x1c]
   200fe: 7eca         	ldrb	r2, [r1, #0x1b]
   20100: 76c2         	strb	r2, [r0, #0x1b]
   20102: 7e8a         	ldrb	r2, [r1, #0x1a]
   20104: 7682         	strb	r2, [r0, #0x1a]
   20106: 7e4a         	ldrb	r2, [r1, #0x19]
   20108: 7642         	strb	r2, [r0, #0x19]
   2010a: 7e0a         	ldrb	r2, [r1, #0x18]
   2010c: 7602         	strb	r2, [r0, #0x18]
   2010e: 7dca         	ldrb	r2, [r1, #0x17]
   20110: 75c2         	strb	r2, [r0, #0x17]
   20112: 7d8a         	ldrb	r2, [r1, #0x16]
   20114: 7582         	strb	r2, [r0, #0x16]
   20116: 7d4a         	ldrb	r2, [r1, #0x15]
   20118: 7542         	strb	r2, [r0, #0x15]
   2011a: 7d0a         	ldrb	r2, [r1, #0x14]
   2011c: 7502         	strb	r2, [r0, #0x14]
   2011e: 7cca         	ldrb	r2, [r1, #0x13]
   20120: 74c2         	strb	r2, [r0, #0x13]
   20122: 7c8a         	ldrb	r2, [r1, #0x12]
   20124: 7482         	strb	r2, [r0, #0x12]
   20126: 7c4a         	ldrb	r2, [r1, #0x11]
   20128: 7442         	strb	r2, [r0, #0x11]
   2012a: 7c0a         	ldrb	r2, [r1, #0x10]
   2012c: 7402         	strb	r2, [r0, #0x10]
   2012e: 7bca         	ldrb	r2, [r1, #0xf]
   20130: 73c2         	strb	r2, [r0, #0xf]
   20132: 7b8a         	ldrb	r2, [r1, #0xe]
   20134: 7382         	strb	r2, [r0, #0xe]
   20136: 7b4a         	ldrb	r2, [r1, #0xd]
   20138: 7342         	strb	r2, [r0, #0xd]
   2013a: 7b0a         	ldrb	r2, [r1, #0xc]
   2013c: 7302         	strb	r2, [r0, #0xc]
   2013e: 7aca         	ldrb	r2, [r1, #0xb]
   20140: 72c2         	strb	r2, [r0, #0xb]
   20142: 7a8a         	ldrb	r2, [r1, #0xa]
   20144: 7282         	strb	r2, [r0, #0xa]
   20146: 7a4a         	ldrb	r2, [r1, #0x9]
   20148: 7242         	strb	r2, [r0, #0x9]
   2014a: 7a0a         	ldrb	r2, [r1, #0x8]
   2014c: 7202         	strb	r2, [r0, #0x8]
   2014e: 79ca         	ldrb	r2, [r1, #0x7]
   20150: 71c2         	strb	r2, [r0, #0x7]
   20152: 798a         	ldrb	r2, [r1, #0x6]
   20154: 7182         	strb	r2, [r0, #0x6]
   20156: 794a         	ldrb	r2, [r1, #0x5]
   20158: 7142         	strb	r2, [r0, #0x5]
   2015a: 790a         	ldrb	r2, [r1, #0x4]
   2015c: 7102         	strb	r2, [r0, #0x4]
   2015e: 780a         	ldrb	r2, [r1]
   20160: 784b         	ldrb	r3, [r1, #0x1]
   20162: 788c         	ldrb	r4, [r1, #0x2]
   20164: 78c9         	ldrb	r1, [r1, #0x3]
   20166: 70c1         	strb	r1, [r0, #0x3]
   20168: 7084         	strb	r4, [r0, #0x2]
   2016a: 7043         	strb	r3, [r0, #0x1]
   2016c: 7002         	strb	r2, [r0]
   2016e: bd10         	pop	{r4, pc}

00020170 <__aeabi_unwind_cpp_pr0>:
   20170: 4770         	bx	lr

which is creating a byte-by-byte copy instead of using whatever smart instruction is possible.

I'd even say this is a problematic miscompilation which might affect a lot of other non-vectorized targets as well

[andrewrk]

Agreed, it's a release blocker.

Easy to fix, we simply need to drop a good implementation in there.

[andrewrk]

Please check 0.14.0-dev.2987+183bb8b08 (available now on download page).

[marnix]

Please check 0.14.0-dev.2987+183bb8b08 (available now on download page).

Just now I tested with Zig 0.14.0-dev.2989+bf6ee7cb3, so here again is the size of __aeabi_mem... + related code in my binary (see earlier link), per ...-objectdump -d, for various optimization modes.

mode	addresses	size	size with Zig 0.13.0 (see earlier)
-Doptimize=Debug	0x080042bc..0x0800441e	354 bytes	246 bytes
-Doptimize=ReleaseSafe	0x0800048c..0x080005e2	342 bytes	230 bytes
-Doptimize=ReleaseFast	0x080003e8..0x0800053e	342 bytes	230 bytes
-Doptimize=ReleaseSmall	0x080003d2..0x080003fe	44 bytes	54 bytes

So that is close to the Zig 0.13.0 sizes again, roughly (~110 bytes or ~45% larger, except small is 10 bytes smaller).

[andrewrk]

Looks good then - any problems remaining?

[marnix]

@andrewrk I'm happy on this one!

I'll leave it to @dweiller and you to see whether or not to put #22606 in 0.14.0 as well, I'm watching that one, and I plan to do the same measurements on the same code again after that lands on the Download page.