clang12: Using as compiler generates 30K warnings of -Wcompound-token-split-by-macro

[jkeenan]

Today I had occasion to try to build perl on FreeBSD-12 with, for the first time clang-12. Using this version of clang introduced over 30,000 new build-time warnings, all of type -Wcompound-token-split-by-macro.

$ uname -mrs
FreeBSD 12.2-STABLE amd64

$ ./perl -Ilib -V:config_args
config_args='-des -Dusedevel -Dusethreads -Dcc=clang12';

$ clang12 --version
clang version 12.0.0
Target: x86_64-portbld-freebsd12.2
Thread model: posix
InstalledDir: /usr/local/llvm12/bin

$ ~/bin/perl/report-build-warnings 1e43c13e1f.freebsd.clang12.maketp.output.txt.gz
File:  1e43c13e1f.freebsd.clang12.maketp.output.txt.gz

  Wcompound-token-split-by-macro           30106
  Wstring-compare                            2

$ zgrep "source =>" 1e43c13e1f.freebsd.clang12.parse.output.txt.gz | sort | uniq -c
  16     source => "../../../hv_func.h",
 144     source => "../../../inline.h",
 208     source => "../../../perl_siphash.h",
  96     source => "../../../sbox32_hash.h",
 272     source => "../../../zaphod32_hash.h",
 122     source => "../../hv_func.h",
1124     source => "../../inline.h",
1586     source => "../../perl_siphash.h",
 732     source => "../../sbox32_hash.h",
2074     source => "../../zaphod32_hash.h",
  36     source => "./const-c.inc",
  12     source => "./const-xs.inc",
  26     source => "./constants.xs",
  86     source => "./hv_func.h",
 774     source => "./inline.h",
  10     source => "./invlist_inline.h",
   6     source => "./numeric.xs",
1118     source => "./perl_siphash.h",
 516     source => "./sbox32_hash.h",
  16     source => "./vutil.c",
  32     source => "./vxs.inc",
1462     source => "./zaphod32_hash.h",
1180     source => "APItest.c",
 506     source => "APItest.xs",
 198     source => "B.c",
 140     source => "B.xs",
  20     source => "Base64.c",
   8     source => "Base64.xs",
  10     source => "Byte.c",
  90     source => "Bzip2.c",
  52     source => "Bzip2.xs",
  10     source => "CN.c",
  10     source => "Call.c",
  16     source => "Call.xs",
  14     source => "Collate.c",
  26     source => "Collate.xs",
  16     source => "Cwd.c",
  30     source => "Cwd.xs",
 170     source => "DB_File.c",
  60     source => "DB_File.xs",
   2     source => "DosGlob.c",
   2     source => "Dumper.c",
  62     source => "Dumper.xs",
  22     source => "DynaLoader.c",
  10     source => "EBCDIC.c",
  42     source => "Encode.c",
 120     source => "Encode.xs",
  20     source => "FastCalc.c",
  26     source => "FastCalc.xs",
  12     source => "Fcntl.xs",
  18     source => "FieldHash.c",
  14     source => "FieldHash.xs",
 132     source => "GDBM_File.c",
  32     source => "GDBM_File.xs",
  10     source => "Glob.c",
  24     source => "Glob.xs",
  84     source => "HiRes.c",
  18     source => "HiRes.xs",
   6     source => "Hostname.xs",
  46     source => "IO.c",
  20     source => "IO.xs",
  10     source => "JP.c",
  10     source => "KR.c",
   2     source => "Langinfo.c",
  66     source => "ListUtil.c",
 320     source => "ListUtil.xs",
   8     source => "MD5.c",
  24     source => "MD5.xs",
  64     source => "NDBM_File.c",
   2     source => "NDBM_File.xs",
  38     source => "Normalize.c",
  20     source => "Normalize.xs",
  24     source => "Opcode.c",
  22     source => "Opcode.xs",
 342     source => "POSIX.c",
  98     source => "POSIX.xs",
  28     source => "Peek.c",
   4     source => "Piece.c",
   4     source => "Piece.xs",
1036     source => "RealPPPort.c",
 696     source => "RealPPPort.xs",
  70     source => "SDBM_File.c",
   2     source => "SDBM_File.xs",
  38     source => "SHA.c",
  60     source => "SHA.xs",
  24     source => "Socket.c",
  46     source => "Socket.xs",
  26     source => "Storable.c",
2012     source => "Storable.xs",
  10     source => "Symbol.c",
  14     source => "SysV.c",
  96     source => "SysV.xs",
  48     source => "Syslog.c",
  10     source => "TW.c",
 264     source => "Typemap.c",
  42     source => "Typemap.xs",
   4     source => "Unicode.c",
  48     source => "Unicode.xs",
  20     source => "Util.c",
  86     source => "Util.xs",
   8     source => "XSUB-redefined-macros.xs",
   4     source => "XSUB-undef-XS_VERSION.xs",
 284     source => "Zlib.c",
  58     source => "Zlib.xs",
  20     source => "attributes.xs",
   8     source => "av.c",
   6     source => "caretx.c",
  62     source => "doio.c",
  42     source => "doop.c",
  42     source => "dump.c",
 134     source => "encoding.xs",
   6     source => "exception.c",
  64     source => "gv.c",
  34     source => "hv.c",
   8     source => "locale.c",
   2     source => "mathoms.c",
  98     source => "mg.c",
   6     source => "module3.c",
  52     source => "mro.xs",
   4     source => "mro_core.c",
  20     source => "numeric.c",
 592     source => "op.c",
  24     source => "pad.c",
 388     source => "perl.c",
  28     source => "perlio.c",
   6     source => "perly.c",
  12     source => "perly.y",
 834     source => "pp.c",
 252     source => "pp_ctl.c",
 366     source => "pp_hot.c",
 264     source => "pp_pack.c",
  30     source => "pp_sort.c",
 494     source => "pp_sys.c",
   2     source => "re.c",
  20     source => "re.xs",
2000     source => "re_comp.c",
 712     source => "re_exec.c",
  12     source => "reentr.c",
1476     source => "regcomp.c",
 266     source => "regexec.c",
   8     source => "scalar.xs",
   6     source => "scope.c",
  52     source => "shared.c",
 354     source => "shared.xs",
 760     source => "sv.c",
  42     source => "threads.c",
 190     source => "threads.xs",
  30     source => "time64.c",
 164     source => "toke.c",
 178     source => "universal.c",
  20     source => "utf8.c",
  58     source => "util.c",
  22     source => "via.xs",

The new warnings occur in pairs like this:

[
  {
    char   => 5,
    group  => "Wcompound-token-split-by-macro",
    line   => 150,
    source => "./zaphod32_hash.h",
    text   => "'(' and '{' tokens introducing statement expression appear in different macro expansion contexts",
  },
  {
    char   => 5,
    group  => "Wcompound-token-split-by-macro",
    line   => 150,
    source => "./zaphod32_hash.h",
    text   => "'}' and ')' tokens terminating statement expression appear in different macro expansion contexts",
  },

Or in the make output:

echo @`sh  cflags "optimize='-O2'" opmini.o`  -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB opmini.c
@clang12 -c -DPERL_CORE -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -std=c89 -O2 -Wall -Werror=pointer-arith -Wextra -Wc++-compat -Wwrite-strings -Werror=declaration-after-statement -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB opmini.c
In file included from op.c:163:
In file included from ./perl.h:4085:
In file included from ./hv.h:659:
In file included from ./hv_func.h:34:
In file included from ./sbox32_hash.h:4:
./zaphod32_hash.h:150:5: warning: '(' and '{' tokens introducing statement expression appear in different macro expansion contexts [-Wcompound-token-split-by-macro]
    ZAPHOD32_SCRAMBLE32(state[0],0x9fade23b);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./zaphod32_hash.h:80:38: note: expanded from macro 'ZAPHOD32_SCRAMBLE32'
#define ZAPHOD32_SCRAMBLE32(v,prime) STMT_START {  \
                                     ^~~~~~~~~~
./perl.h:737:29: note: expanded from macro 'STMT_START'
#   define STMT_START   (void)( /* gcc supports "({ STATEMENTS; })" */
                              ^
./zaphod32_hash.h:150:5: note: '{' token is here
    ZAPHOD32_SCRAMBLE32(state[0],0x9fade23b);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./zaphod32_hash.h:80:49: note: expanded from macro 'ZAPHOD32_SCRAMBLE32'
#define ZAPHOD32_SCRAMBLE32(v,prime) STMT_START {  \
                                                ^
./zaphod32_hash.h:150:5: warning: '}' and ')' tokens terminating statement expression appear in different macro expansion contexts [-Wcompound-token-split-by-macro]
    ZAPHOD32_SCRAMBLE32(state[0],0x9fade23b);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./zaphod32_hash.h:87:41: note: expanded from macro 'ZAPHOD32_SCRAMBLE32'
    v ^= (v>>23);                       \
                                        ^
./zaphod32_hash.h:150:5: note: ')' token is here
    ZAPHOD32_SCRAMBLE32(state[0],0x9fade23b);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./zaphod32_hash.h:88:3: note: expanded from macro 'ZAPHOD32_SCRAMBLE32'
} STMT_END
  ^~~~~~~~
./perl.h:738:21: note: expanded from macro 'STMT_END'
#   define STMT_END     )
                        ^

[Leont]

Sounds like we need to disable that warning? I'm not seeing the use of this warning.

[jkeenan]

One data point: The presence of this warning in our make output is new to clang12. I just installed LLVM/clang 11 on the same machine and built perl with the same config_args. The -Wcompound-token-split-by-macro warnings don't appear in clang11.

$ ~/bin/perl/report-build-warnings 1e43c13e1f.freebsd.clang11.maketp.output.txt.gz 
File:  1e43c13e1f.freebsd.clang11.maketp.output.txt.gz

  Wstring-compare                            2

[jkeenan]

Today I inspected the pull request submitted by @Biswa96 in #18879. I built a perl at commit 8ee57ef on FreeBSD-12 using clang12 as the compiler. The pull request eliminates the vast majority of build-time warnings discussed in this ticket.

$ report-build-warnings 8ee57ef583.freebsd.clang12.threaded.Biswa96.maketp.output.txt.gz
File:  8ee57ef583.freebsd.clang12.threaded.Biswa96.maketp.output.txt.gz

  Wcompound-token-split-by-macro           1804
  Wsign-compare                              2

(The -Wsign-compare warnings are from ListUtil.xs for which I have filed a ticket upstream.)

Bucketing the remaining build-time warnings we get:

$ grep "source =>" clang12.Biswa96.parsed-warnings.pl | grep -v 'ListUtil.xs' |sort | uniq -c
  46     source => "IO.c",
  22     source => "IO.xs",
1036     source => "RealPPPort.c",
 694     source => "RealPPPort.xs",
   6     source => "module3.c",

Of these, IO.c, RealPPPort.c and RealPPPort.xs apper to be generated files. module3.c is a test file underneath dist/Devel-PPPort/. That leaves dist/IO/IO.xs to be addressed.

[jkeenan]

I found that applying the change provided by @Biswa96 for perl.h to Devel-PPPort eliminates the remaining 1,804 -Wcompound-token-split-by-macro clang12 build-time warnings.

$ gitshowf |cat
commit 88e366d42b36d12b0fed022ac5209f0151150740
Author:     James E Keenan <[email protected]>
AuthorDate: Tue Jul 13 00:58:50 2021 +0000
Commit:     James E Keenan <[email protected]>
CommitDate: Tue Jul 13 00:58:50 2021 +0000

    Devel-PPPort: clang shows warnings with gcc brace groups
    
    Applying same change that Biswapriyo Nath provided in commit befee8d7e4
    for perl.h.  This eliminates remaining blead-upstream build-time
    warnings generated by compiling with clang12.

diff --git a/dist/Devel-PPPort/parts/inc/misc b/dist/Devel-PPPort/parts/inc/misc
index 9ec0a04df2..3718dd9b5b 100644
--- a/dist/Devel-PPPort/parts/inc/misc
+++ b/dist/Devel-PPPort/parts/inc/misc
@@ -240,7 +240,9 @@ __UNDEF_NOT_PROVIDED__  PERL_GCC_BRACE_GROUPS_FORBIDDEN
 #  endif
 #endif
 
-#if defined(__GNUC__) && !defined(PERL_GCC_BRACE_GROUPS_FORBIDDEN) && !defined(__cplusplus)
+/* clang shows warnings for tokens introducing statement expression appear
+ * in different macro expansion contexts. */
+#if defined(__GNUC__) && !defined(PERL_GCC_BRACE_GROUPS_FORBIDDEN) && !defined(__cplusplus) && !defined(__clang__)
 #  ifndef PERL_USE_GCC_BRACE_GROUPS
 #    define PERL_USE_GCC_BRACE_GROUPS
 #  endif

I applied this commit to the following branch, which is available for smoke-testing:
smoke-me/jkeenan/Biswa96/fix-clang-mingw

Thank you very much.
Jim Keenan

[jkeenan]

Applying the patch provided by @Biswa96 to both perl.h and Devel-PPPort appears to clear up the build-time warnings generated when using clang12 as the compiler.

Evidence for this comes from smoke-test reports on the smoke-me/jkeenan/Biswa96/fix-clang-mingw branch. Consider these two recent reports from [email protected] which appear to have the same configuration and have been run on the same machine.

blead
http://perl.develop-help.com/raw/?id=269540

Automated smoke report for branch blead 5.35.2 patch c9d678845a114f97048519e9c564b342b08ccfd2 v5.35.1-106-gc9d678845a
vier.local: Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz (GenuineIntel 3702MHz) (x86_64/1 cpu[8 cores])
    on        linux - 5.3.12-200.fc30.x86_64 [Fedora 34]
    using     ccache clang version Clang 12.0.0 (Fedora 12.0.0-2.fc34)
    smoketime 4 hours 12 minutes (average 1 hour 3 minutes)
    
v5.35.1-106-gc9d678845a  Configuration (common) 
-Accflags="-Werror=declaration-after-statement -g 
-fno-omit-frame-pointer -fsanitize=address  -fno-common 
-fsanitize-blacklist=`pwd`/asan_ignore" 
-Aldflags="-fsanitize=address" -Dcc="ccache clang"


$ zgrep -c -E 'warning:.*-W' logc9d678845a114f97048519e9c564b342b08ccfd2.log.gz 
134467

smoke-me/jkeenan/Biswa96/fix-clang-mingw
http://perl.develop-help.com/raw/?id=269579

Automated smoke report for branch smoke-me/jkeenan/Biswa96/fix-clang-mingw 5.35.2 patch 88e366d42b36d12b0fed022ac5209f0151150740 v5.35.1-112-g88e366d42b
vier.local: Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz (GenuineIntel 3702MHz) (x86_64/1 cpu[8 cores])
    on        linux - 5.3.12-200.fc30.x86_64 [Fedora 34]
    using     ccache clang version Clang 12.0.0 (Fedora 12.0.0-2.fc34)
    smoketime 4 hours 16 minutes (average 1 hour 4 minutes)
    
v5.35.1-112-g88e366d42b  Configuration (common) 
-Accflags="-Werror=declaration-after-statement -g 
-fno-omit-frame-pointer -fsanitize=address  -fno-common 
-fsanitize-blacklist=`pwd`/asan_ignore" 
-Aldflags="-fsanitize=address" -Dcc="ccache clang"

$ zgrep -c -E 'warning:.*-W' log88e366d42b36d12b0fed022ac5209f0151150740.log.gz 
8

[tonycoz]

I suspect the -Wcompound-token-split-by-macro warning is intended to highlight macros that might end up causing errors, and I think it's in general a good warning.

#18879 suggested disabling PERL_GCC_BRACE_GROUPS to prevent those warning, but that also disables many useful -DEBUGGING checks in sv.h.

I think a better alternative would be for STMT_START/STMT_END to not depend on PERL_GCC_BRACE_GROUPS, I don't think using the gcc extension here adds any value, since these are documented to not work as an expression.

Unfortunately Devel::PPPort also includes the same definition, so it also needs to be updated.

See #18984

[khwilliamson]

The commit that fixed this breaks Perl 5.13.0 in Devel::PPPort and needs to be reverted or changed

[khwilliamson]

The reason it breaks is related to this commit

Author: Nicholas Clark [email protected]
Date: Wed Sep 11 11:54:42 2013 +0200

Eliminate the only use of VOIDFLAGS, as part of STMT_START in perl.h

STMT_START has used VOIDFLAGS as part of its conditional compilation since it
was added by commit 728e280 (March 1996). The code originally read:

/* Now which other defined()s do we need here ??? */

Since then it has been amended to avoid entering that definition for GCC
on Solaris. Given that all current Solaris compilers are C89 conformant,
VOIDFLAGS will always be true (actually 15), so the test is redundant.
Even back in 1996, it's possible that VOIDFLAGS was always non-zero on
SunOS, rendering the test obsolete from the start.

Spotted by Brian Fraser, and extracted from a larger patch of his.

[khwilliamson]

Now fixed by 8a0214a

[skaji]

Hi,
I added this fix to Devel::PatchPerl too bingos/devel-patchperl#57.
Could you please leave a comment in that PR if you find something odd ?
Thanks.

[jkeenan]

Now fixed by 8a0214a

Confirmed at blead today.

$ ~/bin/perl/parse-one-warnings-group ebfc9d5d0d.freebsd.threaded.clang12.maketp.output.txt.gz "Wcompound-token-split-by-macro"
File:  ebfc9d5d0d.freebsd.threaded.clang12.maketp.output.txt.gz

Warnings group 'Wcompound-token-split-by-macro' not found at /home/jkeenan/bin/perl/parse-one-warnings-group line 31.

[gregoa]

On Wed, 13 Apr 2022 15:02:41 -0700, Leon Timmermans wrote: In that light I would argue `claws-mail` and `Sendmail-Milter` were always wrong to use them as expressions. In case of the latter it's a trivial fix (just remove the parentheses around the XPUSHs), in the former the ternary expressions should probably be rewritten as if/else statements.

Thanks! claws-mail already has such a patch (at least in Debian, don't know if Ricardo has forwarded it yet): https://salsa.debian.org/claws-mail-team/claws-mail/-/blob/master/debian/patches/20cope_with_fix_for_1009149.patch For Sendmail-Milter I followed your advice to remove the parentheses: https://salsa.debian.org/perl-team/modules/packages/libsendmail-milter-perl/raw/master/debian/patches/PERL_USE_GCC_BRACE_GROUPS.patch https://rt.cpan.org/Ticket/Display.html?id=142249 Cheers, gregor

…

-- .''`. https://info.comodo.priv.at -- Debian Developer https://www.debian.org : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D 85FA BB3A 6801 8649 AA06 `. `' Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe `-