Enable link-time optimization (after switching to avr-gcc 4.5 or greater) [imported]

[cmaglie]

This is Issue 660 moved from a Google Code project.
Added by 2011-09-27T22:38:57.000Z by [email protected].
Please review that bug for more context and additional comments, but update this bug.

Original labels: Type-Defect, Priority-Medium

Original description

avr-gcc 4.5 has support for LTO. Enabling it would probably allow sizable object size reductions, especially with -Os when linking libraries spanning multiple compilation units.

[matthijskooijman]

Here's a comment copied from Google Code:

LTO stands for link-time optimization and basically it means deferring most optimizations until all compilation units (each file being compiled, including libraries) have been processed.
As an example, consider a library function that's called only once in the current program. Without LTO both caller and callee will be compiled separately. With LTO the callee will be inlined (it's always profitable to inline a function that's called only once). While just inlining might look not much of a win, take into consideration that this also allows code specialization at compile-time.
For more information, look up the -flto and -fwhole-program compile options (http://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#Optimize-Options)

I've been looking at LTO and whole program optimization as well, since it greatly increases optimization opportunities for a templated I/O library I've been working on.

To get this working, -flto needs to be passed to all C/C++ compiler commands and linker commands. It seems sensible to pass -O0 to the compiler commands since optimization is done in the linker stage, but a quick test shows that this greatly increases resulting binary size, so I guess not all optimizations are run at link time and -Os must still be passed both when compiling and linking.

I've tested this here on avr-gcc 4.7 in Debian using Arduino-mk and it indeed helps to reduce code size (haven't looked closely at code speed yet).

However, for optimal LTO, gcc needs a "linker plugin", which doesn't seem to be available in my setup (not sure if this is a limitation of the AVR target and/or if this was / will be fixed in a more recent GCC version, thogh). AFAIU this linker plugin serves two purposes: It allows gcc to get LTO information out of archives (allowing the arduino core code in libcore.a to be included in the LTO process) and it allows gcc to know which functions are externally visible (in this case, only main()) so it can really tell if a function is used or not.

As an alternative for such a linker plugin, -fwhole-program can be passed to the linker. This flag tells gcc, "I've passed you all the code for this program and only main() and any functions with the externally_visible attribute are externally visible". However, since no linker plugin is used, any .o files compiled without -flto and any .a files included in the link are not included in the LTO and the optimizer will pretend they don't exist. In the Arduino case, the LTO will not see the actual main() function (which is in libcore.a) referencing setup() and loop(), making it remove the setup() and loop() functions (and probably all of the rest of your code as well). After LTO, the linker will then try to link your (now-empty) program with libcore.a and error out with "undefined symbol setup" and "undefined symbol loop".

Making -fwhole-program can be done by marking setup and loop as externally visible. e.g. put this in Arduino.h (or any other place included in the .ino file, just in main.cpp isn't enough):

void loop(void) __attribute__((externally_visible));
void setup(void) __attribute__((externally_visible));

Doing this makes the program a bit smaller and probably faster as well (it allows inlining and removing functions that are called in on place only, for example). However, it is probably a bit error prone to manually keep this list of externally visible functions, so I would advise only applying it if linker plugin support for AVR isn't going to happen soon and it turns out the optimization gain of -fwhole-program is significant.

[matthijskooijman]

It seems that since gcc 4.8, avr-gcc has the linker plugin working, meaning we can enable -flto and it should just work :-)

[cmaglie]

This may be enable after merging #1903 (just to keep a note inside the pull request).

[mikaelpatel]

The -flto option needs to be passed to the linker as well (compiler.c.elf). Below are the extra flags I am using for Cosa build (See https://github.com/mikaelpatel/Cosa/blob/master/platform.txt-gcc-4.8.1)

# These can be overridden in platform.local.txt
compiler.c.extra_flags=-Wextra -flto
compiler.c.elf.extra_flags=-w -flto
compiler.S.extra_flags=
compiler.cpp.extra_flags=-Wextra -flto
compiler.ar.extra_flags=
compiler.objcopy.eep.extra_flags=
compiler.elf2hex.extra_flags=

This work nicely for the Cosa Arduino Framework. So far have all tests passed. Please note that some warnings from the link needs to suppressed (See http://gcc.gnu.org/bugzilla/show_bug.cgi?id=59396)

Cheers! Mikael

[ffissore]

Current toolchain supports link time optimization, which is disabled by default. If you wish to use you can either modify platform.txt or provide your own custom core

[mikaelpatel]

There are still problems with the LTO plugin (for Windows). It is not always included and "collect" will fail. I have disabled LTO in the Arduino IDE build (platform.txt) and only use this from the Cosa command line build (Makefile).

[matthijskooijman]

Is it still the ambition to enable LTO, if this is possible without introducing any broken behaviour? If so, perhaps this issue should be left open, to track any problems and solutions with LTO found so far?

I just noticed a problem with LTO giving incorrect ISR warnings. This is reported upstream and fixed in 4.8.3.

[cmaglie]

I guess so, let's reopen this one and collect all the problems around LTO:

@mikaelpatel
what's the problem you're facing exactly with the plugin? are you using the same toolchain shipped with Arduino for your makefile?

@Lauszus @xxxajk
IIRC there was a problem with LTO and USB_Host_2 library?

if anyone has information and links, please post here.

[cmaglie]

Dev-list discussion here:
https://groups.google.com/a/arduino.cc/d/msg/developers/21G5w2HbUOg/000Ql0O6CCwJ

[xxxajk]

@cmaglie we disable optimization in the code for the versions of GCC that we know have the problem.

[facchinm]

Some goodies for the weekend:
a new shiny avr-gcc 5.1.0 for all major platforms with LTO plugin support!

Windows x86
OSX x86_64
Linux x86_64

All the toolchains has been cross compiled on Linux using this script (in case you want to compile it by yourself).

To enable LTO simply add the following flags to your platform.txt

compiler.c.extra_flags=-Wextra -flto
compiler.c.elf.extra_flags=-w -flto -fuse-linker-plugin
compiler.S.extra_flags=-flto
compiler.cpp.extra_flags=-Wextra -flto
compiler.objcopy.eep.extra_flags=
compiler.elf2hex.extra_flags=

and a platform-dependant

• UNIX

compiler.ar.extra_flags=--plugin={$extraction_path}/libexec/gcc/avr/5.1.0/liblto_plugin.so

• WINDOWS

compiler.ar.extra_flags=--plugin={$extraction_path}/libexec/gcc/avr/5.1.0/liblto_plugin-0.dll

As usual, testing and error reporting is VERY well accepted 😉

@matthijskooijman - could you give this compiler+flags a spin with your arduino-mass-builder to get some metrics?

[matthijskooijman]

Cool! This script, is it based on on in the avr-toolchain repo? And the patch you linked is the only one applied, or does this include patches from Atmel too (like the ones in the avr-toolchain repo)?

In your compiler flags, I assume -Wextra isn't actually needed for LTO?

I didn't realize about -fuse-linker-plugin, so I tested without that before. Also, I take the ar options allow ar to store lto info inside .a files? From reading the gcc manpage, it looks like this option might not be needed if you call gcc-ar instead of just ar?

Finally, it seems that -fno-fat-lto-objects might be useful to speed up compilation by not generating machine code twice.

I'll run this through my mass-builder script, but I don't have a laptop adapter with me, so can't do that right now :-)

[Lauszus]

@cmaglie I just tried enabling LTO with the current toolchain bundled with the Arduino IDE and unfortunately the issue is still present:

Without LTO:

Sketch uses 21,908 bytes (8%) of program storage space. Maximum is 253,952 bytes.
Global variables use 968 bytes (11%) of dynamic memory, leaving 7,224 bytes for local variables. Maximum is 8,192 bytes.

With LTO:

Sketch uses 70,492 bytes (27%) of program storage space. Maximum is 253,952 bytes.
Global variables use 1,289 bytes (15%) of dynamic memory, leaving 6,903 bytes for local variables. Maximum is 8,192 bytes.

@facchinm I tried to test it with the updated toolchain you linked to, but I am getting the following error:

/Users/Lauszus/Github/Arduino/build/macosx/work/Arduino.app/Contents/Java/hardware/tools/avr/bin/avr-g++ -c -g -Os -Wall -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -MMD -mmcu=atmega2560 -DF_CPU=16000000L -DARDUINO=10606 -DARDUINO_AVR_MEGA2560 -DARDUINO_ARCH_AVR -flto -I/Users/Lauszus/Github/Arduino/build/macosx/work/Arduino.app/Contents/Java/hardware/arduino/avr/cores/arduino -I/Users/Lauszus/Github/Arduino/build/macosx/work/Arduino.app/Contents/Java/hardware/arduino/avr/variants/mega -I/Users/Lauszus/Dropbox/Arduino/libraries/USB_Host_Shield_2.0 -I/Users/Lauszus/Dropbox/Arduino/libraries/spi4teensy3 -I/Users/Lauszus/Github/Arduino/build/macosx/work/Arduino.app/Contents/Java/hardware/arduino/avr/libraries/SPI /var/folders/st/ln3d8j1n7x91gq9gk8304gtw0000gn/T/build1230170329459071487.tmp/sketch/PS3BT.cpp -o /var/folders/st/ln3d8j1n7x91gq9gk8304gtw0000gn/T/build1230170329459071487.tmp/sketch/PS3BT.cpp.o 
FATAL:/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../libexec/as/x86_64/as: I don't understand 'm' flag!
Error compiling.

[facchinm]

@Lauszus it seems that the default x86_64 toolchain from Xcode was chosen for linking...
I suggest to extract the 5.1.0 toolchain in a different folder (not inside Arduino.app ) and then hardcode the path in compiler.path and compiler.ar.extra_flags fields.
I've tested it in a MacMini (10.9) and the paths are taken correctly.
BTW, thanks for testing!!

[facchinm]

@matthijskooijman , you are clearly right about -Wextra, it was a leftover from previous tests.
ar and gcc-ar are alias in an effort to have the exact same compilation flags (and thus the same behaviour) on all the three platforms.

I've tested also with -fno-fat-lto-objects but I couldn't appreciate the speedup (I didn't measured it, though).

Finally, the script is loosely based on avr-toolchain repo but its main target is to enable canadian cross compiling for all the platform with the same flags on a single machine. No Atmel patch applied at the moment (except one for the GCC linker bug)

Eagerly waiting for the results of your tests!

[facchinm]

@harryboo
here you are http://downloads.arduino.cc/experimental-toolchains/avr-gcc-6.1-mingw32.zip
The toolchain is totally untested, use at your own risk 😄

[harryboo]

@facchinm:

Thanx with zip as extension the link is working. But i cannot get this toolchain to work.
I have replaced the content of the avr directory under hardware\tools and copied the libwinpthread-1.dll into the bin directory.
But i get this error in the arduino ide "exec: "/bin/avr-g++": file does not exist". If i double click the exe i get this error message in a popup "the application was unable to start correctly 0xc00007b".

Do you know how to get the toolchain to work?

Thanx

[facchinm]

The first error shows that the IDE didn't recognize the installed core, probably because overwriting the folder content you missed hardware/tools/avr/builtin_tools_versions.txt file which must contain

arduino.avrdude=6.0.1-arduino5
arduino.avr-gcc=4.8.1-arduino5

(even if we are using newer version, the builtin core must be related to the published cores).
About the failure to start, it could be everything, I don't have any time now to test that toolchain on a Win machine... But if you can debug it and/or paste here the error message from a command line shell it would be great 😉

[NicoHood]

FYI the build params on arch (from the official repos):

[arch@arch build]$ avr-gcc -v
Using built-in specs.
Reading specs from /usr/lib/gcc/avr/5.3.0/device-specs/specs-avr2
COLLECT_GCC=avr-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/avr/5.3.0/lto-wrapper
Target: avr
Configured with: /build/avr-gcc/src/gcc-5-20160223/configure --disable-install-libiberty --disable-libssp --disable-libstdcxx-pch --disable-libunwind-exceptions --disable-linker-build-id --disable-nls --disable-werror --enable-__cxa_atexit --enable-checking=release --enable-clocale=gnu --enable-gnu-unique-object --enable-gold --enable-languages=c,c++ --enable-ld=default --enable-lto --enable-plugin --enable-shared --infodir=/usr/share/info --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --prefix=/usr --target=avr --with-as=/usr/bin/avr-as --with-gnu-as --with-gnu-ld --with-ld=/usr/bin/avr-ld --with-plugin-ld=ld.gold --with-system-zlib --with-isl --enable-gnu-indirect-function
Thread model: single
gcc version 5.3.0 (GCC)

[matthijskooijman]

This is fixed in 1.6.10.

[NicoHood]

I am trying to use the upstream avr-gcc from archlinux with arduino now.
I also noticed the segmentation fault and now luckly found your hint (thanks!!).
Hence, adding -fno-devirtualize to c and cpp extra flags solves the segfault.

However I now have this problem. It also seem to occur with older versios of gcc, but Those are quite incompatible with the latest arch now, so its hard to debug.

Linking everything together...
"/usr/bin/avr-gcc" -w -Os -flto -fuse-linker-plugin -Wl,--gc-sections -mmcu=atmega32u4  -o "/tmp/build44e5805ec640d1d5675b7c658d939d94.tmp/SoftwareSerialExample.ino.elf" "/tmp/build44e5805ec640d1d5675b7c658d939d94.tmp/sketch/SoftwareSerialExample.ino.cpp.o" "/tmp/build44e5805ec640d1d5675b7c658d939d94.tmp/libraries/SoftwareSerial/SoftwareSerial.cpp.o" "/tmp/build44e5805ec640d1d5675b7c658d939d94.tmp/core/core.a" "-L/tmp/build44e5805ec640d1d5675b7c658d939d94.tmp" -lm
/tmp/ccNp6vGT.ltrans0.ltrans.o: In function `_GLOBAL__sub_I_mySerial':
<artificial>:(.text.startup+0x25a): undefined reference to `__dso_handle'
<artificial>:(.text.startup+0x25c): undefined reference to `__dso_handle'
<artificial>:(.text.startup+0x26e): undefined reference to `__cxa_atexit'
/usr/bin/avr-ld: /tmp/build44e5805ec640d1d5675b7c658d939d94.tmp/SoftwareSerialExample.ino.elf: hidden symbol `__dso_handle' isn't defined
/usr/bin/avr-ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

I've searched this online, but I could not find much information about this. Any ideas?
Edit: This error only seem to occur with the Software serial example, other sketches seem to work fine. The error is caused by the deconstructor. If you remove it, the sketch compiles fine. This might be caused by the compiler switch that I've added above.

[xxxajk]

Obviously it is too old and lacks the feature/fix. :-)

I doubt much can be done about it, unless you track down those functions
and replicate them as-needed, and in a way that does not interfere with
working compile environment.

On Tue, Sep 20, 2016 at 12:25 PM, Nico [email protected] wrote:

I am trying to use the upstream avr-gcc from archlinux with arduino now.
I also noticed the segmentation fault and now luckly found your hint
(thanks!!).
Hence, adding -fno-devirtualize to c and cpp extra flags solves the
segfault.

However I now have this problem. It also seem to occur with older versios
of gcc, but Those are quite incompatible with the latest arch now, so its
hard to debug.

Linking everything together...
"/usr/bin/avr-gcc" -w -Os -flto -fuse-linker-plugin -Wl,--gc-sections -mmcu=atmega32u4 -o "/tmp/build44e5805ec640d1d5675b7c658d939d94.tmp/SoftwareSerialExample.ino.elf" "/tmp/build44e5805ec640d1d5675b7c658d939d94.tmp/sketch/SoftwareSerialExample.ino.cpp.o" "/tmp/build44e5805ec640d1d5675b7c658d939d94.tmp/libraries/SoftwareSerial/SoftwareSerial.cpp.o" "/tmp/build44e5805ec640d1d5675b7c658d939d94.tmp/core/core.a" "-L/tmp/build44e5805ec640d1d5675b7c658d939d94.tmp" -lm
/tmp/ccNp6vGT.ltrans0.ltrans.o: In function _GLOBAL__sub_I_mySerial': <artificial>:(.text.startup+0x25a): undefined reference to__dso_handle'
:(.text.startup+0x25c): undefined reference to __dso_handle' <artificial>:(.text.startup+0x26e): undefined reference to__cxa_atexit'
/usr/bin/avr-ld: /tmp/build44e5805ec640d1d5675b7c658d939d94.tmp/SoftwareSerialExample.ino.elf: hidden symbol `__dso_handle' isn't defined
/usr/bin/avr-ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

I've searched this online, but I could not find much information about
this. Any ideas?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#660 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADskzL8FDy0-ZiC4LI9sLSKkg-T_xBfRks5qsAjhgaJpZM4AQWJi
.

Visit my github for awesome Arduino code @ https://github.com/xxxajk

[NicoHood]

It is too old? I am using avr-gcc 6.2!?

The problem is caused by the deconstructor. If I disable LTO I get the same long error as above. LTO just gives a segfault then, but this is not the cause. The cause is that the deconstructor does not work for some reason.

[xxxajk]

In that case, it could be any of the following in all 8 combinations:

1. not compiled properly
2. regression
3. optimizer bug

On Tue, Sep 20, 2016 at 12:45 PM, Nico [email protected] wrote:

It is too old? I am using avr-gcc 6.2!?

The problem is caused by the deconstructor. If I disable LTO I get the
same long error as above. LTO just gives a segfault then, but this is not
the cause. The cause is that the deconstructor does not work for some
reason.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#660 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADskzOTmj7h21GjRDYsK1xp6ph3P0jkiks5qsA24gaJpZM4AQWJi
.

Visit my github for awesome Arduino code @ https://github.com/xxxajk

[NicoHood]

The avr-gcc 5.1 from above works fine.

The minimum avr-gcc that I can get working on archlinux is 5.2. 5.1 was never compiled for archlinux. So the options are:

• The bug occured in 5.2
• Archlinux uses different compiler settings.

I will try to find out more about this and edit my findings.

I currently try to recompile 5.3 with the settings that worked before:
https://github.com/NicoHood/AVR-Development-Environment-Script/tree/a617789b7ae592471f45613596ae8b4fad5d6745

[matthijskooijman]

@NicoHood, those symbols look like symbols that libstdc++ or something like that normally supplies, but AVR's libc is a bit lacking in some areas. OTOH, DSO apparently means Dynamic Shared Object, which I don't think should apply here, so perhaps there is some other issue with the compilation options or so?

[NicoHood]

I've got problems recompiling avr-gcc 5.3 with the script above on arch. But I found a backup from my old ubuntu installation. And with this setup, avr-gcc 5.3 works fine! This means archlinux configures avr-gcc somehow wrong.

avr-gcc 5.3 built with the script at this commit on elementary os freya (ubuntu 14.04) but possibly with a newer gcc (4.9.x or 5.x) -> working (tested without lto)

[arch@arch ~]$ avr53/bin/avr-gcc -v
Using built-in specs.
Reading specs from /home/arch/avr53/bin/../lib/gcc/avr/5.3.0/device-specs/specs-avr2
COLLECT_GCC=avr53/bin/avr-gcc
COLLECT_LTO_WRAPPER=/home/arch/avr53/bin/../libexec/gcc/avr/5.3.0/lto-wrapper
Target: avr
Configured with: /home/linuxuser/Downloads/tmp/avrgcc53/gcc-5.3.0/configure --disable-install-libiberty --disable-libssp --disable-libstdcxx-pch --disable-libunwind-exceptions --disable-nls --enable-fixed-point --enable-long-long --disable-werror --disable-__cxa_atexit --enable-checking=release --enable-clocale=gnu --enable-cloog-backend=isl --enable-gnu-unique-object --with-avrlibc=yes --with-dwarf2 --enable-languages=c,c++ --disable-libada --disable-doc --enable-lto --enable-gold --disable-plugin --prefix=/home/linuxuser/Downloads/tmp/avrgcc53/pkg-x86_64-unknown-linux-gnu/ --disable-shared --with-gnu-ld --host=x86_64-unknown-linux-gnu --build=x86_64-unknown-linux-gnu --target=avr
Thread model: single
gcc version 5.3.0 (GCC)

avr-gcc 5.3.0-2 on archlinux (downgraded) -> does not work (tested with and without lto, error as described above)

[arch@arch ~]$ avr-gcc -v
Using built-in specs.
Reading specs from /usr/lib/gcc/avr/5.3.0/device-specs/specs-avr2
COLLECT_GCC=avr-gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/avr/5.3.0/lto-wrapper
Target: avr
Configured with: /build/avr-gcc/src/gcc-5-20160223/configure --disable-install-libiberty --disable-libssp --disable-libstdcxx-pch --disable-libunwind-exceptions --disable-linker-build-id --disable-nls --disable-werror --enable-__cxa_atexit --enable-checking=release --enable-clocale=gnu --enable-gnu-unique-object --enable-gold --enable-languages=c,c++ --enable-ld=default --enable-lto --enable-plugin --enable-shared --infodir=/usr/share/info --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --prefix=/usr --target=avr --with-as=/usr/bin/avr-as --with-gnu-as --with-gnu-ld --with-ld=/usr/bin/avr-ld --with-plugin-ld=ld.gold --with-system-zlib --with-isl --enable-gnu-indirect-function
Thread model: single
gcc version 5.3.0 (GCC) 

[NicoHood]

I've compared the configuration of the above, reordered them and now you can clearly see some major differences:

[xxxajk]

Looks like one of the suggestions was right that I made. :-)
Compiled "wrong" for the expected useage.
Complain upstream to the distro maintainer.

On Tue, Sep 20, 2016 at 1:42 PM, Nico [email protected] wrote:

I've compared the configuration of the above, reordered them and now you
can clearly see some major differences:
[image: screenshot_2016-09-20_19-41-24]
https://cloud.githubusercontent.com/assets/6888294/18681905/574b997e-7f6a-11e6-8395-bae71044fad9.png

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#660 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADskzK3VeLXWIENuqICq01TlsVzLdiUpks5qsBshgaJpZM4AQWJi
.

Visit my github for awesome Arduino code @ https://github.com/xxxajk

[NicoHood]

Bug already filed here:
https://bugs.archlinux.org/task/50848

I am currently trying to recompile with different settings, but recompiling avr-gcc takes even longer than recompiling a kernel (at least inside the vm).

[NicoHood]

I could track down this issue a bit further:

The Issue

<artificial>:(.text.startup+0x25c): undefined reference to `__dso_handle'
<artificial>:(.text.startup+0x26e): undefined reference to `__cxa_atexit'

could be fixed with --disable-__cxa_atexit as configuration switch. (With LTO disabled in platform.txt).

The LTO bug could be disabled with removing -flto from the cpp flags. People in the arch chat suggested to use avr-g++ as linker command. This did not solve anything, but why isnt this used by default?

So there seem to be another LTO related bug (for c++ files). Since @facchinm also had this bug with 6.x I am not sure if this is an avr-gcc 6.x bug or just how we configure avr-gcc. Building older avr-gccs might be tricky on arch.

The LTO bug does not appear with all sketches. CPP In general does work. For example Serial.write() gives no segmentation fault while Serial.print() which uses virtual inhertation does give a segmentation fault. -fno-devirtualize as cpp flag solves this last issue. Any ideas why?

[matthijskooijman]

From your arch bug report, I see that the atexit problem is fixed with newer avr-libc versions (which makes sense, since --enable-__cxa_atexit tells gcc that libc provides the cxa_atexit function, so that is the right place to fix this).

The LTO bug does not appear with all sketches. CPP In general does work. For example Serial.write() gives no segmentation fault while Serial.print() which uses virtual inhertation does give a segmentation fault. -fno-devirtualize as cpp flag solves this last issue. Any ideas why?

That sounds like a compiler bug in the optimizer to me. If you can reduce that to a self-contained c++ file, it should be reported to gcc.