Generated code has multiple levels of indirection via macros

[mexicowilly]

I really appreciate the work you've done on this project, so this is not an existential concern. It's just that when generated code has macros that refer to macros that refer to macros, it becomes very difficult to make sense of it. Since the code is generated, wouldn't it be easier to read and use if all the macros were expanded? After all, this is generated code.

[mikkelfj]

Hi, if anyone, I am aware of how difficult it is to work with these macros.

Note that macros used in flatcc are of a code generating nature, they are not direct macro expressions. This means that type safe inline functions are produced. It would be good with some autogenerated documentation for that output - and contributions are welcome, but it is not easy to do properly in a maintainable fashion.

If you want to see the expanded output this is easy to achieve, while it is next to impossible to see the reason behind the generated code if it was expanded directly.

Enter include/flatcc/reflection and run the following, or use a similar tool for your platform:

clang -E -DNDEBUG reflection_reader.h -I ../.. | clang-format

Over time the tendency has become to move as much as possible into macros such that each field in the output only produces one, or a few lines. This avoids huge code repositories when checking in source and makes it easy to inspect them, especially for differences.

The layering of macros has several purposes - some of it is inheritance to ensure, for example, that all vectors have certain operations and addition that strings has some special string operations. This could of course be generated directly, but generated code is even messier because you have to think in both native code, generated code and in the macros that still exists.

Another purpose of layering is the limitations in the C preprocessors limitation in name expansions which yields non-intuitive expansion rules some of which require double expansion.

Name space prefixes are also more easily handled in macros - this makes it is possible to build FlatBuffer variants with different sizes using different names, but this is not fully supported. The geenrating code still has lots of namespace printing going on.

There has been a few problems with name clashes between macro parameter and local names in the generated functions during macro expansion. This has been mostly fixed in the reader which is why you see a lot of underscores in tmp variables. The builder less so, but for some reason it does not tend to cause collisions as easily. Collisions can happen due to the multiple layering where a name is injected into a macro depending on how expansion works for that macro, and it is nearly impossible to predict. But on a balance, everything seems to work ok.

Most important is that the result is type safe and generates compile time errors if anything goes wrong.

[mikkelfj]

Here is a zip of the expanded reflection_reader.h

reflection_reader_expanded.zip

[mikkelfj]

Note that the file sizes are, in bytes:

• expanded: 335140
• expanded and formatted: 385788
• expanded formatted source without included content: 219937 (57% of total size)
• source header: 14750

The system header files, the flatcc header files and the common reader header consumes about 43% and the schema specific content about 57%. So if this were directly expanded, the code would be fairly large.

The compiler immediately strips out most of this content because the content is static or static inline functions that are unreferenced, leaving only the parts that the end user program actually uses, and optimizes this quite efficiently.

[mikkelfj]

Added a documentation section: https://github.com/dvidelabs/flatcc#use-of-macros-in-generated-code

[mikkelfj]

EDIT: this does not capture all output - a more complete command will follow

The documentation also has the following example added here for completeness:

cd include/flatcc/reflection
clang -E -DNDEBUG reflection_reader.h -I ../.. | \
clang-format -style="WebKit" | \
grep "static inline reflection_Field_" | \
cut -d '{' -f 1 \
> Field.txt

Resulting in Field.txt holding the following content:

static inline reflection_Field_table_t reflection_Field_vec_at(reflection_Field_vec_t vec, size_t i)
static inline reflection_Field_table_t reflection_Field_as_root_with_identifier(const void* buffer__tmp, const
char* fid__tmp) static inline reflection_Field_table_t
reflection_Field_as_root_with_type_hash(const void* buffer__tmp, flatbuffers_thash_t thash__tmp)
static inline reflection_Field_table_t reflection_Field_as_root(const void* buffer__tmp)
static inline reflection_Field_table_t reflection_Field_as_typed_root(const void* buffer__tmp)
static inline reflection_Field_vec_t reflection_Object_fields_get(reflection_Object_table_t t__tmp)
static inline reflection_Field_vec_t reflection_Object_fields(reflection_Object_table_t t__tmp)

[mikkelfj]

Added scripts/flatcc-doc.sh, see

https://github.com/dvidelabs/flatcc#extracting-documentation

[mexicowilly]

Thanks very much for the detailed information. That helps me understand. Also, the script for extracting the API is extremely helpful. Thank you.

[mikkelfj]

Thanks, yes I missed that script myself, it was long overdue.