Backport of some concepts from util to C++17 #1741

gpicciuca · 2025-01-31T09:57:03Z

Proposing a first bulk of changes aiming to backport some of the concepts used throughout the util module. (Not ready for merging)

There are some places where things are quite intricate and complicated, so feedback would be appreciated.

Problematic areas:

~~SerializeVector.h ends up with an incomplete type when building with C++17 flag~~
ConfigOptionProxy.h seems to not forward the underlying type correctly, causing the ValidatorFunction constraints to fail when using the addValidator method from ConfigOption.h (See commented code in BenchmarkExamples.cpp)
~~A solution has to be found for the isInstantiation constraints imposed in some parts of ConfigOption.h, which I currently had to remove to make it build with v3-ranges.~~

Update:
I have a working cpp20 util module so far with all concepts rewritten using the v3-range macros.
cpp17 version is in progress.

Couple of remarks which I found on the way:

StringUtils.h implements different compile-time string concatenation methods by leveraging the new c++20 template value feature. This is not supported in C++17 and may need some rework. The same goes for the ProgramOptionsHelpers.h which is used in RuntimeConfiguration.
In LambdaHelpers.h I had to resort toQL_CONCEPT_OR_NOTHING, just like in Iterators.h otherwise I'd end up with compile-time errors due to copy and move semantics being deleted (due to the default constructor/methods, which cannot be templated otherwise they need to be explicitly implemented)
In TypeTraits.h we have the type-alias TupleCat which takes variadic template arguments. Problem is that the v3-range concept macros append other stuff at the end of the template<> definition and therefore make the compiler complain about typename... Tuples not being the last element of the list. Edit: Was an easy fix for the TypeTraits.h, but in ConstexprUtils.h we also have cases with variadic arguments which are also causing issues here.

codecov · 2025-01-31T10:42:20Z

Codecov Report

Attention: Patch coverage is 83.63636% with 27 lines in your changes missing coverage. Please review.

Project coverage is 88.63%. Comparing base (949e7db) to head (8b2f5f5).

Files with missing lines	Patch %	Lines
src/util/ConfigManager/ConfigManager.cpp	0.00%	13 Missing ⚠️
src/util/TypeTraits.h	0.00%	7 Missing ⚠️
src/util/http/HttpUtils.h	57.14%	3 Missing ⚠️
src/util/ConfigManager/ConfigManager.h	0.00%	2 Missing ⚠️
src/engine/Server.cpp	0.00%	1 Missing ⚠️
src/util/ConstexprUtils.h	91.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1741      +/-   ##
==========================================
- Coverage   90.04%   88.63%   -1.42%     
==========================================
  Files         396      396              
  Lines       37928    37865      -63     
  Branches     4262     4261       -1     
==========================================
- Hits        34154    33563     -591     
- Misses       2477     3012     +535     
+ Partials     1297     1290       -7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

joka921 · 2025-01-31T15:00:46Z

Thank you very much,
I played around a bit and learned some things.
For the future, please tell me, which targets you compiled (I assume benchmark among others), or whether you manually compiled even single header files etc. Otherwise I also get many errors of things that would have been easy to fix.

I update your branch, so make sure to git pull and have a look at my changes first:

General:

Inside a CPP_template_def you have to use CPP_and_def (which I just implemented) instead of CPP_and
Inside a CPP_template(_def) you have to use CPP_NOT(someConstraint) instead of !someConstraint.
(I will also add those two points to the Wiki)

SerializeVector.h ends up with an incomplete type when building with C++17 flag
I found some issues with the AD_SERIALIZE_FUNCTION_WITH_CONSTRAINT macros (in particular they currently don't support commas inside their arguments, I worked around for some of them, have a look and see if you can fix your problem in a similar way.

ConfigOptionProxy.h seems to not forward the underlying type correctly, causing the ValidatorFunction constraints to fail when using the addValidator method from ConfigOption.h (See commented code in BenchmarkExamples.cpp)
This works now (the problem was, that std::invoke_result_t is not at all SFINAE-friendly, but still works with C++20 concepts. See my changes in TypeTraits.h for details.

A solution has to be found for the isInstantiation constraints imposed in some parts of ConfigOption.h, which I currently had to remove to make it build with v3-ranges.

I have fixed the isInstantiation and (as a proof of concept) readded it to the first occurence in ConfigOption.h, feel free to extend it to the remaining occurences.

I hope that with these changes and suggestions you can continue, please let me know when there is additional trouble, then we can communicate further (probably next week).

gpicciuca · 2025-01-31T15:06:40Z

Thank you very much, I played around a bit and learned some things. For the future, please tell me, which targets you compiled (I assume benchmark among others), or whether you manually compiled even single header files etc. Otherwise I also get many errors of things that would have been easy to fix.

I update your branch, so make sure to git pull and have a look at my changes first:

General:

Inside a CPP_template_def you have to use CPP_and_def (which I just implemented) instead of CPP_and

Inside a CPP_template(_def) you have to use CPP_NOT(someConstraint) instead of !someConstraint.
(I will also add those two points to the Wiki)

SerializeVector.h ends up with an incomplete type when building with C++17 flag
I found some issues with the AD_SERIALIZE_FUNCTION_WITH_CONSTRAINT macros (in particular they currently don't support commas inside their arguments, I worked around for some of them, have a look and see if you can fix your problem in a similar way.

ConfigOptionProxy.h seems to not forward the underlying type correctly, causing the ValidatorFunction constraints to fail when using the addValidator method from ConfigOption.h (See commented code in BenchmarkExamples.cpp)
This works now (the problem was, that std::invoke_result_t is not at all SFINAE-friendly, but still works with C++20 concepts. See my changes in TypeTraits.h for details.

A solution has to be found for the isInstantiation constraints imposed in some parts of ConfigOption.h, which I currently had to remove to make it build with v3-ranges.

I have fixed the isInstantiation and (as a proof of concept) readded it to the first occurence in ConfigOption.h, feel free to extend it to the remaining occurences.

I hope that with these changes and suggestions you can continue, please let me know when there is additional trouble, then we can communicate further (probably next week).

Thank you! For reference, I build the whole engine just to make sure that everything still builds especially since I'm currently working on the utility library which is used throughout a good portion of the codebase.

Will let you know if I spot more problems.

gpicciuca · 2025-02-05T10:41:12Z

Hello @joka921 ,
Updated the PR description with some remarks. Feedback would be appreciated. Thanks!

joka921 · 2025-02-05T13:13:46Z

@gpicciuca
I have just fixed some issues, but others remain:

When using CPP_template the CPP_and must not be inside additional parentheses wrt the requires.
So ... (requires ( something<T> CPP_and something<U>)) does not compile.
Variadic templates (template <typename...>) don't work with CPP_template. I have chosen QL_CONCEPT_OR_NOTHING in those cases.
Same goes for partial specializations (the issues in the json.h.
In the Views.h You have to use CPP_ret because you can't use CPP_template in member functions, when the outer templated class already uses CPP_template (Due to the internals of this macro). The disadvantage is, that for CPP_ret you need the exact return type. This is something where you could continue, I have started with the first occurence.

TLDR:
With things I have pushed you can compile the util library, The next step would be to compile ViewsTest .

In general you should work on changing one thing, and then making sure that it compiles in both modes, currently I am doing a lot of cleanup in different places where it would be possible to discuss the occuring problems in advance, but we are all just learning to use these macros and their limitations (I have also learned several things about template metaprogramming in the past week).
Let me know, when you are working on your branch, so we won't get in conflict.

gpicciuca · 2025-02-05T13:19:40Z

@gpicciuca I have just fixed some issues, but others remain:

When using CPP_template the CPP_and must not be inside additional parentheses wrt the requires.
So ... (requires ( something<T> CPP_and something<U>)) does not compile.

Variadic templates (template <typename...>) don't work with CPP_template. I have chosen QL_CONCEPT_OR_NOTHING in those cases.

Same goes for partial specializations (the issues in the json.h.

In the Views.h You have to use CPP_ret because you can't use CPP_template in member functions, when the outer templated class already uses CPP_template (Due to the internals of this macro). The disadvantage is, that for CPP_ret you need the exact return type. This is something where you could continue, I have started with the first occurence.

TLDR: With things I have pushed you can compile the util library, The next step would be to compile ViewsTest .

In general you should work on changing one thing, and then making sure that it compiles in both modes, currently I am doing a lot of cleanup in different places where it would be possible to discuss the occuring problems in advance, but we are all just learning to use these macros and their limitations (I have also learned several things about template metaprogramming in the past week). Let me know, when you are working on your branch, so we won't get in conflict.

Noted. All those issues are popping out as I build with the cpp17 flag. Easy fix.
I found a better solution for that which keeps the constraints in place for both builds. E.g.:

template <typename... Tuples>
using TupleCat =
   typename std::enable_if<(isTuple<Tuples> && ...),
                           decltype(std::tuple_cat(
                               std::declval<Tuples&>()...))>::type;

A bit more verbose, but it works.
2. For the json.h file I also have a working fix in place without sacrificing constraints. E.g. this one was giving me build errors but works now this way since the base struct definition allows for the second template argument

template <typename T>
struct adl_serializer<std::unique_ptr<T>,
                      std::enable_if_t<std::is_copy_constructible_v<T>>> {

Was about to experiment with that one. Thanks.

gpicciuca · 2025-02-06T15:48:15Z

@joka921
I pushed few more changes among which there are a few small fixes that still make sure constraints are applied even in C++17 (where you went for QL_CONCEPT_... etc).

The changes in Views.h and Iterator.h are somewhat work in progress as I've been experimenting there but still no luck in getting through the errors. ViewsTest are still failing to compile even after applying your suggested approach with CPP_ret. Seems like this method actually changes how constraints are applied to the method. I also tried templating all of those methods applying SFINAE techniques with and without the v3-ranges macros, but still fails.
The issue is in how const-qualifications are propagated through OwningView, BufferedAsyncView, InputRangeMixin. For yet unknown reasons the const qualifications break after applying above mentioned changes, so it tries to call into a const begin() which it can't find.

Will continue working on this next week. Feel free to have a look in the meantime if you have time.

joka921 · 2025-02-07T11:26:41Z

Okay, I have brought this back to compilation in both modes, but it took me quite a lot of time.
I reverted as part of my work most of the concepts back to concept from CPP_concept, as there were a lot of compilation errors, a few of them because of subtleties (for which it is absolutely fine to pass them on to me) and most of them because of "the necessary work was not yet done, but there was nothing special about it". I below propose a plan to mitigate the remaining issues in a productive way for the both of us:

Please continue as follows:

Pick a single concept (start e.g. in TypeTraits.h) and change it to CPP_concept.
Change all the places where this concept is used, s.t. both modes (C++20 and 17 ) compile again.
IMPORTANT. If you don't suceed for some places, because the usage is non-trivial or messy, change all the trivial places and then revert the concept back to concept, such that the code compiles again.
Repeat steps 1-3 for other concepts.

At some point, contact me again with the list of concepts where issues remain (the ones from point 3.) such that I can have a look at the more advanced techniques that we might need there.

Let me know, if I can be of assistance or if there are additional questions.

gpicciuca · 2025-02-10T09:47:57Z

Ignore the force-push I just did please. Rebase went wrong. Working on it.

gpicciuca · 2025-02-10T10:12:29Z

Rebase solved. Both modes build just fine with this last commit. Tackling each concept now individually to not end up in a dead-end again.

joka921 · 2025-02-10T15:31:50Z

@gpicciuca
I am just seeing that you do a lot of manual rewriting using std::enable_if_t instead of using the range-v3 macros as discussed.
In some cases there are additional macros that you can use if the plain CPP_template(...) doesn't work.
Please let me know if you run into trouble before you do a lot of stuff that then has to be reverted again.

gpicciuca · 2025-02-11T08:08:22Z

@gpicciuca I am just seeing that you do a lot of manual rewriting using std::enable_if_t instead of using the range-v3 macros as discussed. In some cases there are additional macros that you can use if the plain CPP_template(...) doesn't work. Please let me know if you run into trouble before you do a lot of stuff that then has to be reverted again.

I'm going through those enable_if_t templates again.
Some places I cannot simply rewrite with the macros, like the changes i made in benchmark/JoinAlgorithmBenchmark.cpp. It says you cannot define templates in block-scope. But the lambda per-se can be templated though..

One thing worth mentioning is that with the v3-ranges macros, you cannot have variadic arguments there.

For example:
In src/engine/sparqlExpressions/SparqlExpressionTypes.h

template <size_t NumOperands,
          QL_CONCEPT_OR_TYPENAME(
              ad_utility::isInstantiation<FunctionAndValueGetters>)
              FunctionAndValueGettersT,
          QL_CONCEPT_OR_TYPENAME(ad_utility::isInstantiation<
                                 SpecializedFunction>)... SpecializedFunctions>
struct Operation {

In these cases (and there are quite a few), we cannot even apply the normal C++17 sfinae techniques as you'd have to specify the enable_if_t at the end of the template argument list (meaning even after the variadic args).

Another example, although just a test:
In test/SparqlAntlrParserTest.cpp

auto matchNary(
    auto makeFunction,
    QL_CONCEPT_OR_NOTHING(ad_utility::SimilarTo<Variable>) auto&&... variables)
    -> ::testing::Matcher<const sparqlExpression::SparqlExpression::Ptr&> {

The CPP_template macro would append some other internal arguments at the end of your manually specified template args, leading to an invalid syntax which the compiler is unhappy about.

Signed-off-by: Johannes Kalmbach <[email protected]>

sparql-conformance · 2025-02-11T09:27:03Z

Conformance check passed ✅

No test result changes.

Details: https://qlever.cs.uni-freiburg.de/sparql-conformance-ui?cur=8b2f5f58e03f61e6202c00d7fac671677db019ed&prev=5caf9f49cbfe5f469721ed15dc988628abd7b307

sonarqubecloud · 2025-02-11T10:46:19Z

Quality Gate passed

Issues
40 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

joka921

I have made a thorough pass. I have marked some nasty details which I will look at myself first. Thanks for the tedious work so far,
I'll get back to you once I have addressed the things I want to change myself.

joka921 · 2025-02-11T11:56:47Z

src/backports/concepts.h

@@ -53,3 +55,25 @@ using namespace std;

 }  // namespace concepts
 }  // namespace ql
+
+// A template with a requires clause
+/// INTERNAL ONLY


I have updated versions of those, I will merge them in here myself later today or tomorrow.

joka921 · 2025-02-11T12:04:07Z

src/global/Pattern.h

@@ -71,6 +72,40 @@ struct CompactStringVectorWriter;

 }

+// TODO<joka921, picciuca> : The following doesn't work if there is no `begin()`


Just noting this TODO for later, s.t. I can have a look at it again.

joka921 · 2025-02-11T12:09:57Z

src/index/VocabularyMergerImpl.h

                                        decltype(sizeOfQueueWord)>(
-          0.8 * memoryToUse, generators, lessThanForQueue);
+          0.8 * memoryToUse, std::move(generators), lessThanForQueue);


Marking this for later to have a look at it.

joka921 · 2025-02-11T13:15:54Z

src/util/CancellationHandle.h

+  template <typename U = void>
+  std::enable_if_t<WatchDogEnabled, U> startWatchDogInternal();


I'll have a look at how we can implement those withouth std::enable_if_t , we'll need some of my new macros probably.

joka921 · 2025-02-11T13:34:06Z

src/util/Iterators.h

+  // TODO<joka921> play with a better solution for defaulting constructors.
+  CPP_template(typename X = void)(
+      requires std::is_default_constructible_v<Accessor>)
+      IteratorForAccessOperator() {}


Pointing out this TODO for later.

joka921 · 2025-02-11T14:19:05Z

src/util/http/HttpServer.h

+  // Note: The following code can not be written with a single declaration in
+  // the `std::enable_if_t` world, because of the following bug in GCC 11:
+  // https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105268
+  // TODO<joka921> Figure out whether this is also an issue on older GCC
+  // versions for which we need this port.
+  template <typename HandlerSupplier>
+  static constexpr bool isValidConstructorArgument =
+      ad_utility::InvocableWithConvertibleReturnType<
+          HandlerSupplier, WebSocketHandler, net::io_context&>;
+  template <
+      typename HandlerSupplier,
+      std::enable_if_t<isValidConstructorArgument<HandlerSupplier>, int> = 0>


Also a thing which I'll have a look at

joka921 · 2025-02-11T14:25:37Z

src/util/json.h

@@ -37,6 +37,7 @@ Convenience header for Nlohmann::Json that sets the default options. Also
 #include "util/SourceLocation.h"

 // For higher flexibility of the custom json helper functions.
+// TODO<joka921, gpicciuca> Make `CPP_concept` again.


Maybe we can address this right now:)

joka921 · 2025-02-11T14:27:22Z

test/ConfigManagerTest.cpp

@@ -42,8 +42,13 @@

 using namespace std::string_literals;

+// TODO<joka921, gpicciuca>, some of those currently don't compile, but we first
+// take care of more important things.
+#if false


I'll have a look at those

joka921 · 2025-02-11T14:28:07Z

test/LambdaHelpersTest.cpp

-  static_assert(!std::is_copy_assignable_v<decltype(hallo)>);
+  // TODO<joka921> This has to be changed, because some concept checks here are
+  // currently disabled.
+  // static_assert(!std::is_copy_assignable_v<decltype(hallo)>);


Another TODO

joka921 · 2025-02-11T14:28:17Z

test/ParametersTest.cpp

@@ -91,8 +91,13 @@ TEST(Parameters, ParameterConcept) {
      IsParameter<DurationParameter<std::chrono::seconds, "Seconds">>);

  // Test some other random types.
+  // TODO<joka921, gpicciuca> This constrained is currently deactivated.
+  static_assert(IsParameter<std::string>);


gpicciuca force-pushed the cpp17-concepts-backport-2 branch from e50ec72 to 0949690 Compare January 31, 2025 10:03

gpicciuca force-pushed the cpp17-concepts-backport-2 branch from 8878380 to 1770f2c Compare February 10, 2025 09:35

gpicciuca force-pushed the cpp17-concepts-backport-2 branch from 1770f2c to 5a83a04 Compare February 10, 2025 10:11

gpicciuca and others added 12 commits February 11, 2025 09:49

Backport of some concepts from util to C++17

2ddfca9

First steps that are working.

d58b3de

Signed-off-by: Johannes Kalmbach <[email protected]>

Also fixed the isInstantion occurence.

0a1bfc0

Signed-off-by: Johannes Kalmbach <[email protected]>

Working cpp20 build with util module concepts backport

f844150

Fix some bugs.

a61f539

Signed-off-by: Johannes Kalmbach <[email protected]>

Small fixes and Views and Iterators backporting

c9511e5

Fix the ViewsTest for now.

cb679e1

Signed-off-by: Johannes Kalmbach <[email protected]>

Bring this back to compilation

ad90c03

Signed-off-by: Johannes Kalmbach <[email protected]>

Backport concept isInstantiation

a035e0a

Backport TypeTraits Similar-concepts

c4af1cc

Backport BulkResultForDecoder

cb610e9

Rewrite enable_if_t occurrences

8b2f5f5

gpicciuca force-pushed the cpp17-concepts-backport-2 branch from dc60f38 to 8b2f5f5 Compare February 11, 2025 08:53

gpicciuca marked this pull request as ready for review February 11, 2025 08:55

joka921 requested changes Feb 11, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backport of some concepts from util to C++17 #1741

Backport of some concepts from util to C++17 #1741

gpicciuca commented Jan 31, 2025 •

edited

Loading

codecov bot commented Jan 31, 2025 •

edited

Loading

joka921 commented Jan 31, 2025

gpicciuca commented Jan 31, 2025

gpicciuca commented Feb 5, 2025

joka921 commented Feb 5, 2025

gpicciuca commented Feb 5, 2025 •

edited

Loading

gpicciuca commented Feb 6, 2025 •

edited

Loading

joka921 commented Feb 7, 2025

gpicciuca commented Feb 10, 2025

gpicciuca commented Feb 10, 2025

joka921 commented Feb 10, 2025

gpicciuca commented Feb 11, 2025 •

edited

Loading

sparql-conformance bot commented Feb 11, 2025

sonarqubecloud bot commented Feb 11, 2025

joka921 left a comment

joka921 Feb 11, 2025

joka921 Feb 11, 2025

joka921 Feb 11, 2025

joka921 Feb 11, 2025

joka921 Feb 11, 2025

joka921 Feb 11, 2025

joka921 Feb 11, 2025

joka921 Feb 11, 2025

joka921 Feb 11, 2025

joka921 Feb 11, 2025

		@@ -71,6 +72,40 @@ struct CompactStringVectorWriter;

		}

		// TODO<joka921, picciuca> : The following doesn't work if there is no `begin()`

		template <typename U = void>
		std::enable_if_t<WatchDogEnabled, U> startWatchDogInternal();

Backport of some concepts from util to C++17 #1741

Are you sure you want to change the base?

Backport of some concepts from util to C++17 #1741

Conversation

gpicciuca commented Jan 31, 2025 • edited Loading

codecov bot commented Jan 31, 2025 • edited Loading

Codecov Report

joka921 commented Jan 31, 2025

gpicciuca commented Jan 31, 2025

gpicciuca commented Feb 5, 2025

joka921 commented Feb 5, 2025

gpicciuca commented Feb 5, 2025 • edited Loading

gpicciuca commented Feb 6, 2025 • edited Loading

joka921 commented Feb 7, 2025

gpicciuca commented Feb 10, 2025

gpicciuca commented Feb 10, 2025

joka921 commented Feb 10, 2025

gpicciuca commented Feb 11, 2025 • edited Loading

sparql-conformance bot commented Feb 11, 2025

Conformance check passed ✅

sonarqubecloud bot commented Feb 11, 2025

Quality Gate passed

joka921 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gpicciuca commented Jan 31, 2025 •

edited

Loading

codecov bot commented Jan 31, 2025 •

edited

Loading

gpicciuca commented Feb 5, 2025 •

edited

Loading

gpicciuca commented Feb 6, 2025 •

edited

Loading

gpicciuca commented Feb 11, 2025 •

edited

Loading