use caffeine in CachedStore and adapt ManagedInternalForm for state p… #3607

thoniTUB · 2024-10-21T17:51:29Z

…ersistence

backend/src/main/java/com/bakdata/conquery/io/storage/xodus/stores/CachedStore.java

backend/src/main/java/com/bakdata/conquery/models/query/ManagedQuery.java

backend/src/main/java/com/bakdata/conquery/apiv1/QueryProcessor.java

backend/src/main/java/com/bakdata/conquery/io/storage/xodus/stores/CachedStore.java

backend/src/main/java/com/bakdata/conquery/io/storage/xodus/stores/XodusStore.java

backend/src/main/java/com/bakdata/conquery/models/auth/basic/LocalAuthenticationRealm.java

backend/src/main/java/com/bakdata/conquery/models/execution/ManagedExecution.java

awildturtok · 2024-10-22T13:33:50Z

backend/src/main/java/com/bakdata/conquery/models/execution/ManagedExecution.java

@@ -258,6 +254,9 @@ public synchronized void finish(ExecutionState executionState) {

 		getExecutionManager().updateState(getId(), executionState);

+		// Persist state of this execution
+		metaStorage.updateExecution(this);


wird finish nicht immer vom execution Manager aufgerufen, so dass der das schon machen sollte?

Ja schon, aber der Zustand, wann eine Execution fertig ist unterscheidet sich zwischen SQL und Distributed, so dass man es hier an einer zentralen stelle abhaken kann. Auch sind wir hier in einen synchronized Block. Den müsste man dann weiter rauszeihen.

backend/src/main/java/com/bakdata/conquery/models/query/ExecutionManager.java

backend/src/main/java/com/bakdata/conquery/models/query/ManagedQuery.java

backend/src/main/java/com/bakdata/conquery/models/query/preview/EntityPreviewForm.java

thoniTUB · 2024-11-08T07:45:17Z

@awildturtok Kannst du bitte nochmal drüber schauen, das Laden der Stores ist jetzt konfigurierbar, wo bei der Bucket und CBlock dennoch bei jedem Start direkt geladen werden durch den BucketManager

Und ich habe den LoadStorageTask abgeändert, so dass er im ManagerNode, ShardNode und Standalone genutzt werden kann.

# Conflicts: # backend/src/main/java/com/bakdata/conquery/commands/ManagerNode.java # backend/src/main/java/com/bakdata/conquery/models/forms/managed/ExternalExecution.java

…ne-cache

awildturtok

Kopf raucht. Bin noch nicht durch.

backend/src/main/java/com/bakdata/conquery/Conquery.java

backend/src/main/java/com/bakdata/conquery/commands/ManagerNode.java

awildturtok · 2025-01-13T10:07:06Z

backend/src/main/java/com/bakdata/conquery/io/storage/ConqueryStorage.java

-	 * @implNote If you implement this method, please do it always from scratch and not using calls to super, it can be quite annoying.
-	 */
-	public abstract ImmutableList<ManagedStore> getStores();
+public interface ConqueryStorage extends Closeable {


Wo hilft es, dass das jetzt ein interface ist?

Dadurch konnte ich einfach den XodusStore einfach ein einen CachedStore wrappen

backend/src/main/java/com/bakdata/conquery/io/storage/NamespacedStorageImpl.java

awildturtok · 2025-01-13T10:41:45Z

backend/src/main/java/com/bakdata/conquery/io/storage/xodus/stores/SerializingStore.java

@@ -385,8 +393,8 @@ public IterationStatistic forEach(StoreEntryConsumer<KEY, VALUE> consumer) {
 				maybeFailed = allJobs.get(30, TimeUnit.SECONDS);
 			}
 			catch (InterruptedException e) {
-				Thread.interrupted();
-				log.debug("Thread was interrupted.");
+				boolean interrupted = Thread.interrupted();


https://stackoverflow.com/questions/10401947/methods-that-clear-the-thread-interrupt-flag

o_O

Sollten wir bei einem Interrupted nicht aufräumen und alles stoppen?

Momentan machen wir ja einfach weiter. Ich denke es wäre am einfachsten die InterruptedException in eine RuntimeException wrappen

Ich teste mit isInterrupted und rethrowe wenn das wirklich so ist

backend/src/main/java/com/bakdata/conquery/io/storage/xodus/stores/XodusStore.java

awildturtok · 2025-01-13T10:43:44Z

backend/src/main/java/com/bakdata/conquery/io/storage/xodus/stores/XodusStore.java

@@ -53,7 +55,8 @@ public ByteIterable get(ByteIterable key) {
 	 *
 	 * @param consumer function called for-each key-value pair.
 	 */
-	public void forEach(BiConsumer<ByteIterable, ByteIterable> consumer) {
+	@Override
+	public SerializingStore.IterationStatistic forEach(StoreEntryConsumer<ByteIterable, ByteIterable> consumer) {
 		AtomicReference<ByteIterable> lastKey = new AtomicReference<>();
 		AtomicBoolean done = new AtomicBoolean(false);
 		while (!done.get()) {


solltest du dann hier nicht auch den iterator verwenden?

Die sind sehr auf den SerializingStore gemünzt. Dass die hier der Return sind ist leaky

backend/src/main/java/com/bakdata/conquery/models/auth/basic/LocalAuthenticationRealm.java

backend/src/main/java/com/bakdata/conquery/models/config/XodusStoreFactory.java

...end/src/main/java/com/bakdata/conquery/models/messages/namespaces/specific/ExecuteQuery.java

backend/src/main/java/com/bakdata/conquery/tasks/LoadStorageTask.java

awildturtok

das binary caching stört mich immer noch aber da wir es mit hoher Wahrscheinlichkeit nicht benutzen werden, ist es mir der diskussion nicht wert

backend/src/test/java/com/bakdata/conquery/integration/tests/FilterResolutionTest.java

backend/src/test/java/com/bakdata/conquery/integration/tests/FilterAutocompleteTest.java

backend/src/test/java/com/bakdata/conquery/integration/tests/MetadataCollectionTest.java

awildturtok · 2025-01-22T15:32:14Z

backend/src/test/java/com/bakdata/conquery/integration/tests/deletion/TableDeletionTest.java

@@ -71,7 +76,10 @@ public void execute(String name, TestConquery testConquery) throws Exception {

 		final Query query = IntegrationUtils.parseQuery(conquery, test.getRawQuery());

-		final long nImports = namespace.getStorage().getAllImports().count();
+		final long nImports;
+		try(Stream<Import> allImports = namespace.getStorage().getAllImports()) {


recherchier mal bitte ob du die streams überhaupt schließen musst, terminating operations sollten den auch schon schließen

Ich habe dazu diesen Kommentar gefunden: https://stackoverflow.com/a/28814024

Es ist wohl unüblich Streams aus releasebaren Resourcen auszugeben. Hier sollten wir vielleicht nochmal die StoreApi ändern.

…ne-cache

…on does not work

…ne-cache

use caffeine in CachedStore and adapt ManagedInternalForm for state p…

8f25433

…ersistence

thoniTUB requested a review from awildturtok October 21, 2024 17:51

thoniTUB added 4 commits October 22, 2024 08:27

Fix serdes problem in LocalAuthenticationRealm

0b96b94

adds test for CachedStore

07e810d

fix race condition in ManagedQuery#lastResultCount

bcf05f6

adds documentation

e283fcd

thoniTUB commented Oct 22, 2024

View reviewed changes

backend/src/main/java/com/bakdata/conquery/io/storage/xodus/stores/CachedStore.java Outdated Show resolved Hide resolved

thoniTUB commented Oct 22, 2024

View reviewed changes

backend/src/main/java/com/bakdata/conquery/models/query/ManagedQuery.java Outdated Show resolved Hide resolved

awildturtok approved these changes Oct 22, 2024

View reviewed changes

thoniTUB added 8 commits October 23, 2024 15:05

pass config on construction not on init

16c4cbb

fix cache invalidation on value update

0ca6358

review changes and fixing the add behaviour

313ed5c

adds metrics

8425e58

fix double metrics recorder

db7a03f

limit sending of RegisterColumnValues jobs with semaphore

26cb189

makes storages load on start configurable

4037f5c

adds task to load specific storages

c161294

thoniTUB marked this pull request as ready for review November 8, 2024 07:37

Merge branch 'develop' into feature/use-caffeine-cache

5532a0a

thoniTUB added 4 commits November 13, 2024 08:07

fix serdes issue with NonPersistentStoreFactory

ad52330

Merge branch 'refs/heads/develop' into feature/use-caffeine-cache

75276ab

# Conflicts: # backend/src/main/java/com/bakdata/conquery/commands/ManagerNode.java # backend/src/main/java/com/bakdata/conquery/models/forms/managed/ExternalExecution.java

Merge remote-tracking branch 'origin/develop' into feature/use-caffei…

efdd9ac

…ne-cache

adds logging for evicted entries

1631fdf

thoniTUB added the backend label Dec 2, 2024

thoniTUB added 5 commits December 4, 2024 08:32

Merge branch 'develop' into feature/use-caffeine-cache

c2227d4

use parallel deserialization in BucketManager

53e8d35

switch back to serial registration of buckets and cblocks

5e6ade9

fix missing Owner relation to ManagedExecutionId and FormConfigId

aad87d9

move direct storage access out of loop

a541b79

thoniTUB added 10 commits December 13, 2024 14:47

improve query plan creation by not resolving buckets

64244ea

improve BucketManager creation time by only loading CBlocks

3a7f482

allow caching of xodus blobs

d674c77

Merge branch 'develop' into feature/use-caffeine-cache

a7f8f7a

fix default option and log

067bb73

fix unfinished transaction finish

9dd290e

renew transactions

5190da1

Merge remote-tracking branch 'origin/develop' into feature/use-caffei…

d0508c4

…ne-cache

defines invalidateCache method on store interface

046f917

extends load storage task by cache invalidation

cb16fae

thoniTUB requested a review from awildturtok January 8, 2025 08:06

awildturtok reviewed Jan 13, 2025

View reviewed changes

awildturtok requested changes Jan 13, 2025

View reviewed changes

thoniTUB added 2 commits January 15, 2025 07:24

review changes

a62661b

cache no binary store by default

0a5c550

thoniTUB requested a review from awildturtok January 22, 2025 07:43

awildturtok approved these changes Jan 22, 2025

View reviewed changes

thoniTUB added 13 commits January 23, 2025 10:54

don't deserialize every bucket upon adding a cblock

f46f757

wrap adding Bucket in Job

7f501fd

Merge branch 'develop' into feature/use-caffeine-cache

ab2f2fb

review changes

6e2e06b

Merge remote-tracking branch 'origin/develop' into feature/use-caffei…

6f502cf

…ne-cache

e2e clear query editor

f8ca20e

don't serialize all buckets for collect column values job

e00266c

e2e reorder upload sequence of test data

564fa3b

Merge branch 'develop' into feature/use-caffeine-cache

86439d8

remove double serdes for worker messages

6f52be4

enhance jackson debug to fall back to hex dump if debug deserializati…

7a2903f

…on does not work

improve logging around network filter and message toString

d4dc983

Merge remote-tracking branch 'origin/develop' into feature/use-caffei…

6329618

…ne-cache

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use caffeine in CachedStore and adapt ManagedInternalForm for state p… #3607

use caffeine in CachedStore and adapt ManagedInternalForm for state p… #3607

thoniTUB commented Oct 21, 2024

awildturtok Oct 22, 2024

thoniTUB Jan 8, 2025

thoniTUB commented Nov 8, 2024

awildturtok left a comment

awildturtok Jan 13, 2025

thoniTUB Jan 15, 2025

awildturtok Jan 13, 2025

thoniTUB Jan 15, 2025

thoniTUB Jan 24, 2025

awildturtok Jan 13, 2025

thoniTUB Jan 15, 2025

awildturtok left a comment

awildturtok Jan 22, 2025

thoniTUB Jan 24, 2025

use caffeine in CachedStore and adapt ManagedInternalForm for state p… #3607

Are you sure you want to change the base?

use caffeine in CachedStore and adapt ManagedInternalForm for state p… #3607

Conversation

thoniTUB commented Oct 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thoniTUB commented Nov 8, 2024

awildturtok left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

awildturtok left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment