Fix pane info in BigQuery load job id #28272

Abacn · 2023-08-31T22:06:56Z

Fixes #28219

Add randomness to job id

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Abacn · 2023-08-31T22:35:07Z

Run PostCommit_Java_Dataflow

Abacn · 2023-08-31T22:35:14Z

Run PostCommit_Java_DataflowV2

Abacn · 2023-08-31T22:48:10Z

There are still multiple code paths relies on the pane info, e.g. write_disposition and create_disposition settings: https://github.com/apache/beam/blob/3ff66d38c41fde475f71254d889ba46440904238/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java#L194C67-L194C67

There might be other issues after fixing this particular issue. Let me think more about it.

github-actions · 2023-08-31T23:34:44Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

Abacn · 2023-09-05T15:55:43Z

there were test failures due to periodic impulse timing. Re-designed test. If passed then ready to review

Abacn · 2023-09-05T15:57:39Z

.../io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java

-            c.sideInput(jobIdToken), finalTableDestination, -1, c.pane().getIndex());
+            c.sideInput(jobIdToken), finalTableDestination, -1);
+
+    if (isFirstPane) {


this is another code path relying on pane info. Checked that upstream transform has a GBK and no ReShuffle in between: https://github.com/apache/beam/blob/0ed4c78a799cf5a6cc6a0b40b23ca498096769c5/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java#L404C5-L404C5

so this one should be fine. Nevertheless it's good to add logging to monitor

Unrelated to this PR but we're actually seeing a bug due to relying on the pane index in this code path: #28309

Abacn · 2023-09-05T16:08:55Z

Run PostCommit_Java_DataflowV2

Abacn · 2023-09-05T16:09:08Z

Run PostCommit_Java_Dataflow

Abacn · 2023-09-05T19:26:38Z

Although Dataflow PostCommit have multiple quota exceeded failure, bigquery tests all passed: https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV2_PR/218/testReport/org.apache.beam.sdk.io.gcp.bigquery/

Abacn · 2023-09-05T19:30:47Z

R: @ahmedabu98

will fix spotless in next iteration

github-actions · 2023-09-05T19:33:20Z

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

ahmedabu98 · 2023-09-05T19:32:53Z

sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/PeriodicImpulse.java

+   * {@link #startAt} and {@link #stopAt}, as the first timestamp is determined at run time
+   * (pipeline starts processing).
+   */
+  public PeriodicImpulse stopAfter(Duration duration) {


mark with @Internal?

P.S. also add to catchUpToNow()

ahmedabu98 · 2023-09-05T19:44:47Z

...cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryResourceNaming.java

+    // add randomness to jobId to avoid conflict
+    String jobId =
+        String.format("%s_%s_%s", prefix, destinationHash, randomUUIDString().substring(0, 16));


I'm worried this may cause duplication of data due to bundle retry.

Let's say a bundle fails during load/copy execution and the BQ job was successful. Beam would process the bundle again and these lines will create a fresh job ID. Under previous circumstances, this job ID is constructed in a deterministic way and would be recognized by BQ as a recently successful job so will be ignored. But now since the ID is always new, BQ will execute the job and we will end up with duplicate data.

ahmedabu98 · 2023-09-05T19:52:07Z

...ogle-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOWriteIT.java

-    return GenerateSequence.from(0)
-        .to(rowCount)
-        .withRate(1, Duration.millis(timestampIntervalInMilliseconds));
+  static class UnboundedStream extends PTransform<PBegin, PCollection<Long>> {


Nice, this fixes the test cases that were supposed to be streaming but weren't.

ahmedabu98 · 2023-09-05T19:56:23Z

.../io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java

-            c.sideInput(jobIdToken), finalTableDestination, -1, c.pane().getIndex());
+            c.sideInput(jobIdToken), finalTableDestination, -1);
+
+    if (isFirstPane) {


Unrelated to this PR but we're actually seeing a bug due to relying on the pane index in this code path: #28309

Abacn · 2023-09-05T23:32:19Z

Run PostCommit_Java_DataflowV2

Abacn · 2023-09-06T14:09:51Z

dataflow v2 postcommit passed; java precommit known flink test flakiness. PTAL @ahmedabu98

ahmedabu98 · 2023-09-06T16:17:39Z

.../io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteRename.java

+      LOG.info(
+          "Setup write disposition {}, create disposition {} for first pane BigQuery job {}",
+          writeDisposition,
+          createDisposition,
+          jobIdPrefix);
    } else {
-      LOG.debug("Setup write disposition {}, create disposition {} for BigQuery job {}",
-          writeDisposition, createDisposition, jobIdPrefix);
+      LOG.debug(
+          "Setup write disposition {}, create disposition {} for BigQuery job {}",
+          writeDisposition,
+          createDisposition,
+          jobIdPrefix);


Can we include the bigquery table destination here too?

the destination name (without hash) may have PII. In the past I put the logs that may contain PII to debug level so they do not get stored by default (unless override to debug).

(note: final table destination is hased to jobId)

ahmedabu98 · 2023-09-06T20:00:49Z

.../io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java

    abstract Boolean isFirstPane();
+
+    abstract Long getPaneIndex();


Why are we maintaining pane index coming out of write tables? I don't see that we're using it anywhere downstream

will revert the WriteTable.Result changes. Originally it was just to keep WritePartition.Result and WriteTable.Result similar.

Abacn · 2023-09-07T19:21:03Z

.../io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java

+        return partitionResult.isFirstPane();
+      }
+
+      public long paneIndex() {


I am going to preserve the change to PendingJobData, as the pane index info would be helpful to resolve possible racing condition of CREATE_TRUNCATE pending jobs

Do you mean for debugging purposes? Will note that paneIndex() is still not being called anywhere

ahmedabu98 · 2023-09-08T14:09:48Z

Run PostCommit_Java_Dataflow

ahmedabu98 · 2023-09-08T14:10:04Z

Run PostCommit_Java_DataflowV2

ahmedabu98

LGTM if tests pass

kennknowles · 2023-09-11T15:22:51Z

Are you sure this is right? Isn't the purpose of the job id to prevent duplicates?

reuvenlax · 2023-09-12T19:57:34Z

I'm not sure what you mean by random upstream keys. Reshuffle currently maps KV -> KV, so no keys are changed. The only difference is that the keys are now spread out on different workers, which is a semantic no op. You may be thinking of Rehsuffle.viaRandomKey, which is a bit different.

…

On Tue, Sep 12, 2023 at 6:33 AM Kenn Knowles ***@***.***> wrote: Actually each GBK has its own sequence of trigger firings, and when it comes to reshuffle there are two possibilities: 1. The reshuffle does a GBK then explodes it (per its expansion) in which case the sequence of pane indices come from the Reshuffle/always trigger on that GBK. 2. The reshuffle is a semantics "no op" at the PCollection level, in which case the pane indices would be preserved from an upstream aggregation, but reshuffle results in bundles from random upstream keys so they are deceptive. — Reply to this email directly, view it on GitHub <#28272 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAYJVLDCEOGYBQCRHRTBSDX2BQDFANCNFSM6AAAAAA4GWNMFI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

kennknowles · 2023-09-13T00:50:08Z

Right. I was, indeed, thinking of reshuffle via random key. But what I'm saying does apply to Reshuffle by assigned keys. I have been a person who always says that Reshuffle is a semantic noop. If we should preserve that we need the v2 and the SDK implementation to record the pane info (aka reify WindowedValue instead of TimestampedValue). This issue has shaken up my beliefs.

If you have Combine -> (stage) -> Reshuffle. Each input to reshuffle has a pane index that corresponds to its key+window in the combine. The main thing is that if you have the original key+window+index you can reassemble the aggregate and/or write it to a sink in a consistent way. But if the stage changes the key you lose that property. Do we even have a good story for what happens to pane info when a DoFn reassigns to new keys?

kennknowles · 2023-09-13T00:52:10Z

Notably we only have tests like ReshuffleTest.testReshufflePreservesTimestamps

beam/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ReshuffleTest.java

Line 115 in cf0cf3b

public void testReshufflePreservesTimestamps() {

Considering that reifying and testing against a full windowed value would have be somewhat of a default thing to do, I would guess the original intent was very aware of this issue.

* Introduce PeriodicImpulse.stopAfter() * Use it in streaming FILE_LOAD integration test * Fix unit test assert to consider randomness in jobId

Abacn · 2023-09-14T17:46:22Z

rebased onto master to test #28312

Abacn · 2023-09-14T17:48:07Z

Run PostCommit_Java_DataflowV2

github-actions · 2023-09-14T17:59:15Z

Test Results

  1 269 files +  1 217   1 269 suites +1 217 2h 44m 14s ⏱️ - 8h 29m 18s
10 612 tests +10 454 10 544 ✔️ +10 415 68 💤 +39 0 ❌ ±0
10 636 runs +10 478 10 568 ✔️ +10 439 68 💤 +39 0 ❌ ±0

Results for commit 43f014c. ± Comparison against base commit 220cae7.

This pull request removes 158 and adds 10612 tests. Note that renamed tests count towards both.

org.apache.beam.examples.complete.TfIdfIT ‑ testE2ETfIdf
org.apache.beam.examples.complete.TrafficMaxLaneFlowIT ‑ testE2ETrafficMaxLaneFlow
org.apache.beam.examples.complete.TrafficRoutesIT ‑ testE2ETrafficRoutes
org.apache.beam.examples.cookbook.BigQueryTornadoesIT ‑ testE2EBigQueryTornadoesWithExport
org.apache.beam.examples.cookbook.BigQueryTornadoesIT ‑ testE2EBigQueryTornadoesWithExportUsingQuery
org.apache.beam.examples.cookbook.BigQueryTornadoesIT ‑ testE2eBigQueryTornadoesWithStorageApi
org.apache.beam.examples.cookbook.BigQueryTornadoesIT ‑ testE2eBigQueryTornadoesWithStorageApiUsingQuery
org.apache.beam.examples.cookbook.CombinePerKeyExamplesIT ‑ testE2ECombinePerKey
org.apache.beam.examples.cookbook.DistinctExampleIT ‑ testE2EDistinctExample
org.apache.beam.examples.cookbook.FilterExamplesIT ‑ testE2EFilterExamples
…

DefaultPackageTest ‑ defaultPackageInvoker
org.apache.beam.examples.DebuggingWordCountTest ‑ testDebuggingWordCount
org.apache.beam.examples.MinimalWordCountTest ‑ testMinimalWordCount
org.apache.beam.examples.WindowedWordCountIT ‑ testWindowedWordCountInBatchDynamicSharding
org.apache.beam.examples.WindowedWordCountIT ‑ testWindowedWordCountInBatchStaticSharding
org.apache.beam.examples.WindowedWordCountIT ‑ testWindowedWordCountInStreamingStaticSharding
org.apache.beam.examples.WordCountIT ‑ testE2EWordCount
org.apache.beam.examples.WordCountTest ‑ testCountWords
org.apache.beam.examples.WordCountTest ‑ testExtractWordsFn
org.apache.beam.examples.complete.AutoCompleteTest ‑ testAutoComplete[0]
…

This pull request removes 29 skipped tests and adds 68 skipped tests. Note that renamed tests count towards both.

org.apache.beam.sdk.io.gcp.bigquery.FileLoadsStreamingIT ‑ testDynamicDestinationsWithAutoShardingAndCopyJobs[0]
org.apache.beam.sdk.io.gcp.bigquery.FileLoadsStreamingIT ‑ testDynamicDestinationsWithFixedShards[0]
org.apache.beam.sdk.io.gcp.bigquery.FileLoadsStreamingIT ‑ testLoadWithAutoShardingAndCopyJobs[0]
org.apache.beam.sdk.io.gcp.bigquery.FileLoadsStreamingIT ‑ testLoadWithFixedShards[0]
org.apache.beam.sdk.io.gcp.bigquery.StorageApiSinkSchemaUpdateIT ‑ testAtLeastOnceWithAutoSchemaUpdate[0]
org.apache.beam.sdk.io.gcp.bigquery.StorageApiSinkSchemaUpdateIT ‑ testAtLeastOnceWithAutoSchemaUpdate[1]
org.apache.beam.sdk.io.gcp.bigquery.StorageApiSinkSchemaUpdateIT ‑ testAtLeastOnceWithAutoSchemaUpdate[2]
org.apache.beam.sdk.io.gcp.bigquery.StorageApiSinkSchemaUpdateIT ‑ testAtLeastOnceWithIgnoreUnknownValues[0]
org.apache.beam.sdk.io.gcp.bigquery.StorageApiSinkSchemaUpdateIT ‑ testAtLeastOnceWithIgnoreUnknownValues[1]
org.apache.beam.sdk.io.gcp.bigquery.StorageApiSinkSchemaUpdateIT ‑ testAtLeastOnceWithIgnoreUnknownValues[2]
…

org.apache.beam.fn.harness.FnApiDoFnRunnerTest$ExecutionTest ‑ testUsingMetrics
org.apache.beam.runners.core.construction.ReadTranslationTest ‑ testToFromProtoBounded[0: org.apache.beam.sdk.io.CountingSource$UnboundedCountingSource@175cb038]
org.apache.beam.runners.core.construction.ReadTranslationTest ‑ testToFromProtoBounded[3: org.apache.beam.runners.core.construction.ReadTranslationTest$TestUnboundedSource@2ace8c5a]
org.apache.beam.runners.core.construction.ReadTranslationTest ‑ testToFromProtoUnbounded[1: [0, 100)]
org.apache.beam.runners.core.construction.ReadTranslationTest ‑ testToFromProtoUnbounded[2: org.apache.beam.runners.core.construction.ReadTranslationTest$TestBoundedSource@537fd806]
org.apache.beam.runners.dataflow.BatchStatefulParDoOverridesTest ‑ testFnApiMultiOutputOverrideNonCrashing
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest ‑ testMaxThreadMetric[0: [streamingEngine=false]]
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest ‑ testMaxThreadMetric[1: [streamingEngine=true]]
org.apache.beam.runners.dataflow.worker.util.MemoryMonitorTest ‑ uploadJfrProfilesOnThrashing
org.apache.beam.runners.direct.ExecutorServiceParallelExecutorTest ‑ ensureMetricsThreadDoesntLeak
…

♻️ This comment has been updated with latest results.

Abacn · 2023-09-14T23:00:10Z

Tests now exercised and passed on Dataflow Runner V2: https://ci-beam.apache.org/job/beam_PostCommit_Java_DataflowV2_PR/226/testReport/org.apache.beam.sdk.io.gcp.bigquery/FileLoadsStreamingIT/

ahmedabu98 · 2023-09-15T10:33:18Z

@reuvenlax as @Abacn mentioned, we broke update-compatibility recently in #28312 so that we can unblock dynamic destinations with copy jobs (#28309). Do we have the green light to merge these changes?
If Dataflow folks decide Runner V2 should preserve pane indices, that support may still take some time to implement. In the meantime, this ensures file loads streaming works on the runner (IIUC it previously never did)

Abacn · 2023-09-15T18:16:26Z

Run Java PreCommit

reuvenlax · 2023-09-17T21:33:33Z

FYI - https://issues.apache.org/jira/browse/BEAM-7195 was caused by the inverse issue. The old code assumed that Reshuffle would produce pane indices, but of course it just forwards them. Let's wait until we figure out the correct behavior. Very few users will be exercising this from runner v2 today, so I would rather be cautious here - especially if the right answer is to fix runner v2.

…

On Fri, Sep 15, 2023 at 3:33 AM Ahmed Abualsaud ***@***.***> wrote: @reuvenlax <https://github.com/reuvenlax> as @Abacn <https://github.com/Abacn> mentioned, we broke update-compatibility recently in #28312 <#28312> so that we can unblock dynamic destinations with copy jobs (#28309 <#28309>). Do we have the green light to merge these changes? If Dataflow folks decide Runner V2 should preserve pane indices, that support may still take some time to implement. In the meantime, this ensures file loads streaming works on the runner (IIUC it previously never did) — Reply to this email directly, view it on GitHub <#28272 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFAYJVINXQC4WCJC6KPYJB3X2QVHVANCNFSM6AAAAAA4GWNMFI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

kennknowles · 2023-09-19T14:29:45Z

Yea let's sync on this. I think I've come around but I want to get it on dev@ in at least a 1 pager for what the spec is. If it turns out to not be dangerous to reinstate, and we can regain the semantic property "reshuffle is a semantic noop" then that is pretty desirable, so that runners can freely add and remove reshuffles.

Noting that here is where Dataflow v1 implemented reshuffle and re-instated the pre-reshuffle pane:

beam/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingGroupAlsoByWindowReshuffleFn.java

Line 61 in 3402490

item.getPane());

Noting that Beam's reference implementation, the closest thing we have to a spec, does not preserve the pane but will produce a sequence of increasing panes.

Abacn · 2023-09-19T16:02:17Z

@kennknowles thanks for pin to the related code path for Dataflow legacy worker. I now understand that Dataflow legacy work does some special PTransform override for ReShuffle. For portable pipelines, is that true that there is no longer similar PTransform override, and a possible fix would change java core's ReShuffle to also recover pane (currently it recovers Timestamp) ?

beam/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Reshuffle.java

Line 105 in 346541b

.apply("RestoreOriginalTimestamps", ReifyTimestamps.extractFromValues());

kennknowles · 2023-09-19T18:33:10Z

I expect there is a similar override inside the Dataflow service and Unified Worker. The implementation in the SDK would be slow and just for reference, if it was executed. A reshuffle really should not have to cross the Fn API.

kennknowles · 2023-09-19T18:33:49Z

But yes if we make the SDK recover the pane info also, then that would make the SDK (direct runner, prism, etc) match Dataflow v1. This would probably not fix v2, but it would fix "the spec".

kennknowles · 2023-09-26T16:25:18Z

OK so when I thought about it I agree with everyone else that the reference implementation is wrong and that the right behavior is to be the same as the identity function on PCollections. That means keeping the pane info exactly. I wrote the dev list about it. I think it is probably such a boring thread that it will not get a lot of comment.

kennknowles · 2023-09-26T16:27:55Z

.../io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java

+              c.sideInput(loadJobIdPrefixView),
+              tableDestination,
+              partition,
+              element.getValue().getPaneIndex());


My concern is that if we have this:

GBK -> processing -> reshuffle -> this DoFn

Can you tell me which case we are in?

The processing left the key the same so the reshuffle is just for checkpoint

The processing changed the key so that the current element key + pane index is no longer unique

Well it's a little bit more subtle. It is the same key, but became ShardedKey after processing

After GBK, key is DestinationT
After processing, key is ShardedKey (shard is added in WritePartitions)

github-actions · 2023-12-18T12:41:44Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions · 2023-12-26T12:39:50Z

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

github-actions bot added java io runners dataflow gcp labels Aug 31, 2023

Abacn mentioned this pull request Aug 31, 2023

[Bug]: BigQuery IO Batch load using File_load causing the same job id ignoring inserts as the job_id is already completed #28219

Closed

15 tasks

Abacn changed the title ~~Add window to BigQuery load job id~~ Add randomness to BigQuery load job id Sep 1, 2023

yeshvantbhavnasi approved these changes Sep 1, 2023

View reviewed changes

Abacn force-pushed the deduploadjobname branch from 5c40094 to 0ed4c78 Compare September 5, 2023 15:54

Abacn commented Sep 5, 2023

View reviewed changes

ahmedabu98 reviewed Sep 5, 2023

View reviewed changes

Abacn changed the title ~~Add randomness to BigQuery load job id~~ Fix pane info in BigQuery load job id Sep 5, 2023

ahmedabu98 reviewed Sep 6, 2023

View reviewed changes

Abacn commented Sep 7, 2023

View reviewed changes

ahmedabu98 approved these changes Sep 8, 2023

View reviewed changes

Abacn added 4 commits September 14, 2023 13:05

Add randomness to job id

6c6ddb2

* Introduce PeriodicImpulse.stopAfter() * Use it in streaming FILE_LOAD integration test * Fix unit test assert to consider randomness in jobId

Alternate approach - store pane index upstreeam

6aadd9f

Revert change to WriteTables.Result

2a131a8

Revert change to add test - added elsewhere

d1dd0d3

Abacn force-pushed the deduploadjobname branch from ed2bf05 to d1dd0d3 Compare September 14, 2023 17:45

Enable testing runner v2

43f014c

Abacn mentioned this pull request Sep 15, 2023

Comment results to PR only when issue_comments #28480

Merged

3 tasks

Abacn mentioned this pull request Sep 18, 2023

Use unbounded source in BigQueryIOStorageWrite streaming tests #28503

Merged

3 tasks

volatilemolotov mentioned this pull request Sep 21, 2023

update concurrency group definition #28373

Merged

3 tasks

kennknowles reviewed Sep 26, 2023

View reviewed changes

Abacn marked this pull request as draft October 18, 2023 20:01

github-actions bot added the stale label Dec 18, 2023

github-actions bot closed this Dec 26, 2023

		abstract Boolean isFirstPane();

		abstract Long getPaneIndex();

Fix pane info in BigQuery load job id #28272

Fix pane info in BigQuery load job id #28272

Conversation

Abacn commented Aug 31, 2023

GitHub Actions Tests Status (on master branch)

Abacn commented Aug 31, 2023

Abacn commented Aug 31, 2023

Abacn commented Aug 31, 2023 • edited Loading

github-actions bot commented Aug 31, 2023

Abacn commented Sep 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

Abacn commented Sep 5, 2023

github-actions bot commented Sep 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmedabu98 Sep 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Abacn commented Sep 5, 2023

Abacn commented Sep 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ahmedabu98 commented Sep 8, 2023

ahmedabu98 commented Sep 8, 2023

ahmedabu98 left a comment

Choose a reason for hiding this comment

kennknowles commented Sep 11, 2023

reuvenlax commented Sep 12, 2023 via email

kennknowles commented Sep 13, 2023 • edited Loading

kennknowles commented Sep 13, 2023

Abacn commented Sep 14, 2023

Abacn commented Sep 14, 2023

github-actions bot commented Sep 14, 2023 • edited Loading

Test Results

Abacn commented Sep 14, 2023

ahmedabu98 commented Sep 15, 2023

Abacn commented Sep 15, 2023

reuvenlax commented Sep 17, 2023 via email

kennknowles commented Sep 19, 2023

Abacn commented Sep 19, 2023

kennknowles commented Sep 19, 2023

kennknowles commented Sep 19, 2023

kennknowles commented Sep 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Dec 18, 2023

github-actions bot commented Dec 26, 2023

Abacn commented Aug 31, 2023 •

edited

Loading

ahmedabu98 Sep 5, 2023 •

edited

Loading

kennknowles commented Sep 13, 2023 •

edited

Loading

github-actions bot commented Sep 14, 2023 •

edited

Loading