bulkio: follow through on random testing #72345
Assignees
Labels
A-disaster-recovery
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-disaster-recovery
Comments
shermanCRL
added
the
C-enhancement
|
cc @cockroachdb/bulk-io |
|
cc @cockroachdb/bulk-io |
msbutler
added a commit
to msbutler/cockroach
that referenced
this issue
Jan 28, 2022
PopulateRandTable populates the caller's table with random data. This helper function aims to make it easier for engineers to develop randomized tests that leverage randgen / sqlsmith. Informs cockroachdb#72345 Release note: None
msbutler
added a commit
to msbutler/cockroach
that referenced
this issue
Jan 31, 2022
PopulateRandTable populates the caller's table with random data. This helper function aims to make it easier for engineers to develop randomized tests that leverage randgen / sqlsmith. Informs cockroachdb#72345 Release note: None
msbutler
added a commit
to msbutler/cockroach
that referenced
this issue
Feb 3, 2022
PopulateRandTable populates the caller's table with random data. This helper function aims to make it easier for engineers to develop randomized tests that leverage randgen / sqlsmith. I considered adding random insert statements into sqlsmith's randtables setup, however the high probably of a faulty insert statement would cause the whole setup to fail. See cockroachdb#75159 Informs cockroachdb#72345 Release note: None
msbutler
added a commit
to msbutler/cockroach
that referenced
this issue
Feb 9, 2022
PopulateRandTable populates the caller's table with random data. This helper function aims to make it easier for engineers to develop randomized tests that leverage randgen / sqlsmith. I considered adding random insert statements into sqlsmith's randtables setup, however the high probably of a faulty insert statement would cause the whole setup to fail. See cockroachdb#75159 In the future, I'd like to develop a new helper function PopulateDatabaseWithRandData which calls PopulateTableWithRandData on each table in the order of the fk dependency graph. Informs cockroachdb#72345 Release note: None
craig bot
pushed a commit
that referenced
this issue
Feb 15, 2022
74563: kv,kvcoord,sql: poison txnCoordSender after a retryable error r=lidorcarmel a=lidorcarmel Previously kv users could lose parts of a transaction without getting an error. After Send() returned a retryable error the state of txn got reset which made it usable again. If the caller ignored the error they could continue applying more operations without realizing the first part of the transaction was discarded. See more details in the issue (#22615). The simple case example is where the retryable closure of DB.Txn() returns nil instead of returning the retryable error back to the retry loop - in this case the retry loop declares success without realizing we lost the first part of the transaction (all the operations before the retryable error). This PR leaves the txn in a "poisoned" state after encountering an error, so that all future operations fail fast. The caller is therefore expected to reset the txn handle back to a usable state intentionally, by calling Txn.PrepareForRetry(). In the simple case of DB.Txn() the retry loop will reset the handle and run the retry even if the callback returned nil. Closes #22615 Release note: None 74662: tsdb: expand mem per worker based on sql pool size r=dhartunian a=dhartunian Previously, the memory limit for all `tsdb` workers was set at a static 64MiB. This cap created issues seen in #24018 where this limit was hit on a 30 node cluster. To alleviate the issue, the number of workers was reduced, raising the per-worker allocation. We've currently hit this limit again as part of load testing with larger clusters and have decided to make the per-query worker memory limit dynamic. The per-worker limit is now raised based on the amount of memory available to the SQL Pool via the `MemoryPoolSize` configuration variable. This is set to be 25% of the system memory by default. The `tsdb` memory cap per-worker is now doubled until it reaches `1/128` of the memory pool setting. For example, on a node with 128 - 256 GiB of memory, this will correspond to 512 MiB allocated for all running `tsdb` queries. In addition, the ts server is now connected to the same `BytesMonitor` instance as the SQL memory monitor and workers will becapped at double the query limit. Results are monitored as before but a cap is not introduced there since we didn't have one present previously. This behavior is gated behind a private cluster setting that's enabled by default and sets the ratio at 1/128 of the SQL memory pool. Resolves #72986 Release note (ops change): customers running clusters with 240 nodes or more can effectively access tsdb metrics. 75677: randgen: add PopulateRandTable r=mgartner a=msbutler PopulateRandTable populates the caller's table with random data. This helper function aims to make it easier for engineers to develop randomized tests that leverage randgen / sqlsmith. Informs #72345 Release note: None 76334: opt: fix missing filters after join reordering r=mgartner a=mgartner #### opt: add TES, SES, and rules to reorderjoins This commit updates the output of the `reorderjoins` opt test command to display the initial state of the `JoinOrderBuilder`. It adds additional information to the output including the TES, SES, and conflict rules for each edge. Release note: None #### opt: fix missing filters after join reordering This commit eliminates logic in the `assoc`, `leftAsscom`, and `rightAsscom` functions in the join order builder that aimed to prevent generating "orphaned" predicates, where one or more referenced relations are not in a join's input. In rare cases, this logic had the side effect of creating invalid conflict rules for edges, which could prevent valid predicates from being added to reordered join trees. It is safe to remove these conditionals because they are unnecessary. The CD-C algorithm already prevents generation of orphaned predicates by checking that the total eligibility set (TES) is a subset of a join's input vertices. In our implementation, this is handled by the `checkNonInnerJoin` and `checkInnerJoin` functions. Fixes #76522 Release note (bug fix): A bug has been fixed which caused the query optimizer to omit join filters in rare cases when reordering joins, which could result in incorrect query results. This bug was present since v20.2. Co-authored-by: Lidor Carmel <[email protected]> Co-authored-by: David Hartunian <[email protected]> Co-authored-by: Michael Butler <[email protected]> Co-authored-by: Marcus Gartner <[email protected]>
RajivTS
pushed a commit
to RajivTS/cockroach
that referenced
this issue
Mar 6, 2022
PopulateRandTable populates the caller's table with random data. This helper function aims to make it easier for engineers to develop randomized tests that leverage randgen / sqlsmith. I considered adding random insert statements into sqlsmith's randtables setup, however the high probably of a faulty insert statement would cause the whole setup to fail. See cockroachdb#75159 In the future, I'd like to develop a new helper function PopulateDatabaseWithRandData which calls PopulateTableWithRandData on each table in the order of the fk dependency graph. Informs cockroachdb#72345 Release note: None
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This PR introduced welcome randomness to BACKUP/RESTORE. A couple of follow-through ideas:
FYI @jordanlewis
Epic CRDB-10683
Jira issue: CRDB-11103
The text was updated successfully, but these errors were encountered: