Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: fix flakes in npgsql test #108414

Open
rafiss opened this issue Aug 9, 2023 · 0 comments
Open

roachtest: fix flakes in npgsql test #108414

rafiss opened this issue Aug 9, 2023 · 0 comments
Labels
A-sql-pgwire pgwire protocol issues. A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)

Comments

@rafiss
Copy link
Collaborator

rafiss commented Aug 9, 2023 •
edited by exalate-issue-sync bot
Loading

Describe the problem

Many tests in the npgsql nightly roachtest are ignored, since they all have a common way of flaking. An example is:

  Failed Cursor_move_RecordsAffected [2 ms]
  Error Message:
   Npgsql.NpgsqlException : Received backend message ReadyForQuery while expecting ParseCompleteMessage. Please file a bug.
  Stack Trace:
     at Npgsql.Util.Statics.ThrowIfMsgWrongType[T](IBackendMessage msg, NpgsqlConnector connector) in /mnt/data1/npgsql/src/Npgsql/Util/PGUtil.cs:line 49
   at Npgsql.Util.Statics.Expect[T](IBackendMessage msg, NpgsqlConnector connector) in /mnt/data1/npgsql/src/Npgsql/Util/PGUtil.cs:line 32
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken) in /mnt/data1/npgsql/src/Npgsql/NpgsqlDataReader.cs:line 451
   at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken) in /mnt/data1/npgsql/src/Npgsql/NpgsqlDataReader.cs:line 608
   at Npgsql.NpgsqlDataReader.NextResult() in /mnt/data1/npgsql/src/Npgsql/NpgsqlDataReader.cs:line 342
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) in /mnt/data1/npgsql/src/Npgsql/NpgsqlCommand.cs:line 1466
   at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) in /mnt/data1/npgsql/src/Npgsql/NpgsqlCommand.cs:line 1542
   at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken) in /mnt/data1/npgsql/src/Npgsql/NpgsqlCommand.cs:line 1177
   at Npgsql.NpgsqlCommand.ExecuteNonQuery() in /mnt/data1/npgsql/src/Npgsql/NpgsqlCommand.cs:line 1159
   at Npgsql.Tests.CommandTests.Cursor_move_RecordsAffected() in /mnt/data1/npgsql/test/Npgsql.Tests/CommandTests.cs:line 485
   at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.GetResult()
   at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.RunTestMethod(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context)
   at NUnit.Framework.Internal.Commands.TimeoutCommand.<>c__DisplayClass5_0.<RunTestOnSeparateThread>b__0()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at NUnit.Framework.Internal.Commands.TimeoutCommand.Execute(TestExecutionContext context)

Determine what causes that, and stop ignoring these upstream tests in this list:

cockroach/pkg/cmd/roachtest/tests/npgsql_blocklist.go

Line 747 in a3a389e

var npgsqlIgnoreList = blocklist{

Jira issue: CRDB-30454

Epic CRDB-43310
@rafiss rafiss added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label Aug 9, 2023
@rafiss rafiss mentioned this issue Aug 9, 2023
Merged
@rafiss rafiss added the T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) label Aug 9, 2023
@rafiss rafiss added A-testing Testing tools and infrastructure A-sql-pgwire pgwire protocol issues. labels Aug 9, 2023
craig bot pushed a commit that referenced this issue Aug 10, 2023
@ericharmeling @rafiss @jbowens
Merge #107388 #108415 #108535
618c73a 
107388: metrics: refactor histogram bucket generation and testing r=ericharmeling a=ericharmeling

This commit refactors histogram bucketing for legibility and composibility. It also introduces a data-driven test for histogram bucket generation.

This refactor should make it easier to add additional metric categories, distributions, and bucket types.

Part of #97144.

Release note: None

108415: roachtest: deflake npgsql test r=rafiss a=rafiss

These upstream tests are flaky, so we ignore them.

informs #108414
fixes #108044
fixes #108504
Release note: None

108535: roachtest: remove rangeTs variants of import-cancellation test r=stevendanna a=jbowens

Previously the import-cancellation roachtest was split into two variants: one with MVCC range tombstones enabled and one without. MVCC range tombstones are always enabled now, so the two variants were effectively identical. This commit consolidates the two tests into a single `import-cancellation` roachtest.

Informs #97869.
Epic: None
Release note: None

Co-authored-by: Eric Harmeling <[email protected]>
Co-authored-by: Rafi Shamim <[email protected]>
Co-authored-by: Jackson Owens <[email protected]>
@rafiss rafiss mentioned this issue Aug 11, 2023
Merged
@rafiss rafiss mentioned this issue Oct 31, 2023
Merged
rafiss added a commit to rafiss/cockroach that referenced this issue Oct 31, 2023
@rafiss
roachtest: mark new npgsql tests as flaky
7af8852 
These tests started flaking due to cockroachdb#108414. We will mark them as flaky
and investigate later.

Release note: None
craig bot pushed a commit that referenced this issue Nov 1, 2023
@rafiss @yuzefovich
Merge #113542 #113550 #113553
63e0b12 
113542: roachtest: mark new npgsql tests as flaky r=rafiss a=rafiss

These tests started flaking due to #108414. We will mark them as flaky and investigate later.

fixes #112686

Release note: None

113550: server/profiler: mark ActiveQueryDumpsEnabled as SystemVisible r=yuzefovich a=yuzefovich

This setting was miscategorized initially - the active query profiler runs in both storage and virtual clusters, so it should be at least SystemVisible, but maybe even ApplicationLevel.

Addresses: #113170.
Epic: None

Release note: None

113553: cli: create new tracing span in mt-start-sql r=yuzefovich a=yuzefovich

This commit fixes a possible misuse of the tracing infrastructure with `mt start-sql` command. In particular, previously it was possible for a tracing span created (and finished) in `runStartInternal` to be used by a couple of things in `makeTenantSQLServerArgs` (main one being `rpc.Context`) that hold on to the context after the corresponding span has been finished. This is disallowed (since we reuse spans after `Finish` has been called on them) and was recently discovered. This problem is now fixed by deriving a child span only in `mt start-sql` code path that is never finished (since those users can be active throughout the whole lifetime of the tenant server).

The bug is pretty minor (I think only serverless is affected), so there is no release note.

Fixes: #113170.

Release note: None

Co-authored-by: Rafi Shamim <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
@blathers-crl blathers-crl bot mentioned this issue Nov 1, 2023
Merged
blathers-crl bot pushed a commit that referenced this issue Nov 1, 2023
@rafiss
roachtest: mark new npgsql tests as flaky
0f9a5f0 
These tests started flaking due to #108414. We will mark them as flaky
and investigate later.

Release note: None
rafiss added a commit to rafiss/cockroach that referenced this issue Nov 1, 2023
@rafiss
roachtest: mark new npgsql tests as flaky
d6826be 
These tests started flaking due to cockroachdb#108414. We will mark them as flaky
and investigate later.

Release note: None
@rafiss rafiss mentioned this issue Nov 1, 2023
Merged
This was referenced Dec 12, 2023
Closed
Closed
@fqazi fqazi mentioned this issue Feb 20, 2024
Closed
@rimadeodhar rimadeodhar mentioned this issue Feb 20, 2024
Closed
cockroach-dev-inf pushed a commit that referenced this issue Feb 20, 2024
@rimadeodhar
release-23.1: roachtest: Fix npgsql test
8ff0506 
One more npgsql test is flaking on 23.1 due to
#108414.
Marking it as a flake to resolve the failure.

Epic: none
Release note: None
Release justication: Low risk test only change
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-sql-pgwire pgwire protocol issues. A-testing Testing tools and infrastructure C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rafiss