Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql/opt/exec/execbuilder: TestExecBuild failed #50863

Closed
cockroach-teamcity opened this issue Jul 1, 2020 · 3 comments · Fixed by #51780
Closed

sql/opt/exec/execbuilder: TestExecBuild failed #50863

cockroach-teamcity opened this issue Jul 1, 2020 · 3 comments · Fixed by #51780
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Milestone

Comments

@cockroach-teamcity
Copy link
Member

(sql/opt/exec/execbuilder).TestExecBuild failed on master@84c0a3e960421ec754583daa708c8c3542b4f1e3:

=== RUN   TestExecBuild/local/srfs/correlated_SRFs
rewrote:
CREATE TABLE xy (x INT8 PRIMARY KEY, y INT8, FAMILY (x, y));


rewrote:
CREATE TABLE xz (x INT8 PRIMARY KEY, z INT8, FAMILY (z, x));


rewrote:
CREATE TABLE groups (id SERIAL8, data JSONB, PRIMARY KEY (id), FAMILY (id, data));


rewrote:
CREATE TABLE articles (id INT8 PRIMARY KEY, body STRING, description STRING, title STRING, slug STRING, tag_list STRING[], user_id STRING, created_at TIMESTAMP, updated_at TIMESTAMP, FAMILY (id, created_at, slug), FAMILY (user_id, title), FAMILY (body, tag_list), FAMILY (description, updated_at));


rewrote:
CREATE TABLE a (x INT8 PRIMARY KEY, y INT8, FAMILY (y, x));
CREATE TABLE b (x INT8 PRIMARY KEY, z INT8, FAMILY (z), FAMILY (x));


--- done: testdata/srfs with config local: 22 tests, 0 failures
--- total progress: 240 statements/queries

More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestExecBuild PKG=./pkg/sql/opt/exec/execbuilder TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. labels Jul 1, 2020
@cockroach-teamcity cockroach-teamcity added this to the 20.2 milestone Jul 1, 2020
@RaduBerinde
Copy link
Member

I can't figure out what the actual failure was here. Logs say

TestExecBuild: test_log_scope.go:77: test logs captured to: /tmp/logTestExecBuild975095505

but there is nothing like that in the artifacts for the build.

I tried to reproduce, no luck so far (though I hit #51027, I'll try again when that is fixed).

@cockroach-teamcity
Copy link
Member Author

(sql/opt/exec/execbuilder).TestExecBuild failed on master@e9a4f83e3eee59510f97db2c6e0df9b57cf6b944:

=== RUN   TestExecBuild
    TestExecBuild: test_log_scope.go:77: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestExecBuild990370903
    TestExecBuild: test_log_scope.go:58: use -show-logs to present logs inline
--- FAIL: TestExecBuild (44.84s)
=== RUN   TestExecBuild/5node/stats
=== PAUSE TestExecBuild/5node/stats
=== CONT  TestExecBuild/5node/stats
--- progress: testdata/distsql_interleaved_join: 40 statements/queries
    TestExecBuild/5node/stats: logic.go:2062: 
         
        testdata/stats:22: EXPLAIN (VERBOSE) SELECT * FROM uv WHERE u = 1 AND v = 1
        expected:
            ·          distribution  full         ·       ·
            ·          vectorized    true         ·       ·
            filter     ·             ·            (u, v)  ·
             │         filter        u = 1        ·       ·
             └── scan  ·             ·            (u, v)  ·
            ·          table         uv@uv_v_idx  ·       ·
            ·          spans         /1-/2        ·       ·
            
        but found (query options: "") :
            ·          distribution  full         ·       ·
            ·          vectorized    true         ·       ·
            filter     ·             ·            (u, v)  ·
             │         filter        v = 1        ·       ·
             └── scan  ·             ·            (u, v)  ·
            ·          table         uv@uv_u_idx  ·       ·
            ·          spans         /1-/2        ·       ·
            
--- done: testdata/stats with config 5node: 5 tests, 1 failures
    TestExecBuild/5node/stats: logic.go:2643: 
        testdata/stats:33: error while processing
    TestExecBuild/5node/stats: logic.go:2643: testdata/stats:33: too many errors encountered, skipping the rest of the input
        --- FAIL: TestExecBuild/5node/stats (1.18s)
=== RUN   TestExecBuild/5node
    --- FAIL: TestExecBuild/5node (0.00s)

More

Parameters:

  • TAGS=
  • GOFLAGS=-parallel=4
make stressrace TESTS=TestExecBuild PKG=./pkg/sql/opt/exec/execbuilder TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

@cockroach-teamcity
Copy link
Member Author

(sql/opt/exec/execbuilder).TestExecBuild failed on master@dafb69aa4af50d965698738323680c9175d59511:

=== RUN   TestExecBuild
    TestExecBuild: logic.go:2762: randomize batchSize to 3
    TestExecBuild: test_log_scope.go:77: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestExecBuild063628629
    TestExecBuild: test_log_scope.go:58: use -show-logs to present logs inline
--- FAIL: TestExecBuild (63.01s)
=== RUN   TestExecBuild/5node/stats
=== PAUSE TestExecBuild/5node/stats
=== CONT  TestExecBuild/5node/stats
        --- FAIL: TestExecBuild/5node/stats (2.46s)
=== RUN   TestExecBuild/5node
    --- FAIL: TestExecBuild/5node (0.00s)

More

Parameters:

  • GOFLAGS=-json
make stressrace TESTS=TestExecBuild PKG=./pkg/sql/opt/exec/execbuilder TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
powered by pkg/cmd/internal/issues

rytaft added a commit to rytaft/cockroach that referenced this issue Jul 22, 2020
This commit fixes a race condition that was introduced in cockroachdb#51616.
The race condition existed because it was possible for two refreshes
to be triggered in close succession, and there was no mechanism to
prevent the first refresh from overwriting the second if it was
delayed.

We fix the race condition by introducing two boolean flags into each
cache entry: refreshing and mustRefreshAgain. If refreshing is true,
that indicates that the current statistics for this table are stale,
and we are in the process of fetching the updated stats from the database.
In the mean time, other callers can use the stale stats and do not need to
wait.

If a goroutine tries to perform a refresh when a refresh is already
in progress, it will see that refreshing=true and will set the
mustRefreshAgain flag to true before returning. When the original
goroutine that was performing the refresh returns from the database and
sees that mustRefreshAgain=true, it will trigger another refresh.

Informs cockroachdb#50863

Release note: None
craig bot pushed a commit that referenced this issue Jul 23, 2020
51691: kvserver: remove ToIsLearner flag from RaftHeartbeat r=yuzefovich a=nvanbenschoten

This commit removes the ToIsLearner flag from the RaftHeartbeat message. This is allowed because 20.2 no longer need to deal with preemptive snapshots. This completes @ajwerner's heroic migration started in 99a031d and followed up by b2850e1.

51692: tree: cast TIMESTAMP to TEXT without timezone data r=rohany,adityamaru a=otan

This better matches postgres. I'm guessing it used to be done this way
to match the CLI output (which we can't fix without changing lib/pq),
but this affects output results from ORM tests. Note that CLI output is
not affected unless TIMESTAMP is cast to text.

Release note (sql change): Casting TIMESTAMP types to TEXT related types
now omits the timezone component. For example, '2001-12-15
15:14:13'::TIMESTAMP will now format as '2001-12-15 15:14:13' instead of
'2001-12-15 15:14:13+00:00'.



51770: cmd/skipped-tests: utility to post skipped tests to slack r=yuzefovich a=petermattis

Release note: None

51773: sql: panic with assertion errors, not raw strings r=jordanlewis a=jordanlewis

Panicking with raw strings will cause the strings to be redacted.

Release note: None

51780: opt: fix flake in TestExecBuild due to new stats cache refresh mechanism r=rytaft a=rytaft

This commit fixes a flake caused by the new stats cache update mechanism,
which refreshes the stats asynchronously. The solution is to retry the
test query until the new stats are available in the cache.

Fixes #50863

Release note: None

51784: sql: disable distribution of plans when transaction has modified a type r=rohany a=rohany

Fixes #50897.

Release note (sql change): Transactions that have modified or created a
type will execute queries on the local node, rather than distributing
the queries to other nodes in the cluster.

51806: roachtest: deflake disk-stalled roachtests r=irfansharif a=irfansharif

Fixes #51724, which failed with the salient error:

  ERROR: no --join flags provided to 'cockroach start'

Since this test is spinning up a single node cluster to test disk
stalls, it suffices to use start-single-node instead.

Release note: None

Co-authored-by: Nathan VanBenschoten <[email protected]>
Co-authored-by: Oliver Tan <[email protected]>
Co-authored-by: Peter Mattis <[email protected]>
Co-authored-by: Jordan Lewis <[email protected]>
Co-authored-by: Rebecca Taft <[email protected]>
Co-authored-by: Rohan Yadav <[email protected]>
Co-authored-by: irfan sharif <[email protected]>
@craig craig bot closed this as completed in 399f590 Jul 23, 2020
rytaft added a commit to rytaft/cockroach that referenced this issue Jul 23, 2020
This commit fixes a race condition that was introduced in cockroachdb#51616.
The race condition existed because it was possible for two refreshes
to be triggered in close succession, and there was no mechanism to
prevent the first refresh from overwriting the second if it was
delayed.

We fix the race condition by introducing two boolean flags into each
cache entry: refreshing and mustRefreshAgain. If refreshing is true,
that indicates that the current statistics for this table are stale,
and we are in the process of fetching the updated stats from the database.
In the mean time, other callers can use the stale stats and do not need to
wait.

If a goroutine tries to perform a refresh when a refresh is already
in progress, it will see that refreshing=true and will set the
mustRefreshAgain flag to true before returning. When the original
goroutine that was performing the refresh returns from the database and
sees that mustRefreshAgain=true, it will trigger another refresh.

Informs cockroachdb#50863

Release note: None
rytaft added a commit to rytaft/cockroach that referenced this issue Jul 23, 2020
This commit fixes a race condition that was introduced in cockroachdb#51616.
The race condition existed because it was possible for two refreshes
to be triggered in close succession, and there was no mechanism to
prevent the first refresh from overwriting the second if it was
delayed.

We fix the race condition by introducing two boolean flags into each
cache entry: refreshing and mustRefreshAgain. If refreshing is true,
that indicates that the current statistics for this table are stale,
and we are in the process of fetching the updated stats from the database.
In the mean time, other callers can use the stale stats and do not need to
wait.

If a goroutine tries to perform a refresh when a refresh is already
in progress, it will see that refreshing=true and will set the
mustRefreshAgain flag to true before returning. When the original
goroutine that was performing the refresh returns from the database and
sees that mustRefreshAgain=true, it will trigger another refresh.

Informs cockroachdb#50863

Release note: None
craig bot pushed a commit that referenced this issue Jul 24, 2020
51805: stats: fix race condition when refreshing cache r=rytaft a=rytaft

This commit fixes a race condition that was introduced in #51616.
The race condition existed because it was possible for two refreshes
to be triggered in close succession, and there was no mechanism to
prevent the first refresh from overwriting the second if it was
delayed.

We fix the race condition by introducing two boolean flags into each
cache entry: `refreshing` and `mustRefreshAgain`. If `refreshing` is true,
that indicates that the current statistics for this table are stale,
and we are in the process of fetching the updated stats from the database.
In the mean time, other callers can use the stale stats and do not need to
wait.

If a goroutine tries to perform a refresh when a refresh is already
in progress, it will see that `refreshing=true` and will set the
`mustRefreshAgain` flag to `true` before returning. When the original
goroutine that was performing the refresh returns from the database and
sees that `mustRefreshAgain=true`, it will trigger another refresh.

Informs #50863

Release note: None

Co-authored-by: Rebecca Taft <[email protected]>
rytaft added a commit to rytaft/cockroach that referenced this issue Jul 31, 2020
This commit fixes a flake caused by the new stats cache update mechanism,
which refreshes the stats asynchronously. The solution is to retry the
test query until the new stats are available in the cache.

Fixes cockroachdb#50863

Release note: None
rytaft added a commit to rytaft/cockroach that referenced this issue Jul 31, 2020
This commit fixes a race condition that was introduced in cockroachdb#51616.
The race condition existed because it was possible for two refreshes
to be triggered in close succession, and there was no mechanism to
prevent the first refresh from overwriting the second if it was
delayed.

We fix the race condition by introducing two boolean flags into each
cache entry: refreshing and mustRefreshAgain. If refreshing is true,
that indicates that the current statistics for this table are stale,
and we are in the process of fetching the updated stats from the database.
In the mean time, other callers can use the stale stats and do not need to
wait.

If a goroutine tries to perform a refresh when a refresh is already
in progress, it will see that refreshing=true and will set the
mustRefreshAgain flag to true before returning. When the original
goroutine that was performing the refresh returns from the database and
sees that mustRefreshAgain=true, it will trigger another refresh.

Informs cockroachdb#50863

Release note: None
rytaft added a commit to rytaft/cockroach that referenced this issue Jul 31, 2020
This commit fixes a flake caused by the new stats cache update mechanism,
which refreshes the stats asynchronously. The solution is to retry the
test query until the new stats are available in the cache.

Fixes cockroachdb#50863

Release note: None
rytaft added a commit to rytaft/cockroach that referenced this issue Jul 31, 2020
This commit fixes a race condition that was introduced in cockroachdb#51616.
The race condition existed because it was possible for two refreshes
to be triggered in close succession, and there was no mechanism to
prevent the first refresh from overwriting the second if it was
delayed.

We fix the race condition by introducing two boolean flags into each
cache entry: refreshing and mustRefreshAgain. If refreshing is true,
that indicates that the current statistics for this table are stale,
and we are in the process of fetching the updated stats from the database.
In the mean time, other callers can use the stale stats and do not need to
wait.

If a goroutine tries to perform a refresh when a refresh is already
in progress, it will see that refreshing=true and will set the
mustRefreshAgain flag to true before returning. When the original
goroutine that was performing the refresh returns from the database and
sees that mustRefreshAgain=true, it will trigger another refresh.

Informs cockroachdb#50863

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants