Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: activerecord failed #94042

Closed
cockroach-teamcity opened this issue Dec 21, 2022 · 2 comments · Fixed by #94112
Closed

roachtest: activerecord failed #94042

cockroach-teamcity opened this issue Dec 21, 2022 · 2 comments · Fixed by #94112
Assignees
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Milestone

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Dec 21, 2022

roachtest.activerecord failed with artifacts on master @ 10266a323f94c3cf397d5de590e512d987e63e22:

test artifacts and logs in: /artifacts/activerecord/run_1
(test_impl.go:297).Fatalf: 
Tests run on Cockroach v23.1.0-alpha.1-208-g10266a323f
Tests run against activerecord 7.0.3
7237 Total Tests Run
7225 tests passed
12 tests failed
26 tests skipped
115 tests ignored
0 tests passed unexpectedly
1 test failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- PASS: SerializedAttributeTest#test_unexpected_serialized_type (expected)
--- PASS: SerializedAttributeTestWithYamlSafeLoad#test_unexpected_serialized_type (expected)
--- FAIL: ActiveRecord::WhereTest#test_invert_where (unexpected)
For a full summary look at the activerecord artifacts
An updated blocklist (activeRecordBlocklist) is available in the artifacts' activerecord log

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

/cc @cockroachdb/sql-sessions

This test on roachdash | Improve this report!

Jira issue: CRDB-22660

@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Dec 21, 2022
@cockroach-teamcity cockroach-teamcity added this to the 23.1 milestone Dec 21, 2022
@andyyang890
Copy link
Collaborator

From the test log:

Error:
ActiveRecord::WhereTest#test_invert_where:
RuntimeError: Wrapped undumpable exception for: ActiveRecord::StatementInvalid: PG::InternalError: ERROR:  internal error: interface conversion: interface {} is *memo.AggregationsExpr, not *memo.FiltersExpr
DETAIL:  stack trace:
github.com/cockroachdb/cockroach/pkg/util/errorutil/catch.go:29: ShouldCatch()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:248: func1()
GOROOT/src/runtime/panic.go:884: gopanic()
GOROOT/src/runtime/iface.go:262: panicdottypeE()
github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/opt/norm/factory.og.go:25300: DynamicConstruct()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/groupby_funcs.go:274: func1()
github.com/cockroachdb/cockroach/pkg/sql/opt/norm/factory.go:207: DisableOptimizationRulesTemporarily()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/groupby_funcs.go:289: GenerateStreamingGroupByLimitOrderingHint()
github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go:4126: exploreLimit()
github.com/cockroachdb/cockroach/bazel-out/k8-opt/bin/pkg/sql/opt/xform/explorer.og.go:62: exploreGroupMember()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/explorer.go:185: exploreGroup()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:536: optimizeGroup()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:704: optimizeEnforcer()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:655: enforceProps()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:563: optimizeGroupMember()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:523: optimizeGroup()
github.com/cockroachdb/cockroach/pkg/sql/opt/xform/optimizer.go:269: Optimize()
github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:501: reuseMemo()
github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:531: buildExecMemo()
github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:237: makeOptimizerPlan()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1518: makeExecPlan()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1125: dispatchToExecutionEngine()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:737: execStmtInOpenState()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:130: func1()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2514: execWithProfiling()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:129: execStmt()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:219: execPortal()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2034: func2()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2036: execCmd()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1878: run()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:834: ServeConn()
github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728: func1()

HINT:  You have encountered an unexpected error.

Please check the public issue tracker to check whether this problem is
already tracked. If you cannot find it there, please report the error
with details by creating a new issue.

If you would rather not post publicly, please contact us directly
using the support form.

We appreciate your feedback.


    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:783:in `exec_prepared'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:783:in `block (2 levels) in exec_cache'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activesupport/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activesupport/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:782:in `block in exec_cache'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activesupport/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activesupport/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activesupport/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activesupport/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:765:in `block in log'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activesupport/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:756:in `log'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:781:in `exec_cache'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:747:in `execute_and_clear'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/connection_adapters/postgresql/database_statements.rb:54:in `exec_query'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:560:in `select'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in `select_all'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:110:in `select_all'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/querying.rb:54:in `_query_by_sql'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation.rb:942:in `block in exec_main_query'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation.rb:962:in `skip_query_cache_if_necessary'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation.rb:928:in `exec_main_query'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation.rb:914:in `block in exec_queries'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation.rb:962:in `skip_query_cache_if_necessary'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation.rb:908:in `exec_queries'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation/record_fetch_warning.rb:19:in `exec_queries'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/association_relation.rb:44:in `exec_queries'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation.rb:695:in `load'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation.rb:250:in `records'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation.rb:245:in `to_ary'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation/finder_methods.rb:549:in `find_nth_with_limit'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation/finder_methods.rb:534:in `find_nth'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/lib/active_record/relation/finder_methods.rb:146:in `first'
    /usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/test/cases/relation/where_test.rb:438:in `test_invert_where'

rails test usr/local/lib/ruby/gems/2.7.0/bundler/gems/rails-5bbd370091d8/activerecord/test/cases/relation/where_test.rb:434

From looking at the PRs that were merged in the last 24 hours, I think this one might be relevant: #93858.

@msirek msirek self-assigned this Dec 21, 2022
@msirek
Copy link
Contributor

msirek commented Dec 21, 2022

The generated code for the GenerateStreamingGroupByLimitOrderingHint rule looks like this:

	// [GenerateStreamingGroupByLimitOrderingHint]
	{
		_partlyExplored := _rootOrd < _rootState.start
		aggregation := _root.Input
		_state := _e.lookupExploreState(aggregation)
		if !_state.fullyExplored {
			_fullyExplored = false
		}
		var _member memo.RelExpr
		for _ord := 0; _ord < _state.end; _ord++ {
			if _member == nil {
				_member = aggregation.FirstExpr()
			} else {
				_member = _member.NextExpr()
			}
			if !_partlyExplored || _ord >= _state.start {
				if _member.Op() == opt.GroupByOp || _member.Op() == opt.DistinctOnOp {

...

_e.funcs.GenerateStreamingGroupByLimitOrderingHint(_root, _root, aggregation, input, aggs, private, ordering)

So, the first member in the group, aggregation, may be a SelectExpr, but we still end up exploring the group for all other group members and check for group-by or distinct-on, yet aggregation still points to the SelectExpr even though according to the match rule it looks like aggregation should be a GroupByExpr or a DistinctOnExpr:

[GenerateStreamingGroupByLimitOrderingHint, Explore]
(Limit
    $aggregation:(GroupBy | DistinctOn
        $input:*
        $aggs:*
        $private:* & (IsCanonicalGroupBy $private)
    )
    (Const $limit:* & (IsPositiveInt $limit))
    $ordering:* &
        (Let
            (
                $groupingCols
                $newOrdering
                $ok
            ):(GroupingColsClosureOverlappingOrdering
                $input
                $private
                $ordering
            )
            $ok
        )
)

msirek pushed a commit to msirek/cockroach that referenced this issue Dec 21, 2022
Fixes cockroachdb#94042

This fixes a slightly confusing feature of optgen where an alias in
a match rule is assigned to the first expression in the group, and
may be of a different expression type than the allowed Ops in the
match rule.

Example:
```
[GenerateStreamingGroupByLimitOrderingHint, Explore]
(Limit
    $aggregation:(GroupBy | DistinctOn
```
Given this match rule, it looks like `$aggregation` can only be a
GroupBy or DistinctOn expression. But in cockroachdb#94042, it is sometimes a
Select expression. The variable is later used to generate a new
expression with the assumption it is one of the allowed Op types,
causing a panic.

This is fixed by updating the optgen rule gen code to keep the alias
up-to-date with the current memo group member being examined in the
match rule, as it cycles through all members in the group.

Release note: None
craig bot pushed a commit that referenced this issue Dec 22, 2022
94112: main: update optgen Op alias to match current memo group member r=DrewKimball a=msirek

Fixes #94042

This fixes a slightly confusing feature of optgen where an alias in a match rule is assigned to the first expression in the group, and may be of a different expression type than the allowed Ops in the match rule.

Example:
```
[GenerateStreamingGroupByLimitOrderingHint, Explore]
(Limit
    $aggregation:(GroupBy | DistinctOn
```
Given this match rule snippet, it looks like `$aggregation` can only be a GroupBy or DistinctOn expression. But in #94042, it is sometimes a Select expression. The variable is later used to generate a new expression with the assumption it is one of the allowed Op types, causing a panic.

This is fixed by updating the optgen rule gen code to keep the alias up-to-date with the current memo group member being examined in the match rule, as it cycles through all members in the group.

Release note: None

Co-authored-by: Mark Sirek <[email protected]>
@craig craig bot closed this as completed in 198c5e2 Dec 22, 2022
msirek pushed a commit to msirek/cockroach that referenced this issue Dec 29, 2022
Fixes cockroachdb#94042

This fixes a slightly confusing feature of optgen where an alias in
a match rule is assigned to the first expression in the group, and
may be of a different expression type than the allowed Ops in the
match rule.

Example:
```
[GenerateStreamingGroupByLimitOrderingHint, Explore]
(Limit
    $aggregation:(GroupBy | DistinctOn
```
Given this match rule, it looks like `$aggregation` can only be a
GroupBy or DistinctOn expression. But in cockroachdb#94042, it is sometimes a
Select expression. The variable is later used to generate a new
expression with the assumption it is one of the allowed Op types,
causing a panic.

This is fixed by updating the optgen rule gen code to keep the alias
up-to-date with the current memo group member being examined in the
match rule, as it cycles through all members in the group.

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants