Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: asyncpg failed [asyncpg has a race condition causing test cleanup to fail] #119291

Closed
cockroach-teamcity opened this issue Feb 16, 2024 · 7 comments · Fixed by #120553
Closed

roachtest: asyncpg failed [asyncpg has a race condition causing test cleanup to fail] #119291

cockroach-teamcity opened this issue Feb 16, 2024 · 7 comments · Fixed by #120553
Assignees
@rafiss
Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Milestone
24.1

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Feb 16, 2024 •
edited by jlinder
Loading

roachtest.asyncpg failed with artifacts on master @ 7d0697b632066ee78735fc57e8150222d5576d0d:

(orm_helpers.go:221).summarizeFailed: 
Tests run on Cockroach v24.1.0-alpha.00000000-dev-7d0697b632066ee78735fc57e8150222d5576d0d
Tests run against asyncpg v0.24.0
226 Total Tests Run
157 tests passed
69 tests failed
34 tests skipped
1 test ignored
0 tests passed unexpectedly
9 tests failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_1 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_no_binary_codec (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_2 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_bytes_like (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_large_rows (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_timeout (unexpected)
--- FAIL: test_copy.TestCopyFrom.test_copy_from_table_large_rows (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_1 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_file_path (unexpected)
For a full summary look at the asyncpg artifacts 
An updated blocklist (asyncpgBlocklist) is available in the artifacts' asyncpg log
test artifacts and logs in: /artifacts/asyncpg/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-36131
@cockroach-teamcity cockroach-teamcity added branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions) labels Feb 16, 2024
@cockroach-teamcity cockroach-teamcity added this to the 24.1 milestone Feb 16, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.asyncpg failed with artifacts on master @ 0b7ae19e2b94b851ed8812914f57032aab699811:

(orm_helpers.go:221).summarizeFailed: 
Tests run on Cockroach v24.1.0-alpha.00000000-dev-0b7ae19e2b94b851ed8812914f57032aab699811
Tests run against asyncpg v0.24.0
227 Total Tests Run
156 tests passed
71 tests failed
34 tests skipped
1 test ignored
0 tests passed unexpectedly
11 tests failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- FAIL: test_listeners.TestListeners.test_dangling_listener_warns (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_large_rows (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_no_binary_codec (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_1 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_2 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_file_path (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_timeout (unexpected)
--- FAIL: test_copy.TestCopyFrom.test_copy_from_table_large_rows (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_bytes_like (unexpected)
--- FAIL: test_copy.TestCopyFrom.test_copy_from_query_basics (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_1 (unexpected)
For a full summary look at the asyncpg artifacts 
An updated blocklist (asyncpgBlocklist) is available in the artifacts' asyncpg log
test artifacts and logs in: /artifacts/asyncpg/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@rafiss
Copy link
Collaborator

rafiss commented Feb 17, 2024

The root issue here is that test_copy_from_table_large_rows started failing.

======================================================================
ERROR: test_copy_from_table_large_rows (test_copy.TestCopyFrom)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/data1/asyncpg/tests/test_copy.py", line 100, in test_copy_from_table_large_rows
    res = await self.con.copy_from_table('copytab', output=f)
  File "/mnt/data1/asyncpg/asyncpg/connection.py", line 740, in copy_from_table
    return await self._copy_out(copy_stmt, output, timeout)
  File "/mnt/data1/asyncpg/asyncpg/connection.py", line 1030, in _copy_out
    return await self._protocol.copy_out(copy_stmt, writer, timeout)
  File "asyncpg/protocol/protocol.pyx", line 361, in copy_out
    self.resume_reading()
  File "asyncpg/protocol/protocol.pyx", line 134, in asyncpg.protocol.protocol.BaseProtocol.resume_reading
    self.transport.resume_reading()
  File "/usr/lib/python3.10/asyncio/sslproto.py", line 343, in resume_reading
    self._ssl_protocol._transport.resume_reading()
AttributeError: 'NoneType' object has no attribute 'resume_reading'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/data1/asyncpg/asyncpg/_testbase/__init__.py", line 92, in wrapper
    self.loop.run_until_complete(coro)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/mnt/data1/asyncpg/tests/test_copy.py", line 117, in test_copy_from_table_large_rows
    await self.con.execute('DROP TABLE public.copytab')
  File "/mnt/data1/asyncpg/asyncpg/connection.py", line 312, in execute
    self._check_open()
  File "/mnt/data1/asyncpg/asyncpg/connection.py", line 1380, in _check_open
    raise exceptions.InterfaceError('connection is closed')
asyncpg.exceptions._base.InterfaceError: connection is closed

Since the cleanup statement DROP TABLE public.copytab failed, that caused a lot of the other tests in the file to fail, since they start by running CREATE TABLE copytab .... For example, see test_copy_records_to_table_no_binary_codec here: https://github.com/MagicStack/asyncpg/blob/74f3a0031532b314d9141b6de12aa74db7726b3d/tests/test_copy.py#L701-L704

The main problem to fix is the 'NoneType' object has no attribute 'resume_reading' error. That should allow the DROP to work. It looks like there is a known issue upstream that is related: MagicStack/asyncpg#240

@rafiss rafiss changed the title roachtest: asyncpg failed roachtest: asyncpg failed [asyncpg has a race condition causing test cleanup to fail] Feb 17, 2024
@rafiss rafiss added P-2 Issues/test failures with a fix SLA of 3 months and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Feb 17, 2024
@cockroach-teamcity
Copy link
Member Author

roachtest.asyncpg failed with artifacts on master @ e39dafe6d8c153301ff43ed2b3ed3e13af9ec72a:

(orm_helpers.go:221).summarizeFailed: 
Tests run on Cockroach v24.1.0-alpha.00000000-dev-e39dafe6d8c153301ff43ed2b3ed3e13af9ec72a
Tests run against asyncpg v0.24.0
226 Total Tests Run
157 tests passed
69 tests failed
34 tests skipped
1 test ignored
0 tests passed unexpectedly
9 tests failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_no_binary_codec (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_file_path (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_2 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_large_rows (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_1 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_1 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_bytes_like (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_timeout (unexpected)
--- FAIL: test_copy.TestCopyFrom.test_copy_from_table_large_rows (unexpected)
For a full summary look at the asyncpg artifacts 
An updated blocklist (asyncpgBlocklist) is available in the artifacts' asyncpg log
test artifacts and logs in: /artifacts/asyncpg/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.asyncpg failed with artifacts on master @ 24414b864f65c6544ea987f7970fa1c192ca3310:

(orm_helpers.go:221).summarizeFailed: 
Tests run on Cockroach v24.1.0-alpha.00000000-dev-24414b864f65c6544ea987f7970fa1c192ca3310
Tests run against asyncpg v0.24.0
226 Total Tests Run
157 tests passed
69 tests failed
34 tests skipped
1 test ignored
0 tests passed unexpectedly
9 tests failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- FAIL: test_copy.TestCopyFrom.test_copy_from_table_large_rows (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_timeout (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_2 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_bytes_like (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_1 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_no_binary_codec (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_file_path (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_large_rows (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_1 (unexpected)
For a full summary look at the asyncpg artifacts 
An updated blocklist (asyncpgBlocklist) is available in the artifacts' asyncpg log
test artifacts and logs in: /artifacts/asyncpg/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.asyncpg failed with artifacts on master @ 791969fa98711e681289fa4f61f3d58c8fe44263:

(orm_helpers.go:221).summarizeFailed: 
Tests run on Cockroach v24.1.0-alpha.2-dev-791969fa98711e681289fa4f61f3d58c8fe44263
Tests run against asyncpg v0.24.0
226 Total Tests Run
157 tests passed
69 tests failed
34 tests skipped
1 test ignored
0 tests passed unexpectedly
9 tests failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_2 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_large_rows (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_no_binary_codec (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_file_path (unexpected)
--- FAIL: test_copy.TestCopyFrom.test_copy_from_table_large_rows (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_timeout (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_1 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_1 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_bytes_like (unexpected)
For a full summary look at the asyncpg artifacts 
An updated blocklist (asyncpgBlocklist) is available in the artifacts' asyncpg log
test artifacts and logs in: /artifacts/asyncpg/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.asyncpg failed with artifacts on master @ 559994e230ac3776dd1188e66678cce9205ab345:

(orm_helpers.go:221).summarizeFailed: 
Tests run on Cockroach v24.1.0-alpha.2-dev-559994e230ac3776dd1188e66678cce9205ab345
Tests run against asyncpg v0.24.0
227 Total Tests Run
156 tests passed
71 tests failed
34 tests skipped
1 test ignored
0 tests passed unexpectedly
11 tests failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- FAIL: test_copy.TestCopyFrom.test_copy_from_query_basics (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_1 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_no_binary_codec (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_bytes_like (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_timeout (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_2 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_file_path (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_large_rows (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_1 (unexpected)
--- FAIL: test_listeners.TestListeners.test_dangling_listener_warns (unexpected)
--- FAIL: test_copy.TestCopyFrom.test_copy_from_table_large_rows (unexpected)
For a full summary look at the asyncpg artifacts 
An updated blocklist (asyncpgBlocklist) is available in the artifacts' asyncpg log
test artifacts and logs in: /artifacts/asyncpg/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.asyncpg failed with artifacts on master @ 2ac5657aef0fd878973191079b6f3aadbc9c9266:

(orm_helpers.go:221).summarizeFailed: 
Tests run on Cockroach v24.1.0-alpha.3-dev-2ac5657aef0fd878973191079b6f3aadbc9c9266
Tests run against asyncpg v0.24.0
226 Total Tests Run
157 tests passed
69 tests failed
34 tests skipped
1 test ignored
0 tests passed unexpectedly
9 tests failed unexpectedly
0 tests expected failed but skipped
0 tests expected failed but not run
---
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_1 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_bytes_like (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_1 (unexpected)
--- FAIL: test_copy.TestCopyFrom.test_copy_from_table_large_rows (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_timeout (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_fail_in_source_2 (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_from_file_path (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_records_to_table_no_binary_codec (unexpected)
--- FAIL: test_copy.TestCopyTo.test_copy_to_table_large_rows (unexpected)
For a full summary look at the asyncpg artifacts 
An updated blocklist (asyncpgBlocklist) is available in the artifacts' asyncpg log
test artifacts and logs in: /artifacts/asyncpg/run_1

Parameters:

  • ROACHTEST_arch=amd64
  • ROACHTEST_cloud=gce
  • ROACHTEST_coverageBuild=false
  • ROACHTEST_cpu=16
  • ROACHTEST_encrypted=false
  • ROACHTEST_fs=ext4
  • ROACHTEST_localSSD=true
  • ROACHTEST_metamorphicBuild=false
  • ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

@cockroach-teamcity cockroach-teamcity mentioned this issue Mar 14, 2024
Closed
@rafiss rafiss self-assigned this Mar 15, 2024
@rafiss rafiss mentioned this issue Mar 15, 2024
Merged
@craig craig bot closed this as completed in 8fcdbef Mar 15, 2024
This was referenced Mar 15, 2024
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rafiss rafiss

Labels
branch-master Failures and bugs on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. P-2 Issues/test failures with a fix SLA of 3 months T-sql-foundations SQL Foundations Team (formerly SQL Schema + SQL Sessions)
Projects
No open projects
SQL Foundations
Status: Done
Milestone
24.1
Development

Successfully merging a pull request may close this issue.

roachtest: add flaky django/asyncpg/npgsql tests to ignore list rafiss/cockroach
2 participants
@rafiss @cockroach-teamcity