Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

General maintenance and tidying to ensure tests pass. #136

Merged
merged 24 commits into from
Mar 27, 2022

Conversation

tillahoffmann
Copy link
Collaborator

@tillahoffmann tillahoffmann commented Apr 16, 2021

This large-ish PR aims to fix a number of challenges that make some of the tests flaky. At a high level:

  • Move from pyodbc to pymssql because the former hasn't proved particularly reliable and also doesn't work on arm platforms.
  • Updated requirements that do not require code changes. Cf. google cloud pubsub container do not expects keyword argument "channel" for publisher and subscriber client #161.
  • Change wait_container_is_ready to accept a list of "transient errors" for which another try should be attempted. Other errors (such as ImportError) reraise the exception immediately. This should avoid some confusion such as PostgresContainer timeout when sqlalchemy not installed #137.
  • Add a function maybe_emulate_amd64 which sets --platform=linux/amd64 if the platform is arm64. This is required for cross-architecture testing because some images are only available as amd64.
  • Refactor of some parts of the docker client to provide easier error messages, such as "port mapping isn't available" as supposed to "NoneType is not indexable".
  • Changed the script of the Kafka container from sh to bash because sh doesn't support some of the syntax used (namely brackets). The Kafka tests are still a bit unstable, and I've marked them as xfail.
  • Some refactor in for MySql and SqlServer containers which used a mix of class variables and instance variables which required some hacks (e.g. using reload in testing).
  • Added some branching to allow for the tests to run on both arm64 and amd64.

@KerstenBreuer, this might be of interest if you want to get involved more with the code base.
@SergeyPirogov, would be great to get your eyes on this.

@codecov-commenter
Copy link

codecov-commenter commented Apr 17, 2021

Codecov Report

Merging #136 (2431c9e) into master (48ea7b8) will increase coverage by 4.65%.
The diff coverage is 91.22%.

❗ Current head 2431c9e differs from pull request most recent head 5e2723b. Consider uploading reports for the commit 5e2723b to get more accurate results

@@            Coverage Diff             @@
##           master     #136      +/-   ##
==========================================
+ Coverage   80.61%   85.26%   +4.65%     
==========================================
  Files          20       24       +4     
  Lines         459      631     +172     
  Branches       34       59      +25     
==========================================
+ Hits          370      538     +168     
- Misses         70       75       +5     
+ Partials       19       18       -1     
Impacted Files Coverage Δ
testcontainers/general.py 0.00% <0.00%> (ø)
testcontainers/core/docker_client.py 51.16% <27.27%> (-5.99%) ⬇️
testcontainers/core/generic.py 87.09% <75.00%> (-12.91%) ⬇️
testcontainers/core/container.py 78.88% <77.77%> (-2.29%) ⬇️
testcontainers/core/waiting_utils.py 94.73% <86.66%> (+3.30%) ⬆️
testcontainers/rabbitmq.py 93.10% <93.10%> (ø)
testcontainers/neo4j.py 93.75% <93.75%> (ø)
testcontainers/localstack.py 94.44% <94.44%> (ø)
testcontainers/compose.py 100.00% <100.00%> (ø)
testcontainers/core/utils.py 57.14% <100.00%> (+15.20%) ⬆️
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6641b25...5e2723b. Read the comment docs.

@tillahoffmann tillahoffmann force-pushed the m1 branch 3 times, most recently from b2e9277 to 4e10204 Compare December 15, 2021 00:51
@tillahoffmann tillahoffmann changed the title Add M1 support and update dependencies. General maintenance and tidying to ensure tests pass. Dec 15, 2021
@tillahoffmann
Copy link
Collaborator Author

@yakimka, planning to merge this one soon as it's been lingering for almost a year. It's a bit of a beast (see description) above.

@tillahoffmann tillahoffmann merged commit fc5f2df into testcontainers:master Mar 27, 2022
@yakimka
Copy link
Collaborator

yakimka commented Mar 27, 2022

@tillahoffmann any suggestions on my comments?

@tillahoffmann tillahoffmann deleted the m1 branch March 27, 2022 19:03
@tillahoffmann
Copy link
Collaborator Author

@yakimka, I can't see any comments. Are they possibly still showing as pending on your end?

@@ -35,7 +35,7 @@ def get_bootstrap_server(self):
port = self.get_exposed_port(self.port_to_expose)
return '{}:{}'.format(host, port)

@wait_container_is_ready()
@wait_container_is_ready(UnrecognizedBrokerVersion, NoBrokersAvailable, KafkaError, ValueError)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UnrecognizedBrokerVersion and NoBrokersAvailable is inherited from KafkaError

Therefore, I think there is no need to specify them explicitly

@@ -23,7 +23,7 @@ def __init__(self, image="redis:latest", port_to_expose=6379):
self.port_to_expose = port_to_expose
self.with_exposed_ports(self.port_to_expose)

@wait_container_is_ready()
@wait_container_is_ready(redis.exceptions.ConnectionError)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if not client.ping():
    raise Exception

Will this work after changing the wait_container_is_ready logic?

db.with_bind_ports(3306, 32785)
with db:
url = db.get_connection_url()
container = mysql.MySqlContainer("mariadb:10.6.5")\
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe?

container = mysql.MySqlContainer(
    "mariadb:10.6.5"
).with_bind_ports(3306, 32785).maybe_emulate_amd64()

@@ -1,6 +1,12 @@
ARG version=3.8
FROM python:${version}

WORKDIR /workspace
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WORKDIR /workspace twice?

@yakimka
Copy link
Collaborator

yakimka commented Mar 27, 2022

@tillahoffmann oh, sorry, my fault

@mihaylovich
Copy link

Hello.
I think logging is enough to quickly investigate the bug described by #137.
I mean this added code
logger.debug('container is not yet ready: %s', traceback.format_exc())

But add transient_exceptions tuple is confusing when use some other docker images.
e.g. MS SQL container on certain state of starting server returns the error:
pyodbc.InterfaceError: ('28000', "[28000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user 'SA'. (18456) (SQLDriverConnect); [28000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Login failed for user 'SA'. (18456)")
These changes have also led to the need to list all the required exception types in the library's source code (e.g kafka.py)
When you instantiate any DockerContainer you can specify the concrete image name, but you cannot specify which exceptions should be ignored.

@tillahoffmann
Copy link
Collaborator Author

Hey, we can definitely consider adding transient exceptions as an exposed argument. The motivation behind the change was to not "hide" errors when they occur. E.g. an ImportError would previously time out after two minutes even though it should've failed immediately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants