Add TLS connection parameters #9

Jorge-Rodriguez · 2020-07-23T07:59:46Z

SUMMARY

Add support for TLS REQUIRES options, to enforce TLS client connections for users.

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

mysql_user

ADDITIONAL INFORMATION


PR moved from https://github.com/ansible-collections/community.general/pull/369

Fixes #5

Jorge-Rodriguez · 2020-07-23T08:01:46Z

@Andersson007 The additional tests are still missing. I want to wait for the tests to pass with the existing code first.

It seems the test suite is broken by the mysql_variables module

Andersson007 · 2020-07-23T11:38:47Z

            },
            "sql_log_bin": true,
            "state": "present",
            "tls_requires": null,
            "update_password": "always",
            "user": "db_user1"
        }
    },
    "msg": "Password updated",
    "user": "db_user1"
}

TASK [test_mysql_user : assert] ************************************************
task path: /root/ansible_collections/community/mysql/tests/output/.tmp/integration/test_mysql_user-v9pnhtb_-ÅÑŚÌβŁÈ/tests/integration/targets/test_mysql_user/tasks/resource_limits.yml:72
fatal: [testhost]: FAILED! => {
    "assertion": "result is not changed",
    "changed": false,
    "evaluated_to": false,
    "msg": "Assertion failed"
}

Andersson007 · 2020-07-23T11:39:09Z

@Jorge-Rodriguez line 5990 of https://github.com/ansible-collections/community.mysql/pull/9/checks?check_run_id=901835985

Andersson007 · 2020-07-23T11:49:36Z

there are a lot of errors actually. Strange why they didn't appear when all mysql stuff was moved to this repo recently

Andersson007 · 2020-07-23T11:53:38Z

@bmildren can it be related to #7 ?

Andersson007 · 2020-07-23T12:22:36Z

@bmildren yes, starting from line 2531 of the log

TASK [test_mysql_replication : Start slave that is already running] ************
changed: [testhost] => {"changed": true, "msg": "Slave started ", "queries": ["START SLAVE"]}

TASK [test_mysql_replication : assert] *****************************************
fatal: [testhost]: FAILED! => {
    "assertion": "result is not changed",
    "changed": false,
    "evaluated_to": false,
    "msg": "Assertion failed"
}

The only changes that touched mysql_replication stuff were #7 , could you please take a look

bmildren · 2020-07-23T12:40:38Z

🤔 Hmm that's wierd, it looks like the CI started failing ~4 days ago, when the last commit was ~9 days ago. I can't think of something specific in the tests that might have changed elsewhere and could be affecting the result.. I'll have a dig and see what I can see.

Jorge-Rodriguez · 2020-07-23T12:50:42Z

@Jorge-Rodriguez line 5990 of https://github.com/ansible-collections/community.mysql/pull/9/checks?check_run_id=901835985

@Andersson007 that assertion error comes from a file that's not touched in this PR and it happens before my tests are run.

Andersson007 · 2020-07-23T12:55:55Z

@Jorge-Rodriguez line 5990 of https://github.com/ansible-collections/community.mysql/pull/9/checks?check_run_id=901835985

@Andersson007 that assertion error comes from a file that's not touched in this PR and it happens before my tests are run.

yes, looks unrelated

Andersson007 · 2020-07-23T13:13:16Z

I've just tried to run mysql_replication tests locally by ansible-test integration test_mysql_replication --docker ubuntu1804 --docker-no-pull -vvvv > ~/test.log and it failed with:

9277 TASK [setup_mysql | config | setup replication topology] ***********************
9278 task path: /root/ansible_collections/community/mysql/tests/output/.tmp/integration/test_mysql_replication-i3_k6h6h-ÅÑŚÌβŁÈ/tests/integration/targets/setup_mysql/tasks/c     onfig.yml:12
9279 <testhost> ESTABLISH LOCAL CONNECTION FOR USER: root
9280 <testhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
9281 <testhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1595509686.5712125-2274-202442579016454      `" && echo ansible-tmp-1595509686.5712125-2274-202442579016454="` echo /root/.ansible/tmp/ansible-tmp-1595509686.5712125-2274-202442579016454 `" ) && sleep 0'
9282 Using module file /root/ansible/lib/ansible/modules/command.py
9283 <testhost> PUT /root/.ansible/tmp/ansible-local-171t2274d5x/tmphqia4x3y TO /root/.ansible/tmp/ansible-tmp-1595509686.5712125-2274-202442579016454/AnsiballZ_command.py
9284 <testhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1595509686.5712125-2274-202442579016454/ /root/.ansible/tmp/ansible-tmp-1595509686.5712125-2274-202     442579016454/AnsiballZ_command.py && sleep 0'
9285 <testhost> EXEC /bin/sh -c '/tmp/python-q7hus96b-ansible/python /root/.ansible/tmp/ansible-tmp-1595509686.5712125-2274-202442579016454/AnsiballZ_command.py && sleep 0'
9286 <testhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1595509686.5712125-2274-202442579016454/ > /dev/null 2>&1 && sleep 0'
9287 fatal: [testhost]: FAILED! => {
9288     "changed": true,
9289     "cmd": "dbdeployer deploy multiple 8.0.21 --base-port 3306 --my-cnf-options=\"master_info_repository='TABLE'\" --my-cnf-options=\"relay_log_info_repository='TABLE'\     "",
9290     "delta": "0:00:17.193334",
9291     "end": "2020-07-23 13:08:24.265550",
9292     "invocation": {
9293         "module_args": {
9294             "_raw_params": "dbdeployer deploy multiple 8.0.21 --base-port 3306 --my-cnf-options=\"master_info_repository='TABLE'\" --my-cnf-options=\"relay_log_info_rep     ository='TABLE'\"",
9295             "_uses_shell": true,
9296             "argv": null,
9297             "chdir": null,
9298             "creates": "/root/sandboxes/multi_msb_8_0_21",
9299             "executable": null,
9300             "removes": null,
9301             "stdin": null,
9302             "stdin_add_newline": true,
9303             "strip_empty_ends": true,
9304             "warn": false
9305         }
9306     },
9307     "msg": "non-zero return code",
9308     "rc": 1,
9309     "start": "2020-07-23 13:08:07.072216",
9310     "stderr": "",
9311     "stderr_lines": [],
9312     "stdout": "Installing and starting node 1\n.. sandbox server started\nInstalling and starting node 2\n.. sandbox server started\nInstalling and starting node 3\ncmd     :    /root/sandboxes/multi_msb_8_0_21/node3/init_db\nerr:    exit status 1\nstdout: Error installing database in /root/sandboxes/multi_msb_8_0_21/node3\n\nstderr: 2020-     07-23T13:08:24.146203Z 0 [System] [MY-013169] [Server] /root/opt/mysql/8.0.21/bin/mysqld (mysqld 8.0.21) initializing of server in progress as process 3125\n2020-07-23T     13:08:24.157272Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.\n2020-07-23T13:08:24.172309Z 0 [ERROR] [MY-012681] [InnoDB] mmap(137363456 bytes) fa     iled; errno 12\n2020-07-23T13:08:24.172451Z 1 [ERROR] [MY-012956] [InnoDB] Cannot allocate memory for the buffer pool\n2020-07-23T13:08:24.172534Z 1 [ERROR] [MY-012929]      [InnoDB] InnoDB Database creation was aborted with error Generic error. You may need to delete the ibdata1 file before trying to start up again.\n2020-07-23T13:08:24.1     72715Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.\n2020-07-23T13:08:24.172789Z 0 [ERROR] [MY-013236] [Server] The designated data directory /     root/sandboxes/multi_msb_8_0_21/node3/data/ is unusable. You can remove all files that the server added to it.\n2020-07-23T13:08:24.172894Z 0 [ERROR] [MY-010119] [Serve     r] Aborting\n2020-07-23T13:08:24.173464Z 0 [System] [MY-010910] [Server] /root/opt/mysql/8.0.21/bin/mysqld: Shutdown complete (mysqld 8.0.21)  MySQL Community Server -      GPL.\n\ninit_db output: Error installing database in /root/sandboxes/multi_msb_8_0_21/node3\n\n# Pre-exit cleanup. \n#1 - Executing Rmdir( /root/sandboxes/multi_msb_8_0     _21)\nerror removing directory /root/sandboxes/multi_msb_8_0_21\nremove /root/sandboxes/multi_msb_8_0_21: directory not empty",
9313     "stdout_lines": [
9314         "Installing and starting node 1",
9315         ".. sandbox server started",
9316         "Installing and starting node 2",
9317         ".. sandbox server started",
9318         "Installing and starting node 3",
9319         "cmd:    /root/sandboxes/multi_msb_8_0_21/node3/init_db",
9320         "err:    exit status 1",
9321         "stdout: Error installing database in /root/sandboxes/multi_msb_8_0_21/node3",
9322         "",
9323         "stderr: 2020-07-23T13:08:24.146203Z 0 [System] [MY-013169] [Server] /root/opt/mysql/8.0.21/bin/mysqld (mysqld 8.0.21) initializing of server in progress as pro     cess 3125",
9324         "2020-07-23T13:08:24.157272Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.",
9325         "2020-07-23T13:08:24.172309Z 0 [ERROR] [MY-012681] [InnoDB] mmap(137363456 bytes) failed; errno 12",
9326         "2020-07-23T13:08:24.172451Z 1 [ERROR] [MY-012956] [InnoDB] Cannot allocate memory for the buffer pool",
9327         "2020-07-23T13:08:24.172534Z 1 [ERROR] [MY-012929] [InnoDB] InnoDB Database creation was aborted with error Generic error. You may need to delete the ibdata1 fi     le before trying to start up again.",
9328         "2020-07-23T13:08:24.172715Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.",
9329         "2020-07-23T13:08:24.172789Z 0 [ERROR] [MY-013236] [Server] The designated data directory /root/sandboxes/multi_msb_8_0_21/node3/data/ is unusable. You can remo     ve all files that the server added to it.",
9330         "2020-07-23T13:08:24.172894Z 0 [ERROR] [MY-010119] [Server] Aborting",
9331         "2020-07-23T13:08:24.173464Z 0 [System] [MY-010910] [Server] /root/opt/mysql/8.0.21/bin/mysqld: Shutdown complete (mysqld 8.0.21)  MySQL Community Server - GPL.     ",
9332         "",
9333         "init_db output: Error installing database in /root/sandboxes/multi_msb_8_0_21/node3",
9334         "",
9335         "# Pre-exit cleanup. ",
9336         "#1 - Executing Rmdir( /root/sandboxes/multi_msb_8_0_21)",
9337         "error removing directory /root/sandboxes/multi_msb_8_0_21",
9338         "remove /root/sandboxes/multi_msb_8_0_21: directory not empty"
9339     ]
9340 }

bmildren · 2020-07-23T13:16:24Z

The local error is just down to not having enough memory available for the 3 instances.

Andersson007 · 2020-07-23T13:30:38Z

added 4 GB to the test vm:)
Now there is the same error locally as here

10407 TASK [test_mysql_replication : Start slave that is already running] ************
10408 task path: /root/ansible_collections/community/mysql/tests/output/.tmp/integration/test_mysql_replication-ygs29pjk-ÅÑŚÌβŁÈ/tests/integration/targets/test_mysql_replica      tion/tasks/mysql_replication_initial.yml:168
...
10455     "msg": "Slave started ",
10456     "queries": [
10457         "START SLAVE"
10458     ]
10459 }
10460
10461 TASK [test_mysql_replication : assert] *****************************************
10462 task path: /root/ansible_collections/community/mysql/tests/output/.tmp/integration/test_mysql_replication-ygs29pjk-ÅÑŚÌβŁÈ/tests/integration/targets/test_mysql_replica      tion/tasks/mysql_replication_initial.yml:176
10463 fatal: [testhost]: FAILED! => {
10464     "assertion": "result is not changed",
10465     "changed": false,
10466     "evaluated_to": false,
10467     "msg": "Assertion failed"
10468 }

bmildren · 2020-07-23T13:32:41Z

Yep I've seen the same, I'm just trying to debug it now.. 👍

bmildren · 2020-07-23T14:31:18Z

Looks like these are legit errors, currently we test using pymysql only (when time allows I plan to also add testing for mysqlclient-python), and the errors began when we automatically started pulling pymysql v0.10.0 as opposed to 0.9.3.

v0.10.0 changes the behaviour of how warnings are handled (https://github.com/PyMySQL/PyMySQL/blob/master/CHANGELOG.md#v0100), and since MySQL 5.7 (we test on 5.7 & 8.0, but not 5.6), issuing START SLAVE when the replica is running results in a warning as opposed to an error. So with pymysql v0.10.0 the warning is no longer being reported.

For now we can pin pymysql to 0.9.3, which will make the tests go green again, however we should look to address this with a more appropriate fix.

Jorge-Rodriguez · 2020-07-23T15:03:12Z

@bmildren shall I add the pin on this PR or do I wait to rebase?

bmildren · 2020-07-23T15:31:30Z

Hey @Jorge-Rodriguez , sorry it caught me at a bad time today - it's out scope for this pr, so I'll push a quick fix, then you can rebase 🙂 👍

bmildren · 2020-07-23T15:57:03Z

#10 is merged, feel free to rebase 👍

codecov · 2020-07-24T16:41:53Z

Codecov Report

Merging #9 into main will increase coverage by 0.44%.
The diff coverage is 71.59%.

@@            Coverage Diff             @@
##             main       #9      +/-   ##
==========================================
+ Coverage   68.47%   68.91%   +0.44%     
==========================================
  Files           8        8              
  Lines        1459     1528      +69     
  Branches      386      404      +18     
==========================================
+ Hits          999     1053      +54     
- Misses        315      325      +10     
- Partials      145      150       +5

Impacted Files	Coverage Δ
plugins/modules/mysql_user.py	`67.69% <71.59%> (+1.63%)`	⬆️
plugins/modules/mysql_info.py	`76.56% <0.00%> (+0.52%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ecd70e8...c62b430. Read the comment docs.

Jorge-Rodriguez · 2020-07-24T17:24:36Z

Ok, it seems my code has some unexpected side effects on MySQL 8. I'll see to fix that. The issue with codecov puzzles me though as the file with reduced coverage is not touched in this PR.

Jorge-Rodriguez · 2020-07-25T16:44:16Z

@Andersson007 ok, the integration tests are green now and the new check mode tests are in place. I don't know what's the story with codecov though.

Andersson007 · 2020-07-26T06:52:53Z

@Jorge-Rodriguez thanks!
@bmalynovytch @bmildren could you look at the PR

Jorge-Rodriguez · 2020-07-26T08:26:57Z

@Andersson007 @bmildren It does seem that codecov does not like the Black formatting, which is a pity because we could use some consistency on the use of quotes. The line length fixes aren't too bad either if you don't mind the extra vertical space usage.

Jorge-Rodriguez · 2020-07-26T08:53:49Z

Timeout on line 4096 of the failed integration tests logs.

Jorge-Rodriguez · 2020-07-26T11:59:44Z

@Andersson007 does this look otherwise ok, or should I rebase/squash into a single commit?

Andersson007 · 2020-07-27T07:39:39Z

@Jorge-Rodriguez no, it's ok now, it'll be squashed during merge.
Still waiting for @bmalynovytch or/and @bmildren 's feedback

Andersson007 · 2020-07-27T07:41:11Z

@Jorge-Rodriguez i noticed some tests failed https://github.com/ansible-collections/community.mysql/pull/9/checks?check_run_id=911307917

Andersson007 · 2020-07-27T07:43:51Z

I've just seen your comment. I triggered the tests. in community.general we can relaunch failed tests only, not here unfortunately:(

plugins/modules/mysql_user.py

bmildren · 2020-07-27T18:41:38Z

lgtm 👍

Andersson007

LGTM

Andersson007 · 2020-07-28T06:45:19Z

@Jorge-Rodriguez thanks for the contribution!
@bmildren thanks for reviewing!
Merged

Andersson007 · 2020-07-28T06:47:25Z

@bmildren community.general has been recently released according to ansible-collections/community.general#582
i think it makes sense to release community.mysql 1.0.0 as well

Andersson007 · 2020-07-28T06:52:38Z

https://github.com/ansible/community/wiki/ReleasingCollections#releasing-without-release-branches-for-smaller-collections

bmildren · 2020-07-31T09:33:50Z

yep cool that makes sense 👍

Jorge-Rodriguez mentioned this pull request Jul 23, 2020

Set certificate restrictions via mysql_user #5

Closed

nerijus mentioned this pull request Jul 23, 2020

Add TLS connection parameters ansible-collections/community.general#369

Closed

Andersson007 requested a review from bmildren July 23, 2020 11:49

Add TLS connection parameters

b062f5f

Jorge-Rodriguez force-pushed the tls_require_parameter branch from a18e5ae to b062f5f Compare July 24, 2020 06:23

Jorge-Rodriguez and others added 3 commits July 24, 2020 09:49

Add check mode tests

cde29d2

Fix check mode indentation

7643a37

Run MySQL commands with the mysql_command variable

e7e0354

Jorge-Rodriguez added 3 commits July 25, 2020 13:01

Fix typo

226158a

Restore code lost during cherry pick

5b5fe8e

Fix conditionals to accomodate for MySQL v8

4b4c06b

Fix equal operators

69d3f8b

Jorge-Rodriguez added 3 commits July 26, 2020 10:52

Remove Black formatting in an attempt to make codecov happy

25441df

Remove deprecation notice

91fe5b4

Fix closing bracket

b6e4216

Jorge-Rodriguez force-pushed the tls_require_parameter branch from 8771fd6 to b6e4216 Compare July 26, 2020 08:29

bmildren reviewed Jul 27, 2020

View reviewed changes

plugins/modules/mysql_user.py Outdated Show resolved Hide resolved

Remove code duplication

c62b430

bmildren approved these changes Jul 27, 2020

View reviewed changes

Andersson007 approved these changes Jul 28, 2020

View reviewed changes

Andersson007 merged commit 36e7e66 into ansible-collections:main Jul 28, 2020

bmalynovytch mentioned this pull request Sep 21, 2020

cannot set user password with --strict-password-validation enabled since v2.9.x #21

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add TLS connection parameters #9

Add TLS connection parameters #9

Jorge-Rodriguez commented Jul 23, 2020 •

edited

Loading

Jorge-Rodriguez commented Jul 23, 2020 •

edited

Loading

Andersson007 commented Jul 23, 2020 •

edited

Loading

Andersson007 commented Jul 23, 2020 •

edited

Loading

Andersson007 commented Jul 23, 2020

Andersson007 commented Jul 23, 2020

Andersson007 commented Jul 23, 2020

bmildren commented Jul 23, 2020

Jorge-Rodriguez commented Jul 23, 2020

Andersson007 commented Jul 23, 2020

Andersson007 commented Jul 23, 2020

bmildren commented Jul 23, 2020

Andersson007 commented Jul 23, 2020

bmildren commented Jul 23, 2020

bmildren commented Jul 23, 2020

Jorge-Rodriguez commented Jul 23, 2020

bmildren commented Jul 23, 2020

bmildren commented Jul 23, 2020

codecov bot commented Jul 24, 2020 •

edited

Loading

Jorge-Rodriguez commented Jul 24, 2020

Jorge-Rodriguez commented Jul 25, 2020

Andersson007 commented Jul 26, 2020

Jorge-Rodriguez commented Jul 26, 2020

Jorge-Rodriguez commented Jul 26, 2020 •

edited

Loading

Jorge-Rodriguez commented Jul 26, 2020

Andersson007 commented Jul 27, 2020

Andersson007 commented Jul 27, 2020

Andersson007 commented Jul 27, 2020 •

edited

Loading

bmildren commented Jul 27, 2020

Andersson007 left a comment

Andersson007 commented Jul 28, 2020

Andersson007 commented Jul 28, 2020

Andersson007 commented Jul 28, 2020

bmildren commented Jul 31, 2020

Add TLS connection parameters #9

Add TLS connection parameters #9

Conversation

Jorge-Rodriguez commented Jul 23, 2020 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

Jorge-Rodriguez commented Jul 23, 2020 • edited Loading

Andersson007 commented Jul 23, 2020 • edited Loading

Andersson007 commented Jul 23, 2020 • edited Loading

Andersson007 commented Jul 23, 2020

Andersson007 commented Jul 23, 2020

Andersson007 commented Jul 23, 2020

bmildren commented Jul 23, 2020

Jorge-Rodriguez commented Jul 23, 2020

Andersson007 commented Jul 23, 2020

Andersson007 commented Jul 23, 2020

bmildren commented Jul 23, 2020

Andersson007 commented Jul 23, 2020

bmildren commented Jul 23, 2020

bmildren commented Jul 23, 2020

Jorge-Rodriguez commented Jul 23, 2020

bmildren commented Jul 23, 2020

bmildren commented Jul 23, 2020

codecov bot commented Jul 24, 2020 • edited Loading

Codecov Report

Jorge-Rodriguez commented Jul 24, 2020

Jorge-Rodriguez commented Jul 25, 2020

Andersson007 commented Jul 26, 2020

Jorge-Rodriguez commented Jul 26, 2020

Jorge-Rodriguez commented Jul 26, 2020 • edited Loading

Jorge-Rodriguez commented Jul 26, 2020

Andersson007 commented Jul 27, 2020

Andersson007 commented Jul 27, 2020

Andersson007 commented Jul 27, 2020 • edited Loading

bmildren commented Jul 27, 2020

Andersson007 left a comment

Choose a reason for hiding this comment

Andersson007 commented Jul 28, 2020

Andersson007 commented Jul 28, 2020

Andersson007 commented Jul 28, 2020

bmildren commented Jul 31, 2020

Jorge-Rodriguez commented Jul 23, 2020 •

edited

Loading

Jorge-Rodriguez commented Jul 23, 2020 •

edited

Loading

Andersson007 commented Jul 23, 2020 •

edited

Loading

Andersson007 commented Jul 23, 2020 •

edited

Loading

codecov bot commented Jul 24, 2020 •

edited

Loading

Jorge-Rodriguez commented Jul 26, 2020 •

edited

Loading

Andersson007 commented Jul 27, 2020 •

edited

Loading