add service name to plugin pam/auth_pam usage

[hubiongithub]

SUMMARY

for plugin = pam (MariaDB)
or plugin = auth_pam (MySQL)
use an optional plugin_auth_string to set the pam service to use

CREATE USER user IDENTIFIED WITH plugin USING plugin_auth_string

As MySQL and MariaDB store this option in authentication_string column this reuse of plugin_auth_string seems vaild.

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

mysql_user

ADDITIONAL INFORMATION

would need help if this should have a unit/integration test

[codecov]

Codecov Report

Merging #445 (eade7ec) into main (eade7ec) will not change coverage.
The diff coverage is n/a.

❗ Current head eade7ec differs from pull request most recent head 8856f4d. Consider uploading reports for the commit 8856f4d to get more accurate results

@@           Coverage Diff           @@
##             main     #445   +/-   ##
=======================================
  Coverage   73.11%   73.11%           
=======================================
  Files          28       28           
  Lines        2332     2332           
  Branches      548      548           
=======================================
  Hits         1705     1705           
  Misses        449      449           
  Partials      178      178           

Flag	Coverage Δ
integration	69.91% <0.00%> (ø)
sanity	15.32% <0.00%> (ø)
units	33.55% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[hubiongithub]

Hello
I'm stuck here, it tries to check things, it says 4 failing, but on what, and how can I fix those?
I need help. @Andersson007 can you have a look where this fails?

[laurent-indermuehle]

@hubiongithub in the sanity test, I can see this:

Running sanity test "ansible-doc"
Run command: ansible-doc -t module community.mysql.mysql_role community.mysql.mysql_user
ERROR: Command "ansible-doc -t module community.mysql.mysql_role community.mysql.mysql_user" returned exit status 1.
>>> Standard Error
ERROR! module community.mysql.mysql_user missing documentation (or could not parse documentation): while scanning a simple key
  in "<unicode string>", line 132, column 9
could not find expected ':'
  in "<unicode string>", line 133, column 5

You can click on "Details" next to each test to see the log. It's often very long and hard to work with. Using the search often help.

[hubiongithub]

Hello @laurent-indermuehle
I see "Plugins CI / Sanity (Ansible: stable-2.12) (pull_request) Failing after 1m" has failed, in details I see at last 3 errors

Running sanity test "ansible-doc"
Run command: ansible-doc -t module community.mysql.mysql_role community.mysql.mysql_user
ERROR: Command "ansible-doc -t module community.mysql.mysql_role community.mysql.mysql_user" returned exit status 1.
>>> Standard Error
ERROR! module community.mysql.mysql_user missing documentation (or could not parse documentation): while scanning a simple key
  in "<unicode string>", line 132, column 9
could not find expected ':'
  in "<unicode string>", line 133, column 5

---

Running sanity test "validate-modules"
Run command: /root/.ansible/test/venv/sanity.validate-modules/3.9/5e1e301c/bin/python /root/ansible/test/lib/ansible_test/_util/controller/tools/collection_detail.py /root/ansible_collections/community/mysql
Run command: /root/.ansible/test/venv/sanity.validate-modules/3.9/5e1e301c/bin/python /root/ansible/test/lib/ansible_test/_util/controller/sanity/validate-modules/validate-modules --format json --arg-spec plugins/modules/mysql_role.py plugins/modules/mysql_user.py --collection ansible_collections/community/mysql --collection-version 3.5.1
ERROR: Command "/root/.ansible/test/venv/sanity.validate-modules/3.9/5e1e301c/bin/python /root/ansible/test/lib/ansible_test/_util/controller/sanity/validate-modules/validate-modules --format json --arg-spec plugins/modules/mysql_role.py plugins/modules/mysql_user.py --collection ansible_collections/community/mysql --collection-version 3.5.1" returned exit status 1.
>>> Standard Error
Traceback (most recent call last):
  File "/root/ansible/test/lib/ansible_test/_util/controller/sanity/validate-modules/validate_modules/utils.py", line 154, in parse_yaml
    data = yaml_load(value, Loader=loader)
  File "/root/.ansible/test/venv/sanity.validate-modules/3.9/5e1e301c/lib/python3.9/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/root/.ansible/test/venv/sanity.validate-modules/3.9/5e1e301c/lib/python3.9/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()

---

AttributeError: attribute 'line' of 'yaml._yaml.Mark' objects is not writable
Removing custom PyPI config: /root/.pydistutils.cfg
Removing custom PyPI config: /root/.pip/pip.conf
Run command: docker exec -i ansible-test-controller-0hJXD45U sh -c 'tar cf - -C /root/ansible_collections/community/mysql/tests --exclude .tmp output | gzip'
Run command: tar oxzf - -C /home/runner/work/community.mysql/community.mysql/ansible_collections/community/mysql/tests
ERROR: Command "docker exec ansible-test-controller-0hJXD45U /usr/bin/env ANSIBLE_TEST_CONTENT_ROOT=/root/ansible_collections/community/mysql LC_ALL=en_US.UTF-8 /usr/bin/python3.9 /root/ansible/bin/ansible-test sanity -v --containers '{"control": {"__pypi_proxy__": {"pypi-test-container-0hJXD45U": {"host_ip": "172.17.0.2", "names": ["pypi-test-container-0hJXD45U"], "ports": [3141]}}}, "managed": {"__pypi_proxy__": {"pypi-test-container-0hJXD45U": {"host_ip": "172.17.0.2", "names": ["pypi-test-container-0hJXD45U"], "ports": [3141]}}}}' --junit --truncate 0 --color yes --host-path tests/output/.tmp/host-_2o9m6n1 --metadata tests/output/.tmp/metadata-dnes7w5e.json --require plugins/module_utils/user.py --require plugins/modules/mysql_role.py --require plugins/modules/mysql_user.py --require tests/unit/plugins/module_utils/test_mysql_user.py --base-branch main" returned exit status 1.
Run command: docker rm -f ansible-test-controller-0hJXD45U
Run command: docker rm -f 2c6a120f2cbe697066082f4a61140cc275709b799876a7e32e80499d69ec5148
Error: Process completed with exit code 1.

But where do I start to look what of my changes may have caused this?

[laurent-indermuehle]

@hubiongithub I've no idea. I pulled your branch and was able to reproduce the GitHub Action error using this command:

set -x ; ~/.local/bin/ansible-test sanity -v --color --changed --base-branch main --junit --docker; set +x

There is something in this block that cause the issue: https://github.com/hubiongithub/community.mysql/blob/add_pam_service_option/plugins/modules/mysql_user.py#L17-L178
But I can't find where. I searched for missing quotes, unquoted ":" and many other things... I'm not a YAML into Python documentalist expert :/

[laurent-indermuehle]

@hubiongithub please try to remove the ":" after the "MariaDB" on this line: https://github.com/hubiongithub/community.mysql/blob/add_pam_service_option/plugins/modules/mysql_user.py#L141
I tried to use single-quote, double-quote. The only way I got green was by removing this semi-colon.

[hubiongithub]

@laurent-indermuehle oh, ok, removed the ":" lets see what else might geht wrong.
Thanks for looking into it.

[hubiongithub]

Hello
@laurent-indermuehle on the github website it now says: All checks have passed
but I still get emails with: Roles CI / Molecule (Python: 3.6, Ansible: stable-2.11, MySQL: 2.0.12) Failed in 3 seconds
the URL to github in that email:

Run actions/setup-python@v2
Version 3.6 was not found in the local cache
Error: Version 3.6 with arch x64 not found
The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json

I think I'm not able to fix that on my side.

[Andersson007]

@hubiongithub hello, thanks for the PR! and sorry for the late response - was on long PTO.

Could you please:

• elaborate a little bit more on the issue: I'm not a user, so would be good to understand the case better. Do i understand correctly that when pam and there's MariaDB, we need to use USING instead of BY? If yes, can this be fair for other plugins (not pam), i.e. we always need to use USING when working with any plugin and MariaDB? If it can, should we introduce a new argument to the functions determining RDBMS we're using and make the conditions working based on that argument? Would be grateful for any explanation
• add a changelog fragment
• add integration tests for the case (see the content of tests/integration/targets/test_mysql_user/tasks; you can also find there examples of using condition determining if it's mysql or mariadb)

If you have any questions, we'd be happy to help

[Andersson007]

Suggested change

	- MariaDB If I(plugin) is ``pam`` an optional I(plugin_auth_string) ults in (``CREATE USER user IDENTIFIED WITH plugin USING plugin_auth_string``).
	This was chosen because MySQL and MariaDB store the ``USING plugin_auth_string`` part in the ``authentication_string`` column in ``mysql.user``.
	- MariaDB If I(plugin) is ``pam`` an optional I(plugin_auth_string) ults in (``CREATE USER user IDENTIFIED WITH plugin USING plugin_auth_string``).
	This was chosen because MySQL and MariaDB store the ``USING plugin_auth_string`` part in the ``authentication_string`` column in ``mysql.user``.

I would re-formulate these sentences somehow and fix a couple of typos.

[hubiongithub]

I removed the technical parts of this, now its only describing the special use case of "plugin_auth_string" for pam/auth_pam.
You're right, for a user it does not matter what SQL this generates a lang as they know how to use the module.

[hubiongithub]

@Andersson007
MariaDB and MySQL have different Syntax when it comes to create users with some plugins
With PAM MariadB uses the plugin name "pam"and MySQL uses "auth_pam" (maybe that changed to "authentication_pam" as the link below requests.)

• MariaDB: https://mariadb.com/kb/en/authentication-plugin-pam/#creating-users
  CREATE USER username@hostname IDENTIFIED VIA pam USING 'mariadb';
• https://dev.mysql.com/doc/refman/8.0/en/pam-pluggable-authentication.html#pam-pluggable-authentication-usage
  CREATE USER user IDENTIFIED WITH authentication_pam AS 'auth_string';

The "USING " / "AS " are optional, without it PAM will test the user against all pam modules defined, which may lead to a lot of "unknown user" messages in the system log for every configured pam module.

For other plugins (https://mariadb.com/kb/en/authentication-plugins/) MariaDB seems to always use
IDENTIFIED VIA modulename USING string

MySQL seams to uses
IDENTIFIED WITH plugin name AS string (e.g. https://dev.mysql.com/doc/refman/8.0/en/pam-pluggable-authentication.html)
IDENTIFIED WITH plugin name BY string (e.g. https://dev.mysql.com/doc/refman/8.0/en/kerberos-pluggable-authentication.html)

MariaDB and MySQL uses a different Syntax for "normal" password plugins:

• https://mariadb.com/kb/en/authentication-plugin-mysql_native_password/
  CREATE USER username@hostname IDENTIFIED BY 'mariadb';
  This is when the mysql.user module uses user/password instead of plugin parameters

I tested my changes (some time ago), but now having reread the documentation links above I'm not sure if this part here is correct:

        query_with_args = "CREATE USER %s@%s IDENTIFIED WITH %s AS %s", (user, host, plugin, plugin_hash_string)
    elif plugin and plugin_auth_string:
        # Mysql and MariaDB differ in naming pam plugin and Syntax to set it
        if plugin == 'pam':
            query_with_args = "CREATE USER %s@%s IDENTIFIED WITH %s USING %s", (user, host, plugin, plugin_auth_string)
        else:
            query_with_args = "CREATE USER %s@%s IDENTIFIED WITH %s BY %s", (user, host, plugin, plugin_auth_string)
    elif plugin:
        query_with_args = "CREATE USER %s@%s IDENTIFIED WITH %s", (user, host, plugin)
    else:
        query_with_args = "CREATE USER %s@%s", (user, host)

esp. the mysql part when "plugin and plugin_auth_string" are set (should use "AS" not "BY")
and the mariadb part (should use VIA instead of WITH)

I probably will test this again perhaps the MariaDB/MySQL documentation and real life differs.
I will fix typos I find and add a changelog fragment. On the integration tests I will need help.

[hubiongithub]

Manual tests:
MySQL 8.0.30-22
Both syntax
create user user1@localhost IDENTIFIED WITH 'auth_pam' AS 'service';
create user user1@localhost IDENTIFIED WITH 'auth_pam' BY 'service';

Resulst in mysql.user table entry (select user,host,plugin,authentication_string from mysql.user where plugin = 'auth_pam';)
| user1 | localhost | auth_pam | service |

/var/log/auth.log

mysqld: pam_krb5(oedivmysql:auth): user user1 authenticated as user1@
mysqld: pam_krb5(oedivmysql:auth): user user1 authenticated as user1@

without pam service:
create user user1@localhost IDENTIFIED WITH 'auth_pam';

Resulst in mysql.user table entry
| user1 | localhost | auth_pam | |
login does work, but the user might be authenticated in every possible pam module configured

MariaDB 10.6:
MariaDB [(none)]> create user user1@localhost IDENTIFIED WITH 'pam';
MariaDB [(none)]> create user user1@localhost IDENTIFIED VIA 'pam';
select user,host,plugin,authentication_string from mysql.user where plugin = 'pam';
delivers
| user1 | localhost | auth_pam | |

MariaDB [(none)]> create user user1@localhost IDENTIFIED VIA 'pam' USING 'service';
MariaDB [(none)]> create user user1@localhost IDENTIFIED WITH 'pam' USING 'service';
delivers
| user1 | localhost | auth_pam | service |

all 4 created an user which can login, the first two uses any pam module, the later two only the configured one.

so the code above is functional, as both will accept IDENTIFIED WITH and differ in USING/BY

MariaDB does not accept: create user user1@localhost IDENTIFIED WITH 'pam' BY 'Service';
ERROR 1064 ... near 'BY...'

[hubiongithub]

@Andersson007
A short look into "tests/integration/targets/test_mysql_user/tasks" I see

• create_user.yml
• test_user_plugin_auth.yml
  both included from main.yml

test_user_plugin_auth.yml is probably the right place to start, but only tests "sha256_password" and "mysql_native_password", but nothing else, probably because anything else would need more requirements outside the MySLQ/MariaDB instance, pam plugin would need a running pam server configured with a certain service so a created user can be tested.
Against what kind of OS and MySQL/MariaDB installations are these tests executed? From the output of the automatic tests my code failed, I assume docker containers, can these use pamd inside?

[Andersson007]

one small thing to make the entry looking consistent with the other changelog entries

[Andersson007]

@hubiongithub thanks for the investigating, testing and explaining!
I think we can skip adding integration tests as it requires additional service installation which i'd like to avoid to keep CI simpler:)
Is the PR ready for merge?

[hubiongithub]

@Andersson007 is there anything to do to get a PR "ready to merge"?
the sync fork button shows: "This branch is not behind the upstream ansible-collections:main"

From the point of what did I want to change its ready, from what github might need to be done I'm not sure.

[Andersson007]

@hubiongithub thanks for the contribution!
If there are any related issues that can be closed/updated, please let me know/feel free to put updates in them

[Andersson007]

@hubiongithub fyi we have the #mysql:ansible.com matrix channel https://docs.ansible.com/ansible/devel/community/communication.html#ansible-community-on-matrix, feel free to join

[Andersson007]

@hubiongithub what do you think about #484 (comment) , you use it so you're feedback would be much appreciated there