Fix production image build in main

[kaxil]

Currently, main branch fails in Production image built with following error

https://github.com/apache/airflow/actions/runs/11489117758/job/31977767455?pr=43331

>           obj, end = self.scan_once(s, idx)
E           json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 6 (char 5)

This is because the test test_required_providers_are_installed relies on output of "airflow providers list --output json":

airflow/docker_tests/test_prod_image.py

Lines 85 to 94 in da7ce4a

	def test_required_providers_are_installed(self, default_docker_image):
	if os.environ.get("TEST_SLIM_IMAGE"):
	packages_to_install = set(SLIM_IMAGE_PROVIDERS)
	package_file = "hatch_build.py"
	else:
	packages_to_install = set(REGULAR_IMAGE_PROVIDERS)
	package_file = PROD_IMAGE_PROVIDERS_FILE_PATH
	assert len(packages_to_install) != 0
	output = run_bash_in_docker("airflow providers list --output json", image=default_docker_image)
	providers = json.loads(output)

and the output in main right now is, notice the warnings:

[2024-10-23T22:45:37.453+0000] {configuration.py:994} WARNING - section/key [database/sql_alchemy_engine_args] not found in config
[2024-10-23T22:45:37.455+0000] {configuration.py:994} WARNING - section/key [core/asset_manager_kwargs] not found in config
  [{"package_name": "apache-airflow-providers-amazon", "description": "Amazon integration (including Amazon Web Services (AWS) https://aws.amazon.com/)", "version": "9.0.0.dev0"}, {"package_name": "apache-airflow-providers-celery", "description": "Celery https://docs.celeryq.dev/en/stable/", "version": "3.8.3.dev0"}, {"package_name": "apache-airflow-providers-cncf-kubernetes", "description": "Kubernetes https://kubernetes.io/", "version": "9.0.0.dev0"}, {"package_name": "apache-airflow-providers-common-compat", "description": "Common Compatibility Provider - providing compatibility code for previous Airflow versions.", "version": "1.2.1.dev0"}, {"package_name": "apache-airflow-providers-common-io", "description": "Common IO Provider", "version": "1.4.2.dev0"}, {"package_name": "apache-airflow-providers-common-sql", "description": "Common SQL Provider https://en.wikipedia.org/wiki/SQL", "version": "1.18.0.dev0"}, {"package_name": "apache-airflow-providers-docker", "description": "Docker https://www.docker.com/", "version": "3.14.0.dev0"}, {"package_name": "apache-airflow-providers-elasticsearch", "description": "Elasticsearch https://www.elastic.co/elasticsearch", "version": "5.5.2.dev0"}, {"package_name": "apache-airflow-providers-fab", "description": "Flask App Builder https://flask-appbuilder.readthedocs.io/", "version": "1.4.1.dev0"}, {"package_name": "apache-airflow-providers-ftp", "description": "File Transfer Protocol (FTP) https://tools.ietf.org/html/rfc114", "version": "3.11.1.dev0"}, {"package_name": "apache-airflow-providers-google", "description": "Google services including:\n\n  - Google Ads https://ads.google.com/\n  - Google Cloud (GCP) https://cloud.google.com/\n  - Google Firebase https://firebase.google.com/\n  - Google LevelDB https://github.com/google/leveldb/\n  - Google Marketing Platform https://marketingplatform.google.com/\n  - Google Workspace https://workspace.google.com/ (formerly Google Suite)", "version": "10.24.0.dev0"}, {"package_name": "apache-airflow-providers-grpc", "description": "gRPC https://grpc.io/", "version": "3.6.0.dev0"}, {"package_name": "apache-airflow-providers-hashicorp", "description": "Hashicorp including Hashicorp Vault https://www.vaultproject.io/", "version": "3.8.0.dev0"}, {"package_name": "apache-airflow-providers-http", "description": "Hypertext Transfer Protocol (HTTP) https://www.w3.org/Protocols/", "version": "4.13.1.dev0"}, {"package_name": "apache-airflow-providers-imap", "description": "Internet Message Access Protocol (IMAP) https://tools.ietf.org/html/rfc3501", "version": "3.7.0.dev0"}, {"package_name": "apache-airflow-providers-microsoft-azure", "description": "Microsoft Azure https://azure.microsoft.com/", "version": "10.5.1.dev0"}, {"package_name": "apache-airflow-providers-mysql", "description": "MySQL https://www.mysql.com/", "version": "5.7.2.dev0"}, {"package_name": "apache-airflow-providers-odbc", "description": "ODBC https://github.com/mkleehammer/pyodbc/wiki", "version": "4.7.1.dev0"}, {"package_name": "apache-airflow-providers-openlineage", "description": "OpenLineage https://openlineage.io/", "version": "1.12.2.dev0"}, {"package_name": "apache-airflow-providers-postgres", "description": "PostgreSQL https://www.postgresql.org/", "version": "5.13.1.dev0"}, {"package_name": "apache-airflow-providers-redis", "description": "Redis https://redis.io/", "version": "3.8.0.dev0"}, {"package_name": "apache-airflow-providers-sendgrid", "description": "Sendgrid https://sendgrid.com/", "version": "3.6.0.dev0"}, {"package_name": "apache-airflow-providers-sftp", "description": "SSH File Transfer Protocol (SFTP) https://tools.ietf.org/wg/secsh/draft-ietf-secsh-filexfer/", "version": "4.11.1.dev0"}, {"package_name": "apache-airflow-providers-slack", "description": "Slack https://slack.com/ services integration including:\n\n  - Slack API https://api.slack.com/\n  - Slack Incoming Webhook https://api.slack.com/messaging/webhooks", "version": "8.9.0.dev0"}, {"package_name": "apache-airflow-providers-smtp", "description": "Simple Mail Transfer Protocol (SMTP) https://tools.ietf.org/html/rfc5321", "version": "1.8.0.dev0"}, {"package_name": "apache-airflow-providers-snowflake", "description": "Snowflake https://www.snowflake.com/", "version": "5.8.0.dev0"}, {"package_name": "apache-airflow-providers-sqlite", "description": "SQLite https://www.sqlite.org/", "version": "3.9.0.dev0"}, {"package_name": "apache-airflow-providers-ssh", "description": "Secure Shell (SSH) https://tools.ietf.org/html/rfc4251", "version": "3.13.1.dev0"}, {"package_name": "apache-airflow-providers-standard", "description": "Airflow Standard Provider", "version": "1.0.0.dev0"}]

To fix this and make test more resilient, we change Airflow logging level to ERROR so warnings aren't shown on CLI, since we use its output

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[kaxil]

I will debug the warnings separately

[potiuk]

Good quick fix - but I think the warning should be eventually removed via @suppress_logs_and_warnings

Let me see if I can see why it moght not work.

[kaxil]

Figured out, it was because of the change in #43040

Good quick fix - but I think the warning should be eventually removed via @suppress_logs_and_warnings

Let me see if I can see why it moght not work.