Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Azure data explorer #119089

Merged
merged 9 commits into from
Jun 11, 2024
Merged

Fix Azure data explorer #119089

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
58f3a71
chore: Refactor Azure Data Explorer client initialization
kaareseras Jun 6, 2024
5990240
Refactor Azure Data Explorer client initialization
kaareseras Jun 7, 2024
e47c458
Merge branch 'dev' into adx-bug
kaareseras Jun 7, 2024
f55cf36
fixed filter
kaareseras Jun 7, 2024
b9edda6
fixed data bug
kaareseras Jun 7, 2024
0cd5999
Merge branch 'dev' into adx-bug
kaareseras Jun 7, 2024
af7378c
chore: Fix typos and improve consistency in Azure Data Explorer clien…
kaareseras Jun 8, 2024
53d194e
refactoring client
kaareseras Jun 10, 2024
f39c922
Use Booleanselector
edenhaus Jun 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions homeassistant/components/azure_data_explorer/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,13 +62,13 @@ async def async_setup(hass: HomeAssistant, yaml_config: ConfigType) -> bool:

Adds an empty filter to hass data.
Tries to get a filter from yaml, if present set to hass data.
If config is empty after getting the filter, return, otherwise emit
deprecated warning and pass the rest to the config flow.
"""

hass.data.setdefault(DOMAIN, {DATA_FILTER: {}})
if DOMAIN in yaml_config:
hass.data[DOMAIN][DATA_FILTER] = yaml_config[DOMAIN][CONF_FILTER]
hass.data.setdefault(DOMAIN, {DATA_FILTER: FILTER_SCHEMA({})})
if DOMAIN not in yaml_config:
return True
hass.data[DOMAIN][DATA_FILTER] = yaml_config[DOMAIN].pop(CONF_FILTER)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if DOMAIN not in yaml_config:
return True
hass.data[DOMAIN][DATA_FILTER] = yaml_config[DOMAIN].pop(CONF_FILTER)
if DOMAIN in yaml_config:
hass.data[DOMAIN][DATA_FILTER] = yaml_config[DOMAIN].pop(CONF_FILTER)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Im not sure why, but this change will cause many test to fail

FAILED tests/components/azure_data_explorer/test_init.py::test_connection[Exception] - AssertionError: assert <ConfigEntryState.NOT_LOADED: 'not_loaded'> == <ConfigEntryState.SETUP_ERROR: 'setup_error'>

  • where <ConfigEntryState.NOT_LOADED: 'not_loaded'> = .state
  • and <ConfigEntryState.SETUP_ERROR: 'setup_error'> = ConfigEntryState.SETUP_ERROR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests are running successfully on my machine with the above change. Can you please recheck?

ha-venv@edenhaus ➜ /workspaces/core (pr/kaareseras/119089) $ pytest tests/components/azure_data_explorer
Test session starts (platform: linux, Python 3.12.3, pytest 8.2.0, pytest-sugar 1.0.0)
rootdir: /workspaces/core
configfile: pyproject.toml
plugins: mock-3.14.0, rerunfailures-14.0, typeguard-4.2.1, aiohttp-1.0.5, picked-0.5.0, requests-mock-1.12.1, respx-0.21.1, cov-5.0.0, sugar-1.0.0, timeout-2.3.1, socket-0.7.0, unordered-0.6.0, github-actions-annotate-failures-0.2.0, syrupy-4.6.1, xdist-3.6.1, asyncio-0.23.6, pytest_freezer-0.4.8, anyio-4.3.0
asyncio: mode=Mode.AUTO
collected 19 items                                                                                                                                          

 tests/components/azure_data_explorer/test_config_flow.py ✓✓✓                                                                                  16% █▋        
 tests/components/azure_data_explorer/test_init.py ✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓✓                                                                           100% ██████████

Results (1.12s):
      19 passed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing :-)

Results (3.27s):
11 passed
8 failed
- tests/components/azure_data_explorer/test_config_flow.py:15 test_config_flow
- tests/components/azure_data_explorer/test_init.py:153 test_filter[allowlist]
- tests/components/azure_data_explorer/test_init.py:153 test_filter[denylist]
- tests/components/azure_data_explorer/test_init.py:153 test_filter[filtered_allowlist]
- tests/components/azure_data_explorer/test_init.py:153 test_filter[filtered_denylist]
- tests/components/azure_data_explorer/test_init.py:268 test_connection[KustoServiceError]
- tests/components/azure_data_explorer/test_init.py:268 test_connection[KustoAuthenticationError]
- tests/components/azure_data_explorer/test_init.py:268 test_connection[Exception]

and when debugging the code, I get this error:

2024-06-10 16:45:40.022 ERROR (MainThread) [homeassistant.setup] Error during setup of component azure_data_explorer
Traceback (most recent call last):
File "/workspaces/core/homeassistant/setup.py", line 404, in _async_setup_component
result = await task
^^^^^^^^^^
File "/workspaces/core/homeassistant/components/azure_data_explorer/init.py", line 69, in async_setup
hass.data[DOMAIN][DATA_FILTER] = yaml_config[DOMAIN].pop(CONF_FILTER)
~~~~~~~~~~~^^^^^^^^
KeyError: 'azure_data_explorer'

And the integration does not load

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is something wrong with your dev environment. Can you please check that your are using the latest version of all requirements


return True


Expand Down Expand Up @@ -207,6 +207,6 @@ def _parse_event(
if "\n" in state.state:
return None, dropped + 1

json_event = str(json.dumps(obj=state, cls=JSONEncoder).encode("utf-8"))
json_event = json.dumps(obj=state, cls=JSONEncoder)

return (json_event, dropped)
43 changes: 31 additions & 12 deletions homeassistant/components/azure_data_explorer/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
CONF_APP_REG_ID,
CONF_APP_REG_SECRET,
CONF_AUTHORITY_ID,
CONF_USE_FREE,
CONF_USE_QUEUED_CLIENT,
)

_LOGGER = logging.getLogger(__name__)
Expand All @@ -35,7 +35,14 @@ class AzureDataExplorerClient:
def __init__(self, data: Mapping[str, Any]) -> None:
"""Create the right class."""

# User provided data with the Ingest- in the URI
self._cluster_ingest_uri = data[CONF_ADX_CLUSTER_INGEST_URI]

# Remove the Ingest- from the URI
self._cluster_query_uri = data[CONF_ADX_CLUSTER_INGEST_URI].replace(
"ingest-", ""
)

kaareseras marked this conversation as resolved.
Show resolved Hide resolved
self._database = data[CONF_ADX_DATABASE_NAME]
self._table = data[CONF_ADX_TABLE_NAME]
self._ingestion_properties = IngestionProperties(
Expand All @@ -45,24 +52,36 @@ def __init__(self, data: Mapping[str, Any]) -> None:
ingestion_mapping_reference="ha_json_mapping",
)

# Create cLient for ingesting and querying data
kcsb = KustoConnectionStringBuilder.with_aad_application_key_authentication(
self._cluster_ingest_uri,
data[CONF_APP_REG_ID],
data[CONF_APP_REG_SECRET],
data[CONF_AUTHORITY_ID],
# Create cLient for ingesting data
kaareseras marked this conversation as resolved.
Show resolved Hide resolved
kcsb_ingest = (
KustoConnectionStringBuilder.with_aad_application_key_authentication(
self._cluster_ingest_uri,
kaareseras marked this conversation as resolved.
Show resolved Hide resolved
data[CONF_APP_REG_ID],
data[CONF_APP_REG_SECRET],
data[CONF_AUTHORITY_ID],
)
)

# Create cLient for querying data
kaareseras marked this conversation as resolved.
Show resolved Hide resolved
kcsb_query = (
KustoConnectionStringBuilder.with_aad_application_key_authentication(
self._cluster_query_uri,
kaareseras marked this conversation as resolved.
Show resolved Hide resolved
data[CONF_APP_REG_ID],
data[CONF_APP_REG_SECRET],
data[CONF_AUTHORITY_ID],
)
)

if data[CONF_USE_FREE] is True:
if data[CONF_USE_QUEUED_CLIENT] is True:
# Queded is the only option supported on free tear of ADX
self.write_client = QueuedIngestClient(kcsb)
self.write_client = QueuedIngestClient(kcsb_ingest)
else:
self.write_client = ManagedStreamingIngestClient.from_dm_kcsb(kcsb)
self.write_client = ManagedStreamingIngestClient.from_dm_kcsb(kcsb_ingest)

self.query_client = KustoClient(kcsb)
self.query_client = KustoClient(kcsb_query)

def test_connection(self) -> None:
"""Test connection, will throw Exception when it cannot connect."""
"""Test connection, will throw Exception if it cannot connect."""

query = f"{self._table} | take 1"

Expand Down
4 changes: 2 additions & 2 deletions homeassistant/components/azure_data_explorer/config_flow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
CONF_APP_REG_ID,
CONF_APP_REG_SECRET,
CONF_AUTHORITY_ID,
CONF_USE_FREE,
CONF_USE_QUEUED_CLIENT,
DEFAULT_OPTIONS,
DOMAIN,
)
Expand All @@ -34,7 +34,7 @@
vol.Required(CONF_APP_REG_ID): str,
vol.Required(CONF_APP_REG_SECRET): str,
vol.Required(CONF_AUTHORITY_ID): str,
vol.Optional(CONF_USE_FREE, default=False): bool,
vol.Optional(CONF_USE_QUEUED_CLIENT, default=False): bool,
}
)

Expand Down
2 changes: 1 addition & 1 deletion homeassistant/components/azure_data_explorer/const.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
CONF_SEND_INTERVAL = "send_interval"
CONF_MAX_DELAY = "max_delay"
CONF_FILTER = DATA_FILTER = "filter"
CONF_USE_FREE = "use_queued_ingestion"
CONF_USE_QUEUED_CLIENT = "use_queued_ingestion"
DATA_HUB = "hub"
STEP_USER = "user"

Expand Down
20 changes: 12 additions & 8 deletions homeassistant/components/azure_data_explorer/strings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,21 @@
"config": {
"step": {
"user": {
"title": "Setup your Azure Data Explorer integration",
"description": "Enter connection details.",
"data": {
"cluster_ingest_uri": "Cluster ingest URI",
"database": "Database name",
"table": "Table name",
"cluster_ingest_uri": "Cluster Ingest URI ",
"authority_id": "Authority ID",
"client_id": "Client ID",
"client_secret": "Client secret",
"authority_id": "Authority ID",
"use_queued_ingestion": "Use queued ingestion"
}
"database": "Database name",
"table": "Table name",
"use_queued_ingestion": "Use Queued ingestion (Mandotory for free cluster)"
kaareseras marked this conversation as resolved.
Show resolved Hide resolved
kaareseras marked this conversation as resolved.
Show resolved Hide resolved
},
"data_description": {
"cluster_ingest_uri": "Ingest-URI of the cluster, use the 'https://ingest-xxxx.xx.xx' URI",
kaareseras marked this conversation as resolved.
Show resolved Hide resolved
"use_queued_ingestion": "Check if using free cluster"
},
"description": "Enter connection details.",
"title": "Setup your Azure Data Explorer integration"
kaareseras marked this conversation as resolved.
Show resolved Hide resolved
}
},
"error": {
Expand Down
8 changes: 4 additions & 4 deletions tests/components/azure_data_explorer/const.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
CONF_APP_REG_SECRET,
CONF_AUTHORITY_ID,
CONF_SEND_INTERVAL,
CONF_USE_FREE,
CONF_USE_QUEUED_CLIENT,
)

AZURE_DATA_EXPLORER_PATH = "homeassistant.components.azure_data_explorer"
Expand All @@ -29,7 +29,7 @@
}

BASIC_OPTIONS = {
CONF_USE_FREE: False,
CONF_USE_QUEUED_CLIENT: False,
CONF_SEND_INTERVAL: 5,
}

Expand All @@ -39,10 +39,10 @@

BASE_CONFIG_IMPORT = {
CONF_ADX_CLUSTER_INGEST_URI: "https://cluster.region.kusto.windows.net",
CONF_USE_FREE: False,
CONF_USE_QUEUED_CLIENT: False,
CONF_SEND_INTERVAL: 5,
}

FREE_OPTIONS = {CONF_USE_FREE: True, CONF_SEND_INTERVAL: 5}
FREE_OPTIONS = {CONF_USE_QUEUED_CLIENT: True, CONF_SEND_INTERVAL: 5}

BASE_CONFIG_FREE = BASE_CONFIG | FREE_OPTIONS