Error while setting up uk_bin_collection platform for sensor

[davelegg83]

Name of Council

Dover District Council

Issue Information

Installed via HACS and HA restarted.

Location - Home
Council - Dover District Council

URN - python collect_data.py DoverDistrictCouncil https://collections.dover.gov.uk/property/XXXXXXXXXXX

I replaced xxxx with my UPRN. I can also go direct to https://collections.dover.gov.uk/property/XXXXXXXXXXX using my UPRN and it loads up my collection dates.

However when all submitted it shows no devices or entities.

In the log I have these errors:

`Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:356
integration: Sensor (documentation, issues)
First occurred: 09:41:52 (9 occurrences)
Last logged: 10:08:13

Error while setting up uk_bin_collection platform for sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 315, in _async_refresh
self.data = await self._async_update_data()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/uk_bin_collection/sensor.py", line 133, in _async_update_data
data = await self.hass.async_add_executor_job(self.ukbcd.run)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 96, in run
return self.client_code(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 115, in client_code
return get_bin_data_class.template_method(address_url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 65, in template_method
page = self.get_data(address_url)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 122, in get_data
full_page = requests.get(url, headers, verify=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 697, in send
adapter = self.get_adapter(url=request.url)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 794, in get_adapter
raise InvalidSchema(f"No connection adapters were found for {url!r}")
requests.exceptions.InvalidSchema: No connection adapters were found for 'python collect_data.py DoverDistrictCouncil https://collections.dover.gov.uk/property/100060906311'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 356, in _async_setup_platform
await asyncio.shield(awaitable)
File "/config/custom_components/uk_bin_collection/sensor.py", line 75, in async_setup_entry
await coordinator.async_config_entry_first_refresh()
File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 287, in async_config_entry_first_refresh
raise ex
homeassistant.exceptions.ConfigEntryNotReady: No connection adapters were found for 'python collect_data.py DoverDistrictCouncil https://collections.dover.gov.uk/property/1000'`

and

`Logger: uk_bin_collection.uk_bin_collection.get_bin_data
Source: /usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py:134
First occurred: 09:41:52 (9 occurrences)
Last logged: 10:08:13

Oops: Something Else No connection adapters were found for 'python collect_data.py DoverDistrictCouncil https://collections.dover.gov.uk/property/10006'`

I've removed my UPRN from the code.

Any help greatly appreciated.

Verification

• I searched for similar issues at https://github.com/robbrad/UKBinCollectionData/issues?q=is:issue and found no duplicates
• I have checked my address/postcode/UPRN works on the council's website
• I have provided a detailed explanation of the issue as well as steps to replicate the issue
• I understand that this project is run by volunteer contributors therefore completion of this issue cannot be guaranteed

[robbrad]

Found the issue its because the parser thats been developed cant handle the dash in the date

Code needs to be changed to handle this

[davelegg83]

Thanks @robbrad - Do you know how long it takes to get pushed through, as tried yesterday and today and still no joy

[robbrad]

@davelegg83 It's done - just upgrade the custom component and you should be good

[davelegg83]

Just uninstalled, re-downloaded and set up and still no joy im afraid @robbrad - These are the errors in the log.

Logger: uk_bin_collection.uk_bin_collection.get_bin_data
Source: /usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py:141
First occurred: 12:55:35 (1 occurrences)
Last logged: 12:55:35

Oops: Something Else No connection adapters were found for 'python collect_data.py DoverDistrictCouncil https://collections.dover.gov.uk/property/100060906311'

Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:356
integration: Sensor (documentation, issues)
First occurred: 12:55:35 (1 occurrences)
Last logged: 12:55:35

Error while setting up uk_bin_collection platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 315, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/uk_bin_collection/sensor.py", line 133, in _async_update_data
    data = await self.hass.async_add_executor_job(self.ukbcd.run)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 102, in run
    return self.client_code(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 121, in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 67, in template_method
    page = self.get_data(address_url)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 129, in get_data
    full_page = requests.get(url, headers, verify=False)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 697, in send
    adapter = self.get_adapter(url=request.url)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 794, in get_adapter
    raise InvalidSchema(f"No connection adapters were found for {url!r}")
requests.exceptions.InvalidSchema: No connection adapters were found for 'python collect_data.py DoverDistrictCouncil https://collections.dover.gov.uk/property/100060906311'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 356, in _async_setup_platform
    await asyncio.shield(awaitable)
  File "/config/custom_components/uk_bin_collection/sensor.py", line 75, in async_setup_entry
    await coordinator.async_config_entry_first_refresh()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 287, in async_config_entry_first_refresh
    raise ex
homeassistant.exceptions.ConfigEntryNotReady: No connection adapters were found for 'python collect_data.py DoverDistrictCouncil https://collections.dover.gov.uk/property/100060906311'

[robbrad]

@davelegg83 - now that is a different error !

[davelegg83]

@davelegg83 - now that is a different error !

Sorry to be a pain!

[robbrad]

Your not a pain its fine

Its working for me with you UPRN

[robbrad]

Im wondering if its cos you are on Python 3.12 - we dont yet support that

Ignore that my test Home Assistant is Python 3.12

[davelegg83]

Just to double check i've got the basics right!
Location - Home
Council - Dover District Council
Council details (as per wiki) - python collect_data.py DoverDistrictCouncil https://collections.dover.gov.uk/property/100060906311

Just redownloaded and set up again, but no joy.

Logger: homeassistant.components.sensor
Source: helpers/entity_platform.py:356
integration: Sensor (documentation, issues)
First occurred: 22:45:04 (2 occurrences)
Last logged: 22:47:36

Error while setting up uk_bin_collection platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 315, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/uk_bin_collection/sensor.py", line 133, in _async_update_data
    data = await self.hass.async_add_executor_job(self.ukbcd.run)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 102, in run
    return self.client_code(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 121, in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 67, in template_method
    page = self.get_data(address_url)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 129, in get_data
    full_page = requests.get(url, headers, verify=False)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 697, in send
    adapter = self.get_adapter(url=request.url)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/requests/sessions.py", line 794, in get_adapter
    raise InvalidSchema(f"No connection adapters were found for {url!r}")
requests.exceptions.InvalidSchema: No connection adapters were found for 'python collect_data.py DoverDistrictCouncil https://collections.dover.gov.uk/property/100060906311'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 356, in _async_setup_platform
    await asyncio.shield(awaitable)
  File "/config/custom_components/uk_bin_collection/sensor.py", line 75, in async_setup_entry
    await coordinator.async_config_entry_first_refresh()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 287, in async_config_entry_first_refresh
    raise ex
homeassistant.exceptions.ConfigEntryNotReady: No connection adapters were found for 'python collect_data.py DoverDistrictCouncil https://collections.dover.gov.uk/property/100060906311'

[robbrad]

@davelegg83

[davelegg83]

Thanks @robbrad - Got it working!
For the 'Provide council details' section, I was using the linked wiki guide

Works a treat now I just added the direct URL!

[jezza10d]

I seem to be facing a very similar issue with Neath Port Talbot.
The error seems to be in Home Assistant where it is not setting up the sensors as the start of this thread but when i do some digging it does not seem to be fetching any results during the scrape. I am not convinced the endpoint for NPT is quite right or if the data required is address specific when trying each of the endpoints.

I am running via HACS in Home Assistant and using standalone-chrome within a docker container. I can see the session in standalone chrome but it appears to be blank.

Home Assistant Error

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 315, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/uk_bin_collection/sensor.py", line 133, in _async_update_data
    data = await self.hass.async_add_executor_job(self.ukbcd.run)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 102, in run
    return self.client_code(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 121, in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 80, in template_method
    bin_data_dict = self.parse_data(
                    ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/councils/NeathPortTalbotCouncil.py", line 34, in parse_data
    driver.get("https://www.npt.gov.uk/2195")
  File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 356, in get
    self.execute(Command.GET, {"url": url})
  File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 347, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.12/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: net::ERR_NAME_NOT_RESOLVED
  (Session info: chrome-headless-shell=125.0.6422.76)
Stacktrace:
#0 0x555d32c29e9a <unknown>
#1 0x555d3291345c <unknown>
#2 0x555d3290afd0 <unknown>
#3 0x555d328fab32 <unknown>
#4 0x555d328fc162 <unknown>
#5 0x555d328faded <unknown>
#6 0x555d328fa503 <unknown>
#7 0x555d328fa3f5 <unknown>
#8 0x555d328f8336 <unknown>
#9 0x555d328f883a <unknown>
#10 0x555d32915ea7 <unknown>
#11 0x555d329a1ed5 <unknown>
#12 0x555d329824b2 <unknown>
#13 0x555d329a12cc <unknown>
#14 0x555d32982253 <unknown>
#15 0x555d329521c7 <unknown>
#16 0x555d32952b3e <unknown>
#17 0x555d32bf02db <unknown>
#18 0x555d32bf4387 <unknown>
#19 0x555d32bdce0e <unknown>
#20 0x555d32bf4e52 <unknown>
#21 0x555d32bc17af <unknown>
#22 0x555d32c19188 <unknown>
#23 0x555d32c1935b <unknown>
#24 0x555d32c28fcc <unknown>
#25 0x7f1c4a364ac3 <unknown>


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 356, in _async_setup_platform
    await asyncio.shield(awaitable)
  File "/config/custom_components/uk_bin_collection/sensor.py", line 75, in async_setup_entry
    await coordinator.async_config_entry_first_refresh()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 287, in async_config_entry_first_refresh
    raise ex
homeassistant.exceptions.ConfigEntryNotReady: Message: unknown error: net::ERR_NAME_NOT_RESOLVED
  (Session info: chrome-headless-shell=125.0.6422.76)
Stacktrace:
#0 0x555d32c29e9a <unknown>
#1 0x555d3291345c <unknown>
#2 0x555d3290afd0 <unknown>
#3 0x555d328fab32 <unknown>
#4 0x555d328fc162 <unknown>
#5 0x555d328faded <unknown>
#6 0x555d328fa503 <unknown>
#7 0x555d328fa3f5 <unknown>
#8 0x555d328f8336 <unknown>
#9 0x555d328f883a <unknown>
#10 0x555d32915ea7 <unknown>
#11 0x555d329a1ed5 <unknown>
#12 0x555d329824b2 <unknown>
#13 0x555d329a12cc <unknown>
#14 0x555d32982253 <unknown>
#15 0x555d329521c7 <unknown>
#16 0x555d32952b3e <unknown>
#17 0x555d32bf02db <unknown>
#18 0x555d32bf4387 <unknown>
#19 0x555d32bdce0e <unknown>
#20 0x555d32bf4e52 <unknown>
#21 0x555d32bc17af <unknown>
#22 0x555d32c19188 <unknown>
#23 0x555d32c1935b <unknown>
#24 0x555d32c28fcc <unknown>
#25 0x7f1c4a364ac3 <unknown>`

Any help in getting this working would be greatly appreciated.

[robbrad]

@jezza10d This isnt related to the problem @davelegg83 was facing

NeathPortTalbot have changed their website and the scraper needs to be updated.

Please raise an issue for the broken council scraper (NeathPortTalbot)