Feed for Harrow Council will be changing soon

[ratmash]

Name of Council

London Borough of Harrow

Issue Information

This is my personal GitHub account and I am not acting in any official capacity here. I am the developer of the script you are using to consume the data for Harrow Council. Just giving you a heads up we are currently working on a new version and chances are once it goes live the format of the data in the new version may change quite a bit.

EDIT: To clarify: I am the developer at Harrow Council who wrote the script you are using as the data source, not the code within this project that connects to it.

Verification

• I searched for similar issues at https://github.com/robbrad/UKBinCollectionData/issues?q=is:issue and found no duplicates
• I have checked my address/postcode/UPRN works on the council's website
• I have provided a detailed explanation of the issue as well as steps to replicate the issue
• I understand that this project is run by volunteer contributors therefore completion of this issue cannot be guaranteed

[robbrad]

Thank you @ratmash !!

What's the best way to address this ? Wait for the change and act?

Is there any api we could hit instead of scrapping etc?

Thanks Rob

[ratmash]

There is a limit to how much detail I can go into. There is no API, nor any plans for one. The script you are tapping into is doing A LOT more than you really need, with some significant processing required. However my PERSONAL opinion (I cannot speak for my colleagues/management) is that the goal of what you are trying to achieve with this home assistant integration aligns with what we are ultimately trying to achieve (getting people to put the right bins out on the right days, reducing missed bin complaints), I even wrote an Alexa skill for it at one point.

I was looking through the web server logs and noticed that someone using this is hitting it every 10 minutes. If this gets widespread adoption that will have a significant impact on our systems. I am trying to get ahead of my colleagues noticing and asking me to block it. I don't see how starting a 'cat and mouse' game will be productive or useful for anyone. If I have to spend time/effort on this one way or the other anyway, I think a better idea would be for me to work with you and give you something better suited to your use case.

I need to install this and look at the final output you are generating from your scraper. Then I can get a better idea of what I might be able to do for you.

Once again, these are my PERSONAL thoughts. My comments here are not endorsed by, or necessarily reflect the official line taken by my management or Harrow Council.

[ratmash]

I have installed this and looked at the output. I can see you are just extracting the binType and date portion of the eventTime attributes. This will work fine for the future events (where the eventType attribute is 'COLLECTION DUE'), and you will get away with it for the past events 99% of the time (when the crews have marked the bins as collected with no issues). However sometimes an event can get logged against a UPRN that will relate to a problem with the bin collection. For example, there is a 'Vehicle Breakdown' event logged against UPRN 100021303038 on 4th Dec 2024, indicating that the crew could not pick up the bin because the truck had broken down. Since the date on the event time is the same as the scheduled collection date of the bin you have got away with it, but there is no guarantee those will match.

I have spoken to the middleware developer, and he is OK with the idea of exposing the basic bin collection schedule, which will allow me to put together a stripped down endpoint that will give you the correct information, without doing lots of extra stuff you don't need. However, he has to get the OK from his manager. That is out of my hands.