Python 3.9 broke this component #3

leranp · 2021-07-22T10:49:02Z

I upgraded HA docker couple of weeks ago so and the python version was changed to 3.9.5 and now there is a problem with the date conversation.
The camera is working, but the sensors aren't working at all.
Looks like there is a fix for it in the zeep github #mvantellingen/python-zeep#1240

[zeeep.xsd.elements.attribute] Error during xml -> python translation
Traceback (most recent call last):
File "/config/custom_components/onvif/zeeep/xsd/elements/attribute.py", line 22, in parse
return self.type.pythonvalue(value)
File "/config/custom_components/onvif/zeeep/xsd/types/builtins.py", line 44, in _wrapper
return func(self, re.sub(r"[\n\r\t ]", " ", value).strip())
File "/config/custom_components/onvif/zeeep/xsd/types/builtins.py", line 176, in pythonvalue
return isodate.parse_datetime(value)
File "/usr/local/lib/python3.9/site-packages/isodate/isodatetime.py", line 55, in parse_datetime
tmpdate = parse_date(datestring)
File "/usr/local/lib/python3.9/site-packages/isodate/isodates.py", line 201, in parse_date
return date(sign * int(groups['year']),
ValueError: month must be in 1..12

The text was updated successfully, but these errors were encountered:

slydiman · 2021-07-22T17:16:10Z

I have found the same error on some my HA instances.
But I cannot reproduce this issue on my PC with python 3.9.5.
I tried Home Assistant core-2021.6.2 and core-2021.7.4
What camera causes such issue?
Please be sure the camera is set to the correct datetime.
If the camera uses NTP service to update the time be sure NTP is available and reports the correct datetime.

You can try to change onvif/zeeep/xsd/types/builtins.py , line 176 to something like this

        try:
            return isodate.parse_datetime(value)
        except (ValueError):
            print("!!! isodate.parse_datetime('" + value + "') failed !!!")
            return None

to figure out what string cannot be parsed.
Then we need to understand what is the source of that invalid datetime string and how to fix it.

leranp · 2021-07-22T18:01:34Z

I figure how to fix it ,
removing this lines in the file onvif/zeeep/xsd/types/builtins.py :

    @treat_whitespace("collapse")
    def pythonvalue(self, value):

        # Determine based on the length of the value if it only contains a date
        # lazy hack ;-)
        # if len(value) == 10:
            # value += "T00:00:00"
        # elif (len(value) == 19 or len(value) == 26) and value[10] == " ":
            # value = "T".join(value.split(" "))
        return isodate.parse_datetime(value)

and add to onvif/zeeep/exceptions.py this lines:

class DTDForbidden(Error):
    def __init__(self, name, sysid, pubid):
        super(DTDForbidden, self).__init__()
        self.name = name
        self.sysid = sysid
        self.pubid = pubid

    def __str__(self):
        tpl = "DTDForbidden(name='{}', system_id={!r}, public_id={!r})"
        return tpl.format(self.name, self.sysid, self.pubid)


class EntitiesForbidden(Error):
    def __init__(self, name, content):
        super(EntitiesForbidden, self).__init__()
        self.name = name
        self.content = content

    def __str__(self):
        tpl = "EntitiesForbidden(name='{}', content={!r})"
        return tpl.format(self.name, self.content)

leranp · 2021-07-22T18:14:51Z

the error is still there , but the binary sensor is working again , maybe is because I deleted all the cache folders (pycache) ?

leranp · 2021-07-22T19:28:28Z

this is anther error :

2021-07-22 21:32:43 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/config/custom_components/onvif/event.py", line 171, in async_pull_messages
dt_util.as_utc(response.TerminationTime) - dt_util.utcnow()
File "/usr/src/homeassistant/homeassistant/util/dt.py", line 74, in as_utc
if dattim.tzinfo == UTC:
AttributeError: 'NoneType' object has no attribute 'tzinfo'

slydiman · 2021-07-22T21:13:42Z

I don't see the purpose to add classes DTDForbidden and EntitiesForbidden to onvif/zeeep/exceptions.py. Where and how are they used?

the error is still there , but the binary sensor is working again , maybe is because I deleted all the cache folders (pycache) ?

No. It is necessary to log the invalid datetime string to figure out what is wrong.

this is another error :

It means that response.TerminationTime is not a datetime at all.
You can add the following lines

                if not isinstance(response.TerminationTime, dt.datetime):
                    raise Fault(message="Invalid TerminationTime '%s'" % response.TerminationTime)

to /config/custom_components/onvif/event.py, line 170:

                # Renew subscription if less than two hours is left
                if not isinstance(response.TerminationTime, dt.datetime):
                    raise Fault(message="Invalid TerminationTime '%s'" % response.TerminationTime)
                if (
                    dt_util.as_utc(response.TerminationTime) - dt_util.utcnow()
                ).total_seconds() < 7200:
                    await self.async_renew()

to get detailed logs.

leranp · 2021-07-23T20:33:31Z

I don't see the purpose to add classes DTDForbidden and EntitiesForbidden to onvif/zeeep/exceptions.py. Where and how are they used?

the error is still there , but the binary sensor is working again , maybe is because I deleted all the cache folders (pycache) ?

No. It is necessary to log the invalid datetime string to figure out what is wrong.

this is another error :

It means that response.TerminationTime is not a datetime at all.
You can add the following lines
                if not isinstance(response.TerminationTime, dt.datetime):
                    raise Fault(message="Invalid TerminationTime '%s'" % response.TerminationTime)
to /config/custom_components/onvif/event.py, line 170:
                # Renew subscription if less than two hours is left
                if not isinstance(response.TerminationTime, dt.datetime):
                    raise Fault(message="Invalid TerminationTime '%s'" % response.TerminationTime)
                if (
                    dt_util.as_utc(response.TerminationTime) - dt_util.utcnow()
                ).total_seconds() < 7200:
                    await self.async_renew()
to get detailed logs.

This is the error log after the change:

2021-07-23 21:49:05 DEBUG (MainThread) [custom_components.onvif] Restarted ONVIF PullPoint subscription for '00:12:41:ED:50:80'
2021-07-23 23:05:55 WARNING (MainThread) [custom_components.onvif] Failed to fetch ONVIF PullPoint subscription messages for '00:12:41:ED:50:80': Invalid TerminationTime 'None'
2021-07-23 23:05:56 DEBUG (MainThread) [custom_components.onvif] Restarted ONVIF PullPoint subscription for '00:12:41:ED:50:80'

slydiman · 2021-07-23T22:58:20Z

This is the error log after the change:

2021-07-23 21:49:05 DEBUG (MainThread) [custom_components.onvif] Restarted ONVIF PullPoint subscription for '00:12:41:ED:50:80'
2021-07-23 23:05:55 WARNING (MainThread) [custom_components.onvif] Failed to fetch ONVIF PullPoint subscription messages for '00:12:41:ED:50:80': Invalid TerminationTime 'None'
2021-07-23 23:05:56 DEBUG (MainThread) [custom_components.onvif] Restarted ONVIF PullPoint subscription for '00:12:41:ED:50

So HA sent ONVIF PullPoint subscription request to the camera and got the response without TerminationTime at all.
It means that this camera is not up to ONVIF standards.
BTW, I asked the camera model.
You can install HA on Windows PC and use WireShark to record the communication between HA and the camera. Then we can see the real PullPoint subscription request and response and decide whether it is possible to fix it and how.

leranp · 2021-07-24T03:52:38Z

I didn't change anything (firmware version or configuration) in the last week's, only HA update.
It was working fine for couples of months so the only thing that can broke this is the HA update.
My Camera is Chinese brand

https://a.aliexpress.com/_vntz4D

slydiman · 2021-07-24T10:23:09Z

I think it does not depend on python version at all.

This camera is XM (Xiong Mai tech).
You can use XMEye and/or iCSee software for Android. I have no idea about iOS.
Note iCSee may update the camera firmware!
Note a lot of camera settings is unavailable in the mobile software.
It is possible to check/adjust all camera or DVR/NVR/HVR settings using CMS software.
You can download the XM CMS software for Windows here http://www.xiongmaitech.com/en/index.php/service/down_detail/83/198

You can also test the camera using ONVIF Device Manager https://sourceforge.net/projects/onvifdm/

Note about the model and firmware:
For example
I'm using this camera https://aliexpress.ru/item/1005001446624233.html
The model on the box is A5-F30RJ.
But the correct model is XM530_RF50X30_8M.
The version of the camera shown by the software is V5.00.R02.00030746.10010.348517.ONVIF 16.12
00030746 here is the model ID and the latest firmware for that model is available here https://download.xm030.cn/d/MDAwMDEzMjk=

Here is the list of some XM camera models and firmwares (Russian lang) https://www.cctvsp.ru/articles/obnovlenie-proshivok-dlya-ip-kamer-ot-xiong-mai

leranp · 2021-07-24T10:44:38Z

Thank you for the details, i already using the iCSee App, but i blocked the internet access to the camera, so i can use it only when connected to the local network. In the beginning i used the CMS software to configure the camera. My model number is R80X30 . Actually it look like the lines that you told my to add was fixed the problem, i think that sometimes the date value from the camera is wrong but this time it only give an warning and start the subscribe again. So it is good for now. I will check if the camera is configure to use my NTP server or maybe the pi-hole is making troubles

…

On Sat, Jul 24, 2021, 13:23 Dmitry Vasilyev ***@***.***> wrote: I think it does not depend on python version at all. This camera is XM (Xiong Mai tech). You can use XMEye and/or iCSee software for Android. I have no idea about iOS. *Note iCSee may update the camera firmware!* Note a lot of camera settings is unavailable in the mobile software. It is possible to check/adjust all camera or DVR/NVR/HVR settings using CMS software. You can download the XM CMS software for Windows here http://www.xiongmaitech.com/en/index.php/service/down_detail/83/198 You can also test the camera using ONVIF Device Manager https://sourceforge.net/projects/onvifdm/ Note about the model and firmware: For example I'm using this camera https://aliexpress.ru/item/1005001446624233.html The model on the box is A5-F30RJ. But the correct model is XM530_RF50X30_8M. The version of the camera shown by the software is V5.00.R02.00030746.10010.348517.ONVIF 16.12 00030746 here is the model ID and the latest firmware for that model is available here https://download.xm030.cn/d/MDAwMDEzMjk= Here is the list of some XM camera models and firmwares (Russian lang) https://www.cctvsp.ru/articles/obnovlenie-proshivok-dlya-ip-kamer-ot-xiong-mai — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACIDWH7J5AECB4G3TGRGE2TTZKIBTANCNFSM5AZX4YBQ> .

leranp · 2021-07-26T06:21:11Z

Update: Disable NTP didn't fixed the problem

leranp closed this as completed Jul 22, 2021

leranp reopened this Jul 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 3.9 broke this component #3

Python 3.9 broke this component #3

leranp commented Jul 22, 2021 •

edited

Loading

slydiman commented Jul 22, 2021

leranp commented Jul 22, 2021

leranp commented Jul 22, 2021

leranp commented Jul 22, 2021

slydiman commented Jul 22, 2021 •

edited

Loading

leranp commented Jul 23, 2021

slydiman commented Jul 23, 2021 •

edited

Loading

leranp commented Jul 24, 2021 •

edited

Loading

slydiman commented Jul 24, 2021

leranp commented Jul 24, 2021 via email

leranp commented Jul 26, 2021

Python 3.9 broke this component #3

Python 3.9 broke this component #3

Comments

leranp commented Jul 22, 2021 • edited Loading

slydiman commented Jul 22, 2021

leranp commented Jul 22, 2021

leranp commented Jul 22, 2021

leranp commented Jul 22, 2021

slydiman commented Jul 22, 2021 • edited Loading

leranp commented Jul 23, 2021

slydiman commented Jul 23, 2021 • edited Loading

leranp commented Jul 24, 2021 • edited Loading

slydiman commented Jul 24, 2021

leranp commented Jul 24, 2021 via email

leranp commented Jul 26, 2021

leranp commented Jul 22, 2021 •

edited

Loading

slydiman commented Jul 22, 2021 •

edited

Loading

slydiman commented Jul 23, 2021 •

edited

Loading

leranp commented Jul 24, 2021 •

edited

Loading