BUG: Y2k38

[spirillen]

Description

As I was testing my little hobby zone rpz.mypdns.cloud (only ~600k records) I ran into the Y2k38

Configuration

Not relevant

Reproduction

Test the above zone and you'll see an error like this:

ybggb.com.cn                                                                                         ACTIVE      WHOIS      29-jun-2026       301        AVAILABILITY 
Process pyfunceble_tester_worker_34:
Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/cli/processes/workers/base.py", line 388, in run
    raise exception
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/cli/processes/workers/base.py", line 366, in run
    result = self.target(consumed)
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/cli/processes/workers/tester.py", line 275, in target
    self.testing_object.set_subject(test_dataset["idna_subject"])
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/checker/base.py", line 157, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/checker/base.py", line 182, in wrapper
    result = func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/checker/availability/domain_and_ip.py", line 109, in query_status
    result = query_object.query_status()
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/checker/base.py", line 157, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/checker/base.py", line 182, in wrapper
    result = func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/checker/availability/domain.py", line 169, in query_status
    self.try_to_query_status_from_http_status_code()
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/checker/availability/base.py", line 870, in try_to_query_status_from_http_status_code
    lookup_result = self.http_status_code_query_tool.get_status_code()
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/query/http_status_code.py", line 125, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/query/http_status_code.py", line 340, in get_status_code
    req = PyFunceble.factory.Requester.get(
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/query/requests/requester.py", line 157, in wrapper
    req = getattr(self.session, verb.lower())(*args, **kwargs)
  File "/$USER/.local/lib/python3.9/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/$USER/.local/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/$USER/.local/lib/python3.9/site-packages/requests/sessions.py", line 677, in send
    history = [resp for resp in gen]
  File "/$USER/.local/lib/python3.9/site-packages/requests/sessions.py", line 677, in <listcomp>
    history = [resp for resp in gen]
  File "/$USER/.local/lib/python3.9/site-packages/requests/sessions.py", line 237, in resolve_redirects
    resp = self.send(
  File "/$USER/.local/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/query/requests/adapter/https.py", line 92, in send
    hostname_ip = self.resolve(parsed_url.hostname)
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/query/requests/adapter/base.py", line 181, in resolve
    return self.resolve_without_cache(hostname)
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/query/requests/adapter/base.py", line 152, in resolve_without_cache
    if not IPSyntaxChecker(hostname).is_valid():
  File "/$USER/.local/lib/python3.9/site-packages/PyFunceble/checker/base.py", line 153, in wrapper
    raise TypeError(
TypeError: <self.subject> should be <class 'str'>, <class 'NoneType'> given.
zol.com.cn

AND

Process pyfunceble_producer_worker_1:
Traceback (most recent call last):
  File "/home/$USER/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
    self.dialect.do_execute(
  File "/home/$USER/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.DataError: (1264, "Out of range value for column 'epoch' at row 1")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/$USER/.local/lib/python3.9/site-packages/PyFunceble/cli/processes/workers/base.py", line 388, in run
    raise exception
  File "/home/$USER/.local/lib/python3.9/site-packages/PyFunceble/cli/processes/workers/base.py", line 366, in run
    result = self.target(consumed)
  File "/home/$USER/.local/lib/python3.9/site-packages/PyFunceble/cli/processes/workers/producer.py", line 393, in target
    self.run_whois_backup(test_result)
  File "/home/$USER/.local/lib/python3.9/site-packages/PyFunceble/cli/processes/workers/producer.py", line 210, in run_whois_backup
    self.whois_dataset.update(
  File "/home/$USER/.local/lib/python3.9/site-packages/PyFunceble/dataset/db_base.py", line 104, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/home/$USER/.local/lib/python3.9/site-packages/PyFunceble/dataset/whois/mariadb.py", line 149, in update
    super().update(row, ignore_if_exist=ignore_if_exist)
  File "/home/$USER/.local/lib/python3.9/site-packages/PyFunceble/dataset/db_base.py", line 104, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/home/$USER/.local/lib/python3.9/site-packages/PyFunceble/dataset/mariadb_base.py", line 160, in update
    self.add(row)
  File "/home/$USER/.local/lib/python3.9/site-packages/PyFunceble/dataset/db_base.py", line 104, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/home/$USER/.local/lib/python3.9/site-packages/PyFunceble/dataset/mariadb_base.py", line 106, in wrapper
    return func(self, *args, **kwargs)  # pylint: disable=not-callable
  File "/home/$USER/.local/lib/python3.9/site-packages/PyFunceble/dataset/mariadb_base.py", line 299, in add
    self.db_session.execute(self.ORM_OBJ.__table__.insert(), row)
  File "/home/$USER/.local/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1689, in execute
    result = conn._execute_20(statement, params or {}, execution_options)
  File "/home/$USER/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1611, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/home/$USER/.local/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/home/$USER/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1478, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/$USER/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1842, in _execute_context
    self._handle_dbapi_exception(
  File "/home/$USER/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2023, in _handle_dbapi_exception
    util.raise_(
  File "/home/$USER/.local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
    raise exception
  File "/home/$USER/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1799, in _execute_context
    self.dialect.do_execute(
  File "/home/$USER/.local/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 717, in do_execute
    cursor.execute(statement, parameters)
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/home/$USER/.local/lib/python3.9/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
sqlalchemy.exc.DataError: (pymysql.err.DataError) (1264, "Out of range value for column 'epoch' at row 1")
[SQL: INSERT INTO pyfunceble_whois_record (created_at, modified_at, subject, idna_subject, expiration_date, epoch) VALUES (%(created_at)s, %(modified_at)s, %(subject)s, %(idna_subject)s, %(expiration_date)s, %(epoch)s)]
[parameters: {'created_at': datetime.datetime(2021, 10, 15, 20, 52, 43, 573404), 'modified_at': datetime.datetime(2021, 10, 15, 20, 52, 43, 573408), 'subject': 'com.ge', 'idna_subject': 'com.ge', 'expiration_date': '02-jan-2099', 'epoch': '4070991600.0'}]
(Background on this error at: https://sqlalche.me/e/14/9h9h)

Expected behavior

No errors

Screenshots

Sorry, no

Versions

OS: Ubuntu 20

Python Version: 3.9.5

PyFunceble Version: 4.0.2b1.dev

Additional context

@funilrys wrote somewhere else:
This has nothing to do with PyFunceble (directly) sob

It's all because of the Y2k38 thing ...

Then you have two options as this issue will be more and more frequent day by day...

1. Rewrite dates that passes the 2,147,483,647 second to a default
2. Fin another way to store dates in the future.

---

Cause

The latest time since 1 January 1970 that can be stored using a signed 32-bit integer is 03:14:07 on Tuesday, 19 January 2038 (231−1 = 2,147,483,647 seconds after 1 January 1970).

[funilrys]

I'm guessing that vendors will fix this just before 2038 ... So until I can figure out another transport format, I could just automatically change the year by doing YEAR-(YEAR-2038+N) where N is implicitly 2038-N.

[spirillen]

I could just automatically change the year by doing YEAR-(YEAR-2038+N) where N is implicitly 2038-N.

It is something like that I have in mind

[spirillen]

And here it came again...

see #270 for how to redo the test..., let me be honest, it have now been running for 1.5 weeks and I can't even start it over, as the same error just will reappear

sqlalchemy.exc.DataError: (pymysql.err.DataError) (1264, "Out of range value for column 'epoch' at row 1")
[SQL: INSERT INTO pyfunceble_whois_record (created_at, modified_at, subject, idna_subject, expiration_date, epoch) VALUES (%(created_at)s, %(modified_at)s, %(subject)s, %(idna_subject)s, %(expiration_date)s, %(epoch)s)]
[parameters: {'created_at': datetime.datetime(2021, 12, 2, 6, 47, 43, 836904), 'modified_at': datetime.datetime(2021, 12, 2, 6, 47, 43, 836919), 'subject': 'edu.ge', 'idna_subject': 'edu.ge', 'expiration_date': '02-jan-2099', 'epoch': '4070991600.0'}]
(Background on this error at: https://sqlalche.me/e/14/9h9h)