V1Pod no longer pickleable with some logging configs

[jedcunningham]

What happened (please include outputs or screenshots):

Starting with 12.0.0, V1Pod now is no longer pickleable depending on how logging is configured.

What you expected to happen:

V1Pod is pickleable.

How to reproduce it (as minimally and precisely as possible):

>>> from kubernetes.client import models as k8s
>>> import logging
>>> import pickle
>>> logging.basicConfig()
>>> p = k8s.V1Pod()
>>> pickle.dumps(p)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't pickle _thread.RLock objects

Anything else we need to know?:

I believe it is unrelated, but this was initially found in apache/airflow, as it pickles V1Pods.

Environment:

• Python version: 3.6.12
• Python client version: 12.0.0

[ashb]

This behaviour came about when b4d11b0 introduced local_vars_configuration, which contains two logger objects.

[roycaihw]

/assign @palnabarun

[palnabarun]

It looks to me that the change in OpenAPI Generator brought in regression to older Python versions. I can reproduce the above error with Python 3.6.10 but not with Python 3.7.7 or Python 3.8.6. IMHO, it needs further investigation as to why this may be happening.

$ cat logger_pickle.py 
from kubernetes.client import models as k8s
from kubernetes import __version__

import pickle
import logging

logging.basicConfig()

print("Kubernetes Client version: {}".format(__version__))
print("Pickled V1Pod: {}".format(pickle.dumps(k8s.V1Pod())))

Python 3.6.10

(py3.6) $ ipython -i logging_pickle.py
Python 3.6.10 (default, May  7 2020, 12:54:46) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
Kubernetes Client version: 12.0.1
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/c/kubernetes-client/python/tmp/logger_pickle.py in <module>
      8 
      9 print("Kubernetes Client version: {}".format(__version__))
---> 10 print("Pickled V1Pod: {}".format(pickle.dumps(k8s.V1Pod())))
     11 

TypeError: can't pickle _thread.RLock objects

Python 3.7.7

(py3.7) $ ipython -i logger_pickle.py
Python 3.7.7 (default, May  8 2020, 11:39:39) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.
Kubernetes Client version: 12.0.1
Pickled V1Pod: b'\x80\x03ckubernetes.client.models.v1_pod\nV1Pod\nq\x00)\x81q\x.................'

Python 3.8.6

(py3.8) $ ipython -i logger_pickle.py
Python 3.8.6 (default, Sep 30 2020, 04:00:38) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.
Kubernetes Client version: 12.0.1
Pickled V1Pod: b'\x80\x04\x95\xa9\x03\x00\x00\x00\x00\x00\x00\x8c\x1fkubernetes.client.models.v1_pod\......'

@ashb

This behaviour came about when b4d11b0 introduced local_vars_configuration, which contains two logger objects.

I am not very sure if two logger objects are creating the issue since the bug does not surface in Python 3.7 and Python 3.8.

@jedcunningham @ashb -- is it a hard requirement to use Python 3.6 considering Python 3.6 went into security patch only part of it's lifecycle? In that case, the solution needs to be found in the OpenAPI Generator for this regression.

[ashb]

Afraid so - as long as 3.6 is not EoL Airflow supports it.

[mik-laj]

Python 3.6.8

$ python --version
Python 3.6.8
$ python -c "import logging; logging.basicConfig(); import pickle; pickle.dumps(logging.getLogger())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: can't pickle _thread.RLock objects

Python 3.7.2

python -c "import logging; logging.basicConfig(); import pickle; pickle.dumps(logging.getLogger())"

(no problem)

[ashb]

This could be addressed by adding __getstate__ (I think that's what it's called) to control what gets pickled or not.

I think not logging the logger objects make sense.

I'll try and find a time to attempt a pr for this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[ashb]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[ashb]

not stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[ashb]

/remove-lifecycle rotten

You abusive bot. Auto closing issues is terrible.

[palnabarun]

@ashb -- please give the bot some slack. :) It is trying to help to clear out inactive issues.

/lifecycle frozen

[palnabarun]

/unassign

(since I'm not working on this and @ashb mentioned that they will be looking at a patch)

[ashb]

I'm not quite sure how to fix this inside this module, particular as the class I might want to edit says "Do Not edit -- autogenreted".

The fix I have added to Airflow to "workaround" this is to backport the pickle support to Logger objects from 3.7 via the copyreg stdlib module. See apache/airflow#18797

[jedcunningham]

And 3.6 is now EOL.