unexpectly import my local module when it is not necessary

[link89]

In file ai2_kit/domain.py

def fun(ctx):
    def in_add(a, b):
        print (a+b)

    ctx.executor.run_python_fn(in_add)(1, 2)   # this pass
    ctx.executor.run_python_fn(out_add)(1, 2)  # this failed, the error is: ModuleNotFoundError: No module named 'ai2_kit'

def out_add(a, b):
    print(a+b)

the method run_python_fn is defined in ai2_kit/executor.py , the basic idea is use python -c to execute a python script on remote machine.

    def run_python_script(self, script: str):
        return self.connector.run('python -c {}'.format(shlex.quote(script)))

    def run_python_fn(self, fn: T, python_cmd=None) -> T:
        def remote_fn(*args, **kwargs):
            dumped_fn = base64.b64encode(cloudpickle.dumps(lambda: fn(*args, **kwargs), protocol=pickle.DEFAULT_PROTOCOL)) 
            script = '''import base64,pickle; pickle.loads(base64.b64decode({}))()'''.format(repr(dumped_fn))
            self.run_python_script(script=script, python_cmd=python_cmd)

I have no idea why it will import ai2_kit when use a function outside of current function, the method out_add doesn't have any external dependencies. Is there any method to workaround this problem? Thank you! Both local and remote python is v3.9.

The error pickle (base64) for debug:

gASVFAMAAAAAAACMF2Nsb3VkcGlja2xlLmNsb3VkcGlja2xllIwNX2J1aWx0aW5fdHlwZZSTlIwKTGFtYmRhVHlwZZSFlFKUKGgCjAhDb2RlVHlwZZSFlFKUKEsASwBLAEsASwRLE0MOiAGIAGkAiAKkAY4BUwCUToWUKSmMRS9ob21lL2hlbnJ5L3NyYy9jYy1wcm9qZWN0cy9wcm9qZWN0cy9haTIta2l0L2FpMl9raXQvY29yZS9leGVjdXRvci5weZSMCDxsYW1iZGE+lEuMQwCUjARhcmdzlIwCZm6UjAZrd2FyZ3OUh5QpdJRSlH2UKIwLX19wYWNrYWdlX1+UjAxhaTJfa2l0LmNvcmWUjAhfX25hbWVfX5SMFWFpMl9raXQuY29yZS5leGVjdXRvcpSMCF9fZmlsZV9flIxFL2hvbWUvaGVucnkvc3JjL2NjLXByb2plY3RzL3Byb2plY3RzL2FpMi1raXQvYWkyX2tpdC9jb3JlL2V4ZWN1dG9yLnB5lHVOTmgAjBBfbWFrZV9lbXB0eV9jZWxslJOUKVKUaBwpUpRoHClSlIeUdJRSlIwcY2xvdWRwaWNrbGUuY2xvdWRwaWNrbGVfZmFzdJSMEl9mdW5jdGlvbl9zZXRzdGF0ZZSTlGgifZR9lChoF2gMjAxfX3F1YWxuYW1lX1+UjD5IcGNFeGVjdXRvci5ydW5fcHl0aG9uX2ZuLjxsb2NhbHM+LnJlbW90ZV9mbi48bG9jYWxzPi48bGFtYmRhPpSMD19fYW5ub3RhdGlvbnNfX5R9lIwOX19rd2RlZmF1bHRzX1+UTowMX19kZWZhdWx0c19flE6MCl9fbW9kdWxlX1+UaBiMB19fZG9jX1+UTowLX19jbG9zdXJlX1+UaACMCl9tYWtlX2NlbGyUk5RLAUsChpSFlFKUaDKME2FpMl9raXQuZG9tYWluLmNwMmuUjANhZGSUk5SFlFKUaDJ9lIWUUpSHlIwXX2Nsb3VkcGlja2xlX3N1Ym1vZHVsZXOUXZSMC19fZ2xvYmFsc19flH2UdYaUhlIwLg==

[link89]

I figure out a work around after someone explain me the reason: https://stackoverflow.com/questions/75292769

But I think a better solution is for cloudpickle to provide a decorator to marked a function or class that should be serialized by value.

[ogrisel]

You can already do cloudpickle.register_pickle_by_value(my_module) if you wish. Feel free to open a PR if you want to implement a decorator to do that on a function by function basis instead of doing it at the module level.