Python: Transfer A Class Method To Another Computer

I have created an class that is used for analysising a specific type of data that I produce. I use this class on a local computer but occasionally there is too much data to work lo

Solution 1:

I'm the dill author. dill is able to pass a class method to another computer, as seen below.

>$ python
Python 3.5.6 (default, Sep 20 2018, 12:15:10) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo(object):
...   def bar(self, x):
...     return self.y + x
...   def __init__(self, y):
...     self.y = y
... 
>>> import dill
>>>          
>>> f = Foo(5)
>>>                  
>>> with open('foo.pkl', 'wb') as pkl:
...   dill.dump(f.bar, pkl)
... 
>>>

Then in a new session (or on another computer)...

>$ python
Python 3.5.6 (default, Sep 20 2018, 12:15:10) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>> with open('foo.pkl', 'rb') as pkl:
...   b = dill.load(pkl)
... 
>>> b(4)
9

Without more specific code from you, it's hard to say why you aren't seeing this behavior... but dill does provide the ability to pass a class definition (or just a class method) to another computer.

This behavior is what enables code like pathos to pass the class method to another computer within a ParallelPool or a ProcessPool -- the latter is across processes, while the former can be across distributed resources.

dude>$ python
Python 3.5.6 (default, Sep 20 2018, 12:15:10) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> class Foo(object):
...   def bar(self, x):
...     return self.y + x
...   def __init__(self, y):
...     self.y = y
... 
>>> import pathos
>>> p = pathos.pools.ParallelPool()
>>> p.map(Foo(4).bar, [1,2,3])
[5, 6, 7]
>>> p.close(); p.join()
>>>