Inconsistent TypeError: cannot serialize '_io.TextIOWrapper' object

user_3190076 0
Viewed 25779

I am working with Python 3.6.1 on Jupyter 5. My goal is to test how portalocker manage concurrent appending on the same file.

To accomplish that I have made a simple function that appends a single line to the same file and I use multiprocessing.Pool and Pool.map() to run the function in parallel.

Here is the code in Jupyter notebook.

cell 1

from time import time
from multiprocessing import Pool
import portalocker


def f(*args):
    while time() < start + 1:
        pass
    with open('portalocker_test.txt', 'a') as f:
        portalocker.lock(f, portalocker.LOCK_EX)
        f.write(f'{time()}\n')

cell 2

start = time()
with Pool(4) as p:
    p.map(f, range(4))

cell 3

with open('portalocker_test.txt', 'r') as f:
    for line in f:
        print(line, end='')

If I run this code once I get the expected result:

Out of cell 3:

1495614277.189394
1495614277.1893928
1495614277.1893911
1495614277.1894028

But if I run cell 2 again (without restarting the notebook) I get:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-db9c07d32724> in <module>()
      1 start = time()
      2 with Pool(4) as p:
----> 3     p.map(f, range(4))

/Users/xxx/Homebrew/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py in map(self, func, iterable, chunksize)
    258         in a list that is returned.
    259         '''
--> 260         return self._map_async(func, iterable, mapstar, chunksize).get()
    261 
    262     def starmap(self, func, iterable, chunksize=None):

/Users/xxx/Homebrew/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py in get(self, timeout)
    606             return self._value
    607         else:
--> 608             raise self._value
    609 
    610     def _set(self, i, obj):

/Users/xxx/Homebrew/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
    383                         break
    384                     try:
--> 385                         put(task)
    386                     except Exception as e:
    387                         job, ind = task[:2]

/Users/xxx/Homebrew/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py in send(self, obj)
    204         self._check_closed()
    205         self._check_writable()
--> 206         self._send_bytes(_ForkingPickler.dumps(obj))
    207 
    208     def recv_bytes(self, maxlength=None):

/Users/xxx/Homebrew/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/reduction.py in dumps(cls, obj, protocol)
     49     def dumps(cls, obj, protocol=None):
     50         buf = io.BytesIO()
---> 51         cls(buf, protocol).dump(obj)
     52         return buf.getbuffer()
     53 

TypeError: cannot serialize '_io.TextIOWrapper' object

The same error gets raised if I read the file before running cell 2. So, If I never open the file before running cell 2, all goes fine. If I open the file before, then I get that error. This is pretty inconsistent to me. What is going on? How to solve it?

Also, using or not portalocker will not change this behavior, so it is not portalocker the problem. I haven't check it on plain python but I am really interested in running it with Jupyter.

jupyter-notebook python-3.x
4 Answers

I had similar problem. In my case there was a conflict between multi-processing (as in: from multiprocessing import Process) and memory-mapped file (import mmap). Once the code relating to memory mapped file was removed, everything worked fine.

user_55113850

I had a similar problem too: in my case, a class was holding a reference (i.e. a member variable) to a (non-picklable) file handle. Once I cleared it after the work was done (i.e. set to None), everything wrt the multiprocessing worked fine.

The error in my case looked like this:

multiprocessing.pool.MaybeEncodingError: Error sending result: '[<backtest.StrategyBacktest object at 0x7efcd7eb6128>]'. Reason: 'TypeError("cannot serialize '_io.TextIOWrapper' object",)'
user_21146160

I had a similar problem too, the situation is a bit different in my case. I was using the Pool inside my class and the paralleling function is returning self, I removed it and everything works fine.

user_157253370
Related