Error while passing a list of strings in multiprocessing pool.starmap

user_8141330 0
Viewed 2011

I am trying some webscraping. I am passing a list containing URLs to pool.starmap , and I am facing arguments error. Showing a simplified version of my code:

Can somebody help me with this? And sorry if I am doing something silly.


from multiprocessing.dummy import Pool

def func(x):
    print(x)

s = ["cat","foo","bar","you","and","me"] #this list contains ~50 URLs in actual code

with Pool() as pool:
    pool.starmap(func,s)

This gives me error:

Traceback (most recent call last):                                                                                                          
File "g.py", line 8, in <module>                                                                                                            
  pool.starmap(func,s)                                                                                                                    
File "C:\Users\Gunjan\Anaconda3\lib\multiprocessing\pool.py", line 274, in starmap                                                          return 
  self._map_async(func, iterable, starmapstar, chunksize).get()                                                                    
File "C:\Users\Gunjan\Anaconda3\lib\multiprocessing\pool.py", line 644, in get                                                              
  raise self._value                                                                                                                       
File "C:\Users\Gunjan\Anaconda3\lib\multiprocessing\pool.py", line 119, in worker
  result = (True, func(*args, **kwds))                                                                                                    
File "C:\Users\Gunjan\Anaconda3\lib\multiprocessing\pool.py", line 47, in starmapstar
  return list(itertools.starmap(args[0], args[1]))                                                                                      
TypeError: func() takes 1 positional argument but 3 were given
python-3.x python
2 Answers

starmap expects list of iterables. Check here. So use map instead.

user_114508730

you need to use the regular map()

try:

from multiprocessing.dummy import Pool

def func(x):
    print(x)

s = ["cat","foo","bar","you","and","me"] #this list contains ~50 URLs in actual code

with Pool() as pool:
    pool.map(func,s)

starmap() expects each element of the list to be an iterable itself, and it passes the inner iterable's element args to func

your elements are strings, which are iterables by character, so starmap called func('c', 'a', 't') for the first element, etc...

here's the definitions + docstrings of both functions:

    def map(self, func, iterable, chunksize=None):
        '''
        Apply `func` to each element in `iterable`, collecting the results
        in a list that is returned.
        '''
        return self._map_async(func, iterable, mapstar, chunksize).get()

    def starmap(self, func, iterable, chunksize=None):
        '''
        Like `map()` method but the elements of the `iterable` are expected to
        be iterables as well and will be unpacked as arguments. Hence
        `func` and (a, b) becomes func(a, b).
        '''
        return self._map_async(func, iterable, starmapstar, chunksize).get()
user_50523650
Related