Parallel Processing in python

user_341683 0
Viewed 29747

Whats a simple code that does parallel processing in python 2.7? All the examples Ive found online are convoluted and include unnecessary codes.

how would i do a simple brute force integer factoring program where I can factor 1 integer on each core (4)? my real program probably only needs 2 cores, and need to share information.

I know that parallel-python and other libraries exist, but i want to keep the number of libraries used to a minimum, thus I want to use the thread and/or multiprocessing libraries, since they come with python

parallel-processing python
4 Answers

This can be done elegantly with Ray, a system that allows you to easily parallelize and distribute your Python code.

To parallelize your example, you'd need to define your map function with the @ray.remote decorator, and then invoke it with .remote. This will ensure that every instance of the remote function will executed in a different process.

import ray

ray.init()

# Define the function to compute the factors of a number as a remote function.
# This will make sure that a call to this function will run it in a different
# process.
@ray.remote
def compute_factors(x):
    factors = [] 
    for i in range(1, x + 1):
       if x % i == 0:
           factors.append(i)
    return factors    

# List of inputs.
inputs = [67, 24, 18, 312]

# Call a copy of compute_factors() on each element in inputs.
# Each copy will be executed in a separate process.
# Note that a remote function returns a future, i.e., an
# identifier of the result, rather that the result itself.
# This enables the calls to remote function to not be blocking,
# which enables us to call many remote function in parallel. 
result_ids = [compute_factors.remote(x) for x in inputs]

# Now get the results
results = ray.get(result_ids)

# Print the results.
for i in range(len(inputs)):
    print("The factors of", inputs[i], "are", results[i])

There are a number of advantages of using Ray over the multiprocessing module. In particular, the same code will run on a single machine as well as on a cluster of machines. For more advantages of Ray see this related post.

user_109391430
Related