Using multiprocessing to find distances between points in array

The problem is you're not doing multiprocessing correctly. Specifically your code is missing the if __name__ == '__main__': guard. This is fixed in the code below which uses multiprocessing.Pool (which I think would be the best and easiest way to do what you want). It still takes a number of seconds to execute, but it doesn't overwhelm memory and the CPUs.

Information about needing the if __name__ == '__main__': is buried in the Safe importing of main module subsection of The spawn and forkserver start methods section of the multiprocessing module's documentation.

import math
import random
import multiprocessing as mp

screenSize = 1000000

pointsLength = 2000
weightLength = 20000

weightBuffer = screenSize/weightLength
pointBuffer = screenSize/pointsLength

points = []
weights = []
weightPoints = []

counter = 0

for i in range(pointsLength):
    for j in range(pointsLength):
        points.append([random.randint(j * pointBuffer, j * pointBuffer * 2),
        random.randint(i * pointBuffer, i * pointBuffer * 2)])


for i in range(pointsLength):
    for j in range(pointsLength):
        weightPoints.append([j * weightBuffer, i * weightBuffer])
        weights.append(0)


def FindDistance(i):
    row = math.floor((i / weightLength) / (weightLength / pointsLength))
    col = math.floor((i % weightLength) / (weightLength / pointsLength))
    points1d = (pointsLength * row) + col

    dist = math.dist(points[points1d], weightPoints[i])

    weights[i] = dist


if __name__ == '__main__':  # ADDED

    # With Multiprocessing Pool
    sumthing = []
    for i in range(len(weights)):
        sumthing.append(i)

    with mp.Pool(4) as p:
        p.map(FindDistance, sumthing)

You are iterating over the length of weights(2000 as per your code) and spawning a new process for every iteration, which means 2000 processes. No wonder the CPU and RAM are full.

What you need to do is chunk the weights array into 8 smaller arrays, ideally of equal length. Change the FindDistance function to take an array as a parameter. This parameter will be the smaller chunked array.

def FindDistance(i_arr):
    for i in i_arr:
        row = math.floor((i / weightLength) / (weightLength / pointsLength))
        col = math.floor((i % weightLength) / (weightLength / pointsLength))
        points1d = (pointsLength * row) + col

        dist = math.dist(points[points1d], weightPoints[i])

        weights[i] = dist

def chunking(weights):
    # initialise array of empty arrays with Length equal to number of processors
    smaller_chunks = [ [] i for i in range(8) ]
    for index,item in enumerate(weights):
        index_to_push = index % 8
        smaller_chunks[index_to_push].append(item)

    return smaller_chunks
processes = []
chunks = chunking(weights)
for i in range(len(chunks)):
    p = mp.Process(target=FindDistance, args=[i])
    p.start()
    processes.append(p)

for process in processes:
    process.join()

        

The problem lies in the fact that you are looping over the length of weights. which from your code

for i in range(pointsLength): #pointsLength is 2000
    for j in range(pointsLength):
        weightPoints.append([j * weightBuffer, i * weightBuffer])
        weights.append(0)

is 2000*2000 = 40,000

So you're trying to create 40,000 new processes all at once leading your system to crash. What you can do instead is break down your list of weights into n smaller arrays and use that to just create n new processes

We can split the array using the numpy function numpy.array_split. Now update the FindDistance function to accept the whole new array as an input.

def FindDistance(subarr):
    for i in subarr:
        row = math.floor((i / weightLength) / (weightLength / pointsLength))
        col = math.floor((i % weightLength) / (weightLength / pointsLength))
        points1d = (pointsLength * row) + col

        dist = math.dist(points[points1d], weightPoints[i])

        weights[i] = dist

and finally create n process with the new arguments

subarrays = np.array_split(weights, n)
processes = []
for subarr in subarrays:
    p = mp.Process(target=FindDistance, args=[subarr])
    p.start()
    processes.append(p)

for process in processes:
    process.join()