Why does the processing time hardly decrease rather than being inversely proportional to the number of processes with multiprocessing.Pool?

I was testing multiprocessing with the following code which calculate the inverse of a matrix of size 100 by 100.

import numpy as np
import datetime
import multiprocessing

inputs = list(range(1000))

def invmat(I):
    np.random.seed(i)
    matrix1 = np.random.rand(100, 100)
    np.linalg.inv(matrix1)
    return 0

t1 = datetime.datetime.now()

pool = multiprocessing.Pool(processes = num_processes)
outputs = pool.map(invmat, inputs)

t2 = datetime.datetime.now()
print(f"{(t2 - t1).seconds} seconds")

I varied num_processes as 1, 2, 4, 8, 20, 40, and no argument with the command (pool = multiprocessing.Pool()).

I expected the time should be inversely proportional, but the result is quite weird.

The elapsed time was 0, 101, 111, 103, 88, 78, and 78 seconds, respectively.

Why does it not work at all when the number of processes is one, and why are the elapsed times virtually the same when the number of processes is 2, 4, and 8?

The OS is RHEL 7.9, the number of CPU cores is 40, the memory size is about 50GB, and the Python version is 3.9.6. When I run the command multiprocessing.cpu_count(), it returns the value of 40 as expected.