I have 10M sample of 1000 factors. Each factor is stored in a separate file. Therefore there are 1000 files each includes 1M double. If A is the data matrix with d = 1000 rows and n=1M columns,
I need to compute C = A*transpose(A), which is a d * d matrix.
I don’t know an efficient way using pySpark and google cloud(the data is stored there) to compute C.