I have various large files whose structure can be exemplified as "tmp.txt".
1 3.1
2 2.4
3 7.3
4 7.2
1 3.2
2 2.0
3 3.1
4 5.2
1 3.1
2 2.2
3 5.2
4 2.3
I want to read this data file and perform two tasks: i) For unique values of the first column, calculate the sum of the y column and save the data in a single array. ii) Read the file in such a way so that I can create a list of arrays whose elements are the blocks of data, separated by a blank line, in the file.
Using the following code, I can almost satisfy my request.
import numpy as np
def get_sumy(data):
result = {}
for e in data[0,:]:
result[e] = [i for i, j in enumerate(data[0,:]) if j == e]
l=len(np.unique(data[0,:]))
data_out=np.zeros((2,l),dtype=float)
id=0
for val in np.unique(data[0,:]):
tmp=0
for idx in result[val]:
tmp= tmp+data[1,idx]
data_out[0,id]=data[0,id]
data_out[1,id]=tmp
id=id+1
return data_out
def read_file(filename):
array_list = []
with open(filename) as f_data:
for k, g in groupby(f_data, lambda x: x.startswith('\n')):
if not k:
array_list.append(np.array([[float(x) for x in d.split()] for d in g if len(d.strip())]))
return array_list
filename0 ="./tmp.txt"
data= np.loadtxt(filename0).T
data_out=get_sumy(data)
data_list=read_file(filename0)
print(data_out)
print(data_list)
The outputs read
#data_out
[[ 1. 2. 3. 4. ]
[ 9.4 6.6 15.6 14.7]]
#data_list
[array([[1. , 3.1],
[2. , 2.4],
[3. , 7.3],
[4. , 7.2]]),
array([[1. , 3.2],
[2. , 2. ],
[3. , 3.1],
[4. , 5.2]]),
array([[1. , 3.1],
[2. , 2.2],
[3. , 5.2],
[4. , 2.3]])]
My question is now how to optimize these functions. In particular, for some reasons that I don't understand, my read_file sometimes cannot correctly identify a blank line, and it fails to separate arrays. Any suggestion?