Would it be possible to run a code in python to read netcdf and not break?

Question

my name is Thiago.

I've downloaded ERA-5 weather data from 2015 to 2022 (hourly data) and I've wrote a Python script to read netcedf data and turn it into a "table" for further analysis.

I've used at least 500 points (latitude and longitude) from specific planting locations of the company where I work.

So I create a function to read and join the different years (2015 to 2022) on a spreadsheet for each location.

However, in the last part of the code when I want to generate the csv for each point (latitude and longitude) there is a break in the code due to the size of the files.

I would like someone to help me solve this problem, or any suggestions for code improvement for this doesn't happen anymore.

The code I've used is bellow:

Function to read netcdf data:

def teste(dataset, latitude, longitude):
dataset_subset = dataset.sel(longitude=longitude, latitude=latitude, method='nearest')
dataset_subset
datetime = dataset_subset['time'] 
u = dataset_subset['u10']          
v = dataset_subset['v10']
tmed = dataset_subset['t2m'] - 273.15
tmax = dataset_subset['mx2t'] - 273.15
tmin = dataset_subset['mn2t'] - 273.15
dew = dataset_subset['d2m'] - 273.15
pressure = dataset_subset['sp']/100
wet_bulb = tmed - (tmed - dew)/3
e = 6.1078*10**((7.5*wet_bulb)/(237.3+wet_bulb))-0.0008*pressure*(tmed-wet_bulb)
E = 6.1078*10**((7.5*tmed)/(237.3+tmed))
rh = e/E*100
vpd = E-e
wspd = ((u**2 + v**2)**(0.5))*3.6
prec = dataset_subset['tp']*1000
rad = dataset_subset['ssr']

data = {
'longitude': longitude,
'latitude': latitude,
'datetime' : datetime,
'tmed' : tmed,
'rh' : rh,
'vpd': vpd,
'pressure' : pressure,
'prec' : prec,
'ws' : wspd,
'rad': rad
}

dataframe = pd.DataFrame(data)
return dataframe

Lat and long points:

coordenadas = pd.read_excel('C:/Users/toliv16/Documents/Experimentos_analises/Adubacao_clones_sitio/Coordenadas_talhoes.xlsx', sheet_name='lat_long')

Read netcdf data for each lat and long point and append in a single dictionary:

dict_data_frame = {}



for (i, ano) in enumerate(anos):

dataset = xr.open_dataset(f'{OPEN_DIR}/{ano}.nc')

dataset.coords['longitude'] = ((dataset.coords['longitude'] + 180) % 360) - 180

dataset = dataset.sortby(dataset.longitude)

print(dataset['longitude'], dataset['latitude'])

for lat, long, talhao in zip(coordenadas['latitude'], coordenadas['longitude'], talhoes):

    dataframe = teste(dataset=dataset, latitude=lat, longitude=long)



    if i == 0:

        dict_data_frame[talhao] = [dataframe]

    else:

        dict_data_frame[talhao].append(dataframe)

Tranform in csv for further analysis:

final_dict_data_frame = {}

for talhao in talhoes:

final_dict_data_frame[talhao] = pd.concat(dict_data_frame[talhao])

final_dict_data_frame[talhao].to_csv(f'{SAVE_DIR}/{talhao}_clima.csv')

Error after this part:

MemoryError: Unable to allocate 8.54 MiB for an array with shape (7, 10000) and data type