I am trying to convert an xarray Dataset to a Pandas dataframe. It used to take minutes, but after an xarray library update, it takes hours.
I read in a list of 40 large netcdf datasets (each dataset is 1GB, totaling 40GB) using the command:
with xr.open_mfdataset(infile_list, combine='by_coords') as ds:
The data has lon, lat, day, crs dimensions. I select a single lat and lon slice using the command:
station = ds.sel(lon=-84.7250, lat=42.3583, method='nearest')
I try to convert this slice to a pandas dataframe. The output dataframe I expect would have 8 data variables.
df = station.to_dataframe()
I am able to convert a single data variable from the slice to a Pandas series in 1.5 minutes using the command:
df = station["wind_speed"].to_series()
Do I need to change how I read in the large datasets? Or is there a workaround to get to a pandas dataframe faster?