I am trying to load the "California Housing" dataset into a pandas dataframe directly from the source URL. The URL points to a tgz-file that contains two files, cal_housing.data and cal_housing.domain.
Loading the file with pandas read_csv is working fine but it makes one error that I don't understand and want to get rid off: The first value of the dataframe (first row, first column) is replaced by the filename.
This is what cal_housing.data looks like:
0 -122.230000,37.880000,41.000000,880.000000,129.000000,322.000000,126.000000,8.325200,452600.000000
1 -122.220000,37.860000,21.000000,7099.000000,1106.000000,2401.000000,1138.000000,8.301400,358500.000000
2 -122.240000,37.850000,52.000000,1467.000000,190.000000,496.000000,177.000000,7.257400,352100.000000
3 ...
This is what cal_housing.domain looks like:
0 longitude: continuous.
1 latitude: continuous.
2 housingMedianAge: continuous.
3 totalRooms: continuous.
4 totalBedrooms: continuous.
5 population: continuous.
6 households: continuous.
7 medianIncome: continuous.
8 medianHouseValue: continuous.
This is what I do:
import pandas as pd
source = 'http://www.dcc.fc.up.pt/~ltorgo/Regression/cal_housing.tgz'
col_names = ['longitude', 'latitude', 'housingMedianAge', 'totalRooms', 'totalBedrooms', 'population', 'households', 'medianIncome', 'medianHouseValue']
data = pd.read_csv(source, compression='gzip', header=None, names=col_names).dropna()
print(type(data))
This is what I get:
0 CaliforniaHousing/cal_housing.data 37.88 41.0 ...
1 -122.220000 37.86 21.0 ...
2 -122.240000 37.85 52.0 ...
...
And finally, this is what I want to get:
0 -122.230000 37.88 41.0 ...
1 -122.220000 37.86 21.0 ...
2 -122.240000 37.85 52.0 ...
...