I have a dataset on property. It has rental values and deposit amount, number of bedrooms, area, etc. At least 1/3rd of the rental column values are just Zero. There is no value in it. I have to perform clustering.
However, the rent values are highly skewed. Can I ignore 1/3rd of the rows while performing clustering or should I impute values? What is the right method to impute values.
It depends on the aim of clustering. You could ignore the data (ie, delete the column) and proceed with clustering. You will have clusters based on the remaining features like size, number of rooms, etc.
If rent amount is an important feature that can distinguish a property from another, then you should include the column, but remove the rows that have zero (or Nan) values in that column. To impute with mean value is bad, because a 10 bedroom apartment will have different rent value from a 1 bedroom apartment, then you would have added much noise to the data.
What I would do is a few steps:
(1) extract the rows with zero rent value and use them as "test dataset".
(2) use the remaining data to train a regression model to predict the rent value, ie, do the usual train_test_split for train-val-test to get the best performing model.
(3) apply the selected model on the "test dataset" to fill in the rent values
(4) combine the 2 datasets, but do this (I'd explain later): In the "test dataset" with predicted rent values, add a column called "recognise" and give a constant value, say 1001. In the dataset with real rent values, add a column called "recognise" and give a constant value, say 1000. Now you have a full dataset with full rent values to do clustering!
Now let me explain the column "recognise". This column will have little influence on the clustering, because 1000 is close to 1001. But this column can let you recognise which record has real rent values (1000) and which has predicted rent values (1001), for analysis later if needed.
