Compressing data with encoder part of an auto-encoder is giving inconstant classification results

Viewed 48

I have a dataset that contains 11 classes, I'm performing dimensionality reduction using the encoder part of an autoencoder from 115 features to 15, I have trained the autoencoder on the full dataset then saved the encoder as h5 file, after that, i imported the encoder to another project, I was supposed to compress the dataset all at ones, but i didn't have enough RAM to do that (csv file is to big), so i compressed the data belonging to each class on its own and saved the compressed dataset to csv files, so i ended up with 11 csv files each one representing compressed data belonging to one class, after that i imported the compressed csv files to my project and concatinated them into one dataframe, i then shuffled them and did the classification using MLP, I got really good results, after a while i managed to get more ram, so i did the dimensionality reduction part all over again, this time i shuffled the full dataset then compressed it with the encoder and did the classification again, this time the results were bad, i even tried to make the encoder non trainable (because i thoght maybe it's parameters are changing), but still i got the same bad results.

My question is: how come when I compress each attack on its own then concatenate the compressed data into one dataframe I get great results but when I compress the full dataset containing all the classes at once I get bad results with the same encoder.

1 Answers

RAMs have certain number of cores and that can work parallelly. But since you are using it to store the vast data in it it loses capacity and tends to decrease the latency wherever possible and thus it decreases the Ops that would rather have been used if ram was free. And thus the performance decreases.

Related