so I'm having some issues understanding in which way I should store large files. For example, the block size in my HDFS is 128MB, and I have a 1GB file.
I know that saving files that are smaller than the block size is not the best practice and I understand why.
But what should I do with big files, for my 1GB file, should I save 1 file or 8 files of 128MB each, and why?
You can store 1 file with 1GB. Hadoop will autmatically store that file in 8 blocks. Hadoop is designed for bigger files not smaller files. Please note that Block is physical storage in hadoop.
As you did not mention split size in your cluster so i assume it is 128 MB. Split is something that on which you parallelism depend. So if you process 1 GB file on 128 split size 8 mappers will be invoked ( 1 mapper on each split). If you store 8 files of 128 mb each. There will be unneccesary overhead on your Namenode for maintaining info about those 8 files. In case of 8 files performance may be more or less similar as compared to 1 GB file but it will definitely better in case of 1 GB file with 8 blocks.
Do not confuse with Blocks in hadoop they are just storage unit like other file system. Hadoop will autmatically take care of storage no matter how bigger file is and it will divide files in block . Storing small files will be uncessary over head in i/o operations.
