How do I read a file without extension?

user_6470472 0
Viewed 757
  1. I have just download some file from server "hdfs dfs -get /user/hive/warehouse/"
  2. from the downloaded folder the file has no extension (I was expecting it is a parquet file) and when I use command "file *" to check it shows 000000_0 : data
  3. I use val rddFromFile = spark.sparkContext.textFile("path") and it shows that each line is not delimited or with "?" in between each field

May I know what file is it and how should I read it into a proper database/dataframe? I try split("?") but it just doesnt work

apache-spark scala
1 Answers

Use parquet-tools utility to view the data.

hadoop jar ./parquet-tools-<VERSION>.jar cat <hdfs-location-file>

Check this link for more about parquet tools. https://github.com/apache/parquet-mr/tree/master/parquet-tools

OR

simply do hdfs dfs -cat /user/hive/warehouse/000000_0

If it is parquet then structure is as below.

Parquet file:

At a high level, the parquet file consists of a header, one or more blocks and footer. The parquet file format contains a 4-byte magic number in the header (PAR1) and at the end of the footer. This is a magic number indicating that the file is in parquet format.

Use orcfiledump utility to view the content The ORC file dump utility comes with hive (0.11 or higher):

hive --orcfiledump <hdfs-location-of-orc-file>

ORC file:

ORC is a columnar file format. You can visualize the structure of an ORC file as an area that is divided into Header, body and footer.

Header Section: The Header contains the text ORC in case some tools are required to determine the type of file while processing.

Also, you can get files on windows, then you can just open files in the editor(notepad++). Although, it is a binary file. Still You can see in the first line like ORC or PAR1 followed by some other encrypted characters.  

user_22309610
Related