spark read csv behaviour with files containing in one case a header and no header for rest

Viewed 127

airlines is one of a set of public datasets from Databricks.

Here, part-00000 has a header, the others do not.

This:

val paths = Seq(
   "/databricks-datasets/airlines/part-00000"  
   ,"/databricks-datasets/airlines/part-00001"
   ,"/databricks-datasets/airlines/part-00011"
   ,"/databricks-datasets/airlines/part-00071"
   ,"/databricks-datasets/airlines/part-00084"
   ,"/databricks-datasets/airlines/part-00101"
   ,"/databricks-datasets/airlines/part-00105"
   ,"/databricks-datasets/airlines/part-00178")   

val df = 
   spark.read.format("csv")
  .option("sep", ",")
  .option("inferSchema", "true")
  .option("header", "true")
  .load(paths: _*)

df.show(10, false)

returns for schema:

1996:string
8:string
24:string
6:string
1739:string
...

However:

val paths = Seq(
   "/databricks-datasets/airlines/part-00000"  
   ,"/databricks-datasets/airlines/part-00001"
   ,"/databricks-datasets/airlines/part-00011"
   ,"/databricks-datasets/airlines/part-00071"
   ,"/databricks-datasets/airlines/part-00084"
   ,"/databricks-datasets/airlines/part-00101"
   ,"/databricks-datasets/airlines/part-00105")
...

returns:

Year:integer
Month:integer
DayofMonth:integer
DayOfWeek:integer
...

What explains this difference?

Ran also with 1.0 SamplingRatio.

Seems random, noticed that a file with more columns has preference. If same number of columns, then seems random what will be chosen as headings.

0 Answers
Related