airlines is one of a set of public datasets from Databricks.
Here, part-00000 has a header, the others do not.
This:
val paths = Seq(
"/databricks-datasets/airlines/part-00000"
,"/databricks-datasets/airlines/part-00001"
,"/databricks-datasets/airlines/part-00011"
,"/databricks-datasets/airlines/part-00071"
,"/databricks-datasets/airlines/part-00084"
,"/databricks-datasets/airlines/part-00101"
,"/databricks-datasets/airlines/part-00105"
,"/databricks-datasets/airlines/part-00178")
val df =
spark.read.format("csv")
.option("sep", ",")
.option("inferSchema", "true")
.option("header", "true")
.load(paths: _*)
df.show(10, false)
returns for schema:
1996:string
8:string
24:string
6:string
1739:string
...
However:
val paths = Seq(
"/databricks-datasets/airlines/part-00000"
,"/databricks-datasets/airlines/part-00001"
,"/databricks-datasets/airlines/part-00011"
,"/databricks-datasets/airlines/part-00071"
,"/databricks-datasets/airlines/part-00084"
,"/databricks-datasets/airlines/part-00101"
,"/databricks-datasets/airlines/part-00105")
...
returns:
Year:integer
Month:integer
DayofMonth:integer
DayOfWeek:integer
...
What explains this difference?
Ran also with 1.0 SamplingRatio.
Seems random, noticed that a file with more columns has preference. If same number of columns, then seems random what will be chosen as headings.