How read parquet files and only keep files that contain some columns

How can I create a dataframe by excluding the files that do not contain the column "year".

First off I would advise restructuring bucket to separate these files based on their schema, or better yet have a process which transforms these "raw" files into a common schema that would be more easy to work with.

Working with what you have, starting with some parquet files:

val df1 = List(
 ("a", "b", "c")
).toDF("name", "city", "years")

df1.write.parquet("s3://{bucket}/test/a.parquet")

val df2 = List(
 ("aa", "bb")
).toDF("name", "city")

df2.write.parquet("s3://{bucket}/test/b.parquet")

val df3 = List(
 ("aaa", "bbb", "ccc")
).toDF("name", "city", "year")

df3.write.parquet("s3://{bucket}/test/c.parquet")

We can do:

import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.functions.{col, input_file_name}

// determine the s3 paths of all the parquet files, to be read independantly
val s3Paths = spark
  .read
  .schema(schema)
  .parquet("s3://{bucket}/test/")
  .withColumn("filename", input_file_name())
  .select("filename")
  .collect()
  .map(_(0).toString)

// individual DataFrames of each parquet file, where the `year` is not present
val dfs = s3Paths.flatMap {
  path => 
    val df = spark.read.parquet(path)
    if (!df.columns.contains("year")) {
      List(df)
    } else {
      List.empty[DataFrame]
    }
}

// take the first DataFrame, and the rest
val (firstDFs, otherDFs) = (dfs.head, dfs.tail)

// combine all of the DataFrame, unioning the rows
otherDFs.foldLeft(firstDFs) {
  case (acc, df) => acc.unionByName(df, allowMissingColumns = true)
}.show()

Note:

In the above example, when creating the test data:

df1.write.parquet("s3://{bucket}/test/a.parquet")

will have created the a file in s3, for example:

s3://{bucket}/test/a.parquet/part-0000-blah.parquet

for the purpose of creating this example I moved the parquet files up a level into the test/ path.