The Scala DataFrameReader has a function "option" which has the following signature:
def option(key: String, value: String): DataFrameReader
// Adds an input option for the underlying data source.
So what is an "input option" for the underlying data source, can someone share an example here on how to use this function?
sparkSession.read.option("header",true).parquet(fileKey)
In above code snippet, Header=True means : keep column headers.
I found the full API for options are listed on github instead of Java Doc.
When reading files the API accepts several options:
path: Location of files. Similar to Spark can accept standard Hadoop globbing expressions.
rowTag: The row tag of your xml files to treat as a row. For example, in this xml <books> <book><book> ...</books>, the appropriate value would be book. Default is ROW.
samplingRatio: Sampling ratio for inferring schema (0.0 ~ 1). Default is 1. Possible types are StructType, ArrayType, StringType, LongType, DoubleType, BooleanType, TimestampType and NullType, unless user provides a schema for this.
excludeAttribute : Whether you want to exclude attributes in elements or not. Default is false.
treatEmptyValuesAsNulls : (DEPRECATED: use nullValue set to "") Whether you want to treat whitespaces as a null value. Default is false
mode: The mode for dealing with corrupt records during parsing. Default is PERMISSIVE.
PERMISSIVE :
When it encounters a corrupted record, it sets all fields to null and puts the malformed string into a new field configured by columnNameOfCorruptRecord.
When it encounters a field of the wrong datatype, it sets the offending field to null.
DROPMALFORMED : ignores the whole corrupted records.
FAILFAST : throws an exception when it meets corrupted records.
inferSchema: if true, attempts to infer an appropriate type for each resulting DataFrame column, like a boolean, numeric or date type. If false, all resulting columns are of string type. Default is true.
columnNameOfCorruptRecord: The name of new field where malformed strings are stored. Default is _corrupt_record.
attributePrefix: The prefix for attributes so that we can differentiate attributes and elements. This will be the prefix for field names. Default is _. Can be empty, but only for reading XML.
valueTag: The tag used for the value when there are attributes in the element having no child. Default is _VALUE.
charset: Defaults to 'UTF-8' but can be set to other valid charset names
ignoreSurroundingSpaces: Defines whether or not surrounding whitespaces from values being read should be skipped. Default is false.
wildcardColName: Name of a column existing in the provided schema which is interpreted as a 'wildcard'. It must have type string or array of strings. It will match any XML child element that is not otherwise matched by the schema. The XML of the child becomes the string value of the column. If an array, then all unmatched elements will be returned as an array of strings. As its name implies, it is meant to emulate XSD's xs:any type. Default is xs_any. New in 0.11.0.
rowValidationXSDPath: Path to an XSD file that is used to validate the XML for each row individually. Rows that fail to validate are treated like parse errors as above. The XSD does not otherwise affect the schema provided, or inferred. Note that if the same local path is not already also visible on the executors in the cluster, then the XSD and any others it depends on should be added to the Spark executors with SparkContext.addFile. In this case, to use local XSD /foo/bar.xsd, call addFile("/foo/bar.xsd") and pass just "bar.xsd" as rowValidationXSDPath.
ignoreNamespace: If true, namespaces prefixes on XML elements and attributes are ignored. Tags <abc:author> and <def:author> would, for example, be treated as if both are just <author>. Note that, at the moment, namespaces cannot be ignored on the rowTag element, only its children. Note that XML parsing is in general not namespace-aware even if false. Defaults to false. New in 0.11.0.
timestampFormat: Specifies an additional timestamp format that will be tried when parsing values as TimestampType columns. The format is specified as described in DateTimeFormatter. Defaults to try several formats, including ISO_INSTANT, including variations with offset timezones or no timezone (defaults to UTC). New in 0.12.0.
dateFormat: Specifies an additional timestamp format that will be tried when parsing values as DateType columns. The format is specified as described in DateTimeFormatter. Defaults to ISO_DATE. New in 0.12.0.
When writing files the API accepts several options:
path: Location to write files.
rowTag: The row tag of your xml files to treat as a row. For example, in <books> <book><book> ...</books>, the appropriate value would be book. Default is ROW.
rootTag: The root tag of your xml files to treat as the root. For example, in <books> <book><book> ...</books>, the appropriate value would be books. It can include basic attributes by specifying a value like books foo="bar" (as of 0.11.0). Default is ROWS.
declaration: Content of XML declaration to write at the start of every output XML file, before the rootTag. For example, a value of foo causes <?xml foo?> to be written. Set to empty string to suppress. Defaults to version="1.0" encoding="UTF-8" standalone="yes". New in 0.14.0.
nullValue: The value to write null value. Default is string null. When this is null, it does not write attributes and elements for fields.
attributePrefix: The prefix for attributes so that we can differentiating attributes and elements. This will be the prefix for field names. Default is _. Cannot be empty for writing XML.
valueTag: The tag used for the value when there are attributes in the element having no child. Default is _VALUE.
compression: compression codec to use when saving to file. Should be the fully qualified name of a class implementing org.apache.hadoop.io.compress.CompressionCodec or one of case-insensitive shorten names (bzip2, gzip, lz4, and snappy). Defaults to no compression when a codec is not specified.
timestampFormat: Controls the format used to write TimestampType format columns. The format is specified as described in DateTimeFormatter. Defaults to ISO_INSTANT. New in 0.12.0.
dateFormat: Controls the format used to write DateType format columns. The format is specified as described in DateTimeFormatter. Defaults to ISO_DATE. New in 0.12.0.