I am using databricks with pyspark. I would like to read this file (with the 'spark.read' method) which has the first and last lines to be excluded:
<X> example </X>
ID
11111
22222
<X> example </X>
How do I exclude the first row (or multiple rows if that is the case)? I have tried using:
df = spark.read \
.options(header='false')
I was unsuccessful. Alternatively adding a '#' character to the beginning of the file (because then the spark.read command can interpret it as a comment and ignore the line) but I am dealing with very large files and would like to avoid reading or unnecessary steps that lengthen the process
line = """#"""
with open("myfile.txt", 'r+') as file:
file_data = file.read()
file.seek(0, 0)
file.write(line + file_data)