How to exclude the first line with spark.read method?

Viewed 27

I am using databricks with pyspark. I would like to read this file (with the 'spark.read' method) which has the first and last lines to be excluded:

<X> example </X>
ID
11111   
22222    
<X> example </X>

How do I exclude the first row (or multiple rows if that is the case)? I have tried using:

  df = spark.read \
    .options(header='false') 

I was unsuccessful. Alternatively adding a '#' character to the beginning of the file (because then the spark.read command can interpret it as a comment and ignore the line) but I am dealing with very large files and would like to avoid reading or unnecessary steps that lengthen the process

  line = """#"""
  with open("myfile.txt", 'r+') as file: 
    file_data = file.read() 
    file.seek(0, 0) 
    file.write(line + file_data) 
    
1 Answers
Related