Issue with displaying results in the loop after collect()

Viewed 40

What is the problem about?

I have a problem displaying data that has been read from a text file. The file (yields.txt) has 3 lines and it looks like a fourth line is being read as well, with some strange content.

File

File encoding: UTF-8 -> I also check for ASCII but same issue EOL: Unix(LF) -> I also check for Windows (CRLF) but same issue enter image description here

1 -0.0873962663951055 0.0194176287820278 -0.0097985244947938 -0.0457230361016478 -0.0912513154921251 0.0448220622524235
2 0.049279031957286 0.069222988721009 0.0428232461362216 0.0720027150750844 -0.0209348305073702 -0.0641023433269808
3 0.0770763924363555 -0.0790020383071036 -0.0601622344182963 -0.0207625817307966 -0.0193570710130222 -0.0959349375686872

Bug Description

Log from console

in mapper return Row(ID=int(fields[0]),asset_1 = float(fields[1]), asset_2 = float(fields[2]), asset_3 = float(fields3),asset_4 = float(fields[4]), asset_5 = float(fields[5]), asset_6 = float(fields[6])) ValueError: invalid literal for int() with base 10: b'PK\x03\x04\x14\x00\x00\x00\x08\x00AW\xef\xbf\xbdT\xef\xbf\xbdu\xef\xbf\xbdDZ\xef\xbf\xbd\x1e\x03i\x18\xef\xbf\xbd\x07'

I have also tried to find out what is within this content Trying to decode it

and it is some strange data that does not appear in the text file at all which I checked with the script shown below:

import os

DATA_FOLDER_PATHNAME = 
'\\'.join(os.path.dirname(__file__).split('\\') 
[:-1])+'\\'+'data'+'\\'+'yields.txt'

with open(DATA_FOLDER_PATHNAME, 'r', encoding='ansi') as f:
    print(f.read())

You can see that an empty line is visible but I do not know how to improve my code to avoid this bug. empty line

Code

import findspark
import os
findspark.init(PATH_TO_SPARK)

from pyspark.sql import SparkSession
from pyspark.sql import Row

DATA_FOLDER_PATHNAME = '\\'.join(os.path.dirname(__file__).split('\\')[:-1])+'\\'+'data' # location of data file

def mapper(line):
    fields = line.split()
    return Row(ID=int(fields[0]),asset_1 = float(fields[1]), asset_2 = float(fields[2]), asset_3 = float(fields[3]),asset_4 = float(fields[4]), asset_5 = float(fields[5]), asset_6 = float(fields[6]))

spark = SparkSession.builder.appName("SparkSQL").getOrCreate()
lines = spark.sparkContext.textFile(DATA_FOLDER_PATHNAME, minPartitions = 2000, use_unicode = False) 
assets_with_yields_rdd = lines.map(mapper)
assets_with_yields_df = spark.createDataFrame(assets_with_yields_rdd).cache()
assets_with_yields_df.createOrReplaceTempView('assets_with_yields_view')

assets_with_yields_view_df = spark.sql('select * from assets_with_yields_view')

print(80*'-')
for asset in assets_with_yields_view_df.collect():
    print(asset)
print(80*'-')
spark.stop()

Question

Does anyone know what could cause such a weird issue?

1 Answers

The reason for this was that I had several files in one folder and it read them all in turn, so there were some discrepancies.

Related