Encode the languages and special characters in csv file in pyspark and also need help to handle null data

I have tried with csv data file to encode the different languages to handle the null values and to fill it with mean value.

Please find the points here:

1. Only three columns the csv file has but the no of rows are 250000
2. Only target column has null data
3. I have tried with pyspark imputer function to fill null data with its mean and its done but issue found is for few rows the data split into next cells of the same row, hence unable to complete the task

Issues

1. I'm unable to encode all languages from csv dataset file
2. The 'question' column data split into parts and filled in the next cells of the same row and it happened for few rows only

I need help:

• Should not get split the data and should handle all languages by encoding it
• Should fill the null data with mean or median of target column because only the target column has null data

I would appreciate if anybody help me solve this, please

My code

from pyspark.sql import SparkSession
spark = SparkSession.builder.master("local[1]")
    .appName("CSV_NULL")
    .getOrCreate()

df1_encoded = spark.read.options(header="True",         
inferSchema="True")
    .option("encoding", "utf-16")\
    .csv("csv23numNull.csv")

df1_encoded.printSchema()

df1_encoded.show()

======== imputing
from pyspark.ml.feature import Imputer
from pyspark.ml.feature import StringIndexer, VectorIndexer,             
                                OneHotEncoder,
                                VectorAssembler

df1_impute = Imputer(inputCols = ['target'],
                outputCols = ["{}_imputed".format(z) for z in     
                            ['target']]).setStrategy("mean")

df2 = df1_impute.fit(df1_encoded).transform(df1_encoded)

# export into new csv file
df2.coalesce(1).write.option("header","true")
    .option("sep",",")
    .mode("default")
    .csv("csv23EncodedStr2intandMean")

link attached for csv file

https://drive.google.com/file/d/1gBst292_H88HTt7CRovIpM1PIpnl6E7m/view?usp=sharing

This is my output

df2.printSchema()
root
 |-- 煩搬煵敳瑩潮ⱴ慲来�: string (nullable = true)

df2.show()
+------------------------------------+
|                  煩搬煵敳瑩潮ⱴ慲来�|
+------------------------------------+
|  〰〰㈱㘵㌶㑤戹㈳挷收ⱈ潷⁤楤⁑略扥...|
|  〰〰㌲㤳㤰ㄷㄲづ㙥㐴Ⱒ䑯⁹潵⁨慶攠...|
| 〰〰㐱㉣愶攴㘲㡣攲捦ⱗ桹⁤潥猠癥汯...|
|  〰〰㐲扦㠵慡㐹㡣搷㡥ⱈ潷⁤楤⁏瑴漠...|
|  〰〰㐵㕤晡㍥〱敡攳慦ⱃ慮⁉⁣潮癥牴...|
| 〰〰㑦㥡㐶㉡㌵㝣㌳扥Ⱒ䥳⁇慺愠獬潷...|
| 〰〰㔰㔹愰㙥攱㥥ㄱ慤Ⱒ坨礠摯敳⁑畯...|
|  〰〰㔵㥦㠷㔸㌲㜴㕥㉥ⱉ猠楴⁣牡穹⁩...|
|  〰〰㕢搳㐲㙢㉤っ㠳〵Ⱒ䥳⁴桥牥⁳畣...|
|  〰〰㙥㘹㈸挵摦㘰敡换Ⱒ䥳⁩琠橵獴...|
|〰〰㜵昶㝤搵㤵挳摥戵ⱗ桡琠捡渠祯甠...|
| 〰〰㜶昳戴㈷㜶挶㤲摥ⱈ潷⁷敲攠瑨攠...|
| 〰〰㠹㜹㉢㍦挸〲㘷㐱Ⱒ坨慴⁩猠瑨攠...|
| 〰〰㤲愹ぢ捦扦攸捤㠸ⱃ慮⁷攠畳攠潵...|
|  〰〰㤵㘸づ㐱愹愶昶攳Ⱒ䤠慭″〬楶...|
|  〰〰愸㤹㐲攳ㄴ㍥㌳㍡ⱗ桡琠摯⁹潵...|
| 〰〰戸攱㈷㥥慡ち㜰㘲ⱈ潷⁤楦晩捵汴...|
|〰〰扣て㘲㔰て㔵㤵㥦ⱈ慶攠祯甠汩捫...|
|〰〰捥㙣㌱昱㑤㍥〹散ⱄ漠祯甠瑨楮欠...|
| 〰〰搳㈹㌳㈸㐵戸愷晡ⱈ潷慮礠扡牯...|
+------------------------------------+
only showing top 20 rows