I am farely new to pyspark. While running the below piece of code in pycharm I am getting the expected output I want. But I am getting below error
Traceback (most recent call last):
File "C:\Study\Spark\spark\python\lib\py4j-0.10.7-src.zip\py4j\java_gateway.py", line 1067, in start
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:49748)
Traceback (most recent call last):
File "C:\Study\Spark\spark\python\lib\py4j-0.10.7-src.zip\py4j\java_gateway.py", line 929, in _get_connection
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Study\Spark\spark\python\lib\py4j-0.10.7-src.zip\py4j\java_gateway.py", line 1067, in start
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
**Process finished with exit code 0**
As you can see in the last line Process finished with exit code 0 and I also get my expected output
Here is my code sample
Python-3.7
Spark-2.4.5
def func(row):
temp=row.asDict()
temp["concat_val"]="|".join([str(x) for x in row])
put=Row(**temp)
return put
if __name__ == "__main__":
spark = SparkSession\
.builder.\
master("local[*]")\
.appName("PythonWordCount")\
.getOrCreate()
data1=spark.createDataFrame(
[
("1", 'foo'),
("2", 'bar'),
],
['id', 'txt']
row_rdd = data1.rdd.map(func)
print(row_rdd.collect())
concat_df = row_rdd.toDF()
hash_df = concat_df.withColumn("hash_id", md5(F.col("concat_val")))
hash_df.show()
