Change datatype but return null value for Dataframe

user_15458951 0
Viewed 519

I'm new to Pyspark 3.0 and I have this homework where I need to change the string (geolocation) to tuple numeric data type (geolocation1).

Here is my code:

from pyspark.sql.functions import *
from pyspark.sql.types import *

df = df2.withColumn('geolocation1', col('geolocation').cast('double'))

Output:

|                    geolocation             | geolocation1 |
| ------------------------------------------| ----------------- |
| (-37.80899950, 140.96004459) | null                |
| (-37.80899952, 140.96004451) | null                |

What have I done wrong here?

pyspark apache-spark-sql apache-spark python
2 Answers

If you have a string like that, you can remove the parenthesis and split by comma, then cast to array<double>:

import pyspark.sql.functions as F

df = df2.withColumn(
    'geolocation1', 
    F.split(
        F.regexp_replace('geolocation', '[\( \)]', ''), 
        ','
    ).cast('array<double>')
)

df.show(truncate=False)
+----------------------------+---------------------------+
|geolocation                 |geolocation1               |
+----------------------------+---------------------------+
|(-37.80899950, 140.96004459)|[-37.8089995, 140.96004459]|
+----------------------------+---------------------------+

df.printSchema()
root
 |-- geolocation: string (nullable = false)
 |-- geolocation1: array (nullable = false)
 |    |-- element: double (containsNull = true)
user_141657300

I would like to give some suggestions before answering this

  1. First , you need to understand what is a double type. Here you are blindly converting a string which contains non-numeric characters as well to a numeric format.So internally spark will throw an exception which will be caught and null will be populated as an output.

  2. And as I understood from the name of field, its a Geolocation which will be a combination of latitude and longitude. So I assume who ever given you this homework needs these two values as new columns. And if my assumption is correct , below is one of the ways to achieve it.

enter image description here

user_101183930
Related