Pyspark variable has datatype decimal(6,-12). df.dtypes and df.columns gives error ValueError: Could not parse datatype: decimal(6,-12)

I have a spark dataframe for which I am getting error ValueError: Could not parse datatype: decimal(6,-12) whenever I execute df.dtypes or df.columns because of one particular variable which has datatype decimal (6,-12).

    df = spark.read.csv("data.csv",inferSchema=True,header=True)  
    df.columns

Running df.columns or df.dtypes gives the below error

    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-26-0581cf80a9b2> in <module>
    ----> 1 df.columns
    
    /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p4623.11628701/lib/spark/python/pyspark/sql/dataframe.py in columns(self)
        934         ['age', 'name']
        935         """
    --> 936         return [f.name for f in self.schema.fields]
        937 
        938     @since(2.3)
    
    /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p4623.11628701/lib/spark/python/pyspark/sql/dataframe.py in schema(self)
        251         if self._schema is None:
        252             try:
    --> 253                 self._schema = _parse_datatype_json_string(self._jdf.schema().json())
        254             except AttributeError as e:
        255                 raise Exception(
    
    /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p4623.11628701/lib/spark/python/pyspark/sql/types.py in _parse_datatype_json_string(json_string)
        867     >>> check_datatype(complex_maptype)
        868     """
    --> 869     return _parse_datatype_json_value(json.loads(json_string))
        870 
        871 
    
    /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p4623.11628701/lib/spark/python/pyspark/sql/types.py in _parse_datatype_json_value(json_value)
        884         tpe = json_value["type"]
        885         if tpe in _all_complex_types:
    --> 886             return _all_complex_types[tpe].fromJson(json_value)
        887         elif tpe == 'udt':
        888             return UserDefinedType.fromJson(json_value)
    
    /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p4623.11628701/lib/spark/python/pyspark/sql/types.py in fromJson(cls, json)
        575     @classmethod
        576     def fromJson(cls, json):
    --> 577         return StructType([StructField.fromJson(f) for f in json["fields"]])
        578 
        579     def fieldNames(self):
    
    /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p4623.11628701/lib/spark/python/pyspark/sql/types.py in <listcomp>(.0)
        575     @classmethod
        576     def fromJson(cls, json):
    --> 577         return StructType([StructField.fromJson(f) for f in json["fields"]])
        578 
        579     def fieldNames(self):
    
    /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p4623.11628701/lib/spark/python/pyspark/sql/types.py in fromJson(cls, json)
        432     def fromJson(cls, json):
        433         return StructField(json["name"],
    --> 434                            _parse_datatype_json_value(json["type"]),
        435                            json["nullable"],
        436                            json["metadata"])
    
    /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p4623.11628701/lib/spark/python/pyspark/sql/types.py in _parse_datatype_json_value(json_value)
        880             return DecimalType(int(m.group(1)), int(m.group(2)))
        881         else:
    --> 882             raise ValueError("Could not parse datatype: %s" % json_value)
        883     else:
        884         tpe = json_value["type"]
    
    ValueError: Could not parse datatype: decimal(6,-12)

If I change the column type to double or string, I am able to proceed further. But I am working on an automation tool and need a solution which can work on all datasets.

I tried the solution given in df.columns is giving ValueError: in pyspark which is given below.

    from pyspark.sql import SparkSession
    from pyspark.sql.types import *
    
    spark = SparkSession.builder.appName("basics").getOrCreate()
    df = spark.read.csv("data.csv",inferSchema=True,header=True)  
    for column_type in df.dtypes:
        if 'string' in column_type[1]:
            df = df.withColumn(column_type[0], df[column_type[0]].cast(StringType()))
        elif 'double' in column_type[1]:
            df = df.withColumn(column_type[0],df[column_type[0]].cast(DoubleType()))
        elif 'int' in column_type[1]:
            df = df.withColumn(column_type[0],df[column_type[0]].cast(IntegerType()))
        elif 'bool' in column_type[1]:
            df = df.withColumn(column_type[0], df[column_type[0]].cast(BooleanType()))
        elif 'decimal' in column_type[1]:
            df = df.withColumn(column_type[0],df[column_type[0]].cast(DoubleType()))
        # add as many conditions as you need for types
    
    df.schema

But unfortunately df.dtypes mentioned in this code gives the same error.

The only piece of code through which I was able to check the datatype was df.printSchema(). Is there a way through which I can read the output of df.printSchema() and change the datatypes of variables with datatype decimal to double type?

    df.select('variable_name').printSchema()
    
    root
     |-- variable_name: decimal(6,-12) (nullable = true)