Why results structure change when I change column name in Python spark?

user_1492229 0
Viewed 47

My dataframe dff is this 

       10311  105903003  373873005  385055001  392521001  ...  26  27  28  29  30
0       21.0        5.0        5.0       21.0        8.0  ...   0   0   0   0   1
1        0.0        3.0        3.0        0.0        6.0  ...   0   0   0   0   1
2       32.0        8.0        8.0       32.0        4.0  ...   0   0   0   0   1
3       15.0        7.0        7.0       15.0        5.0  ...   0   0   0   0   1
4        0.0        4.0        4.0        0.0        4.0  ...   0   0   0   0   1
     ...        ...        ...        ...        ...  ...  ..  ..  ..  ..  ..
52699    0.0        2.0        2.0        0.0        6.0  ...   0   0   0   0   1
52700    0.0        2.0        2.0        0.0        6.0  ...   0   0   0   0   1
52701   22.0        4.0        4.0       22.0        9.0  ...   0   0   0   0   1
52702    0.0        4.0        4.0        0.0        8.0  ...   0   0   0   0   1
52703    0.0        2.0        2.0        0.0        2.0  ...   0   0   0   0   1
[52704 rows x 43 columns]

and this is my code

from pyspark.sql import SQLContext
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)

spark_dff = sqlContext.createDataFrame(dff.astype(float))
spark_dff.head(4)

Returns this

Out[99]: 
[Row(10311=21.0, 105903003=5.0, 373873005=5.0, 385055001=21.0, 392521001=8.0, 410942007=5.0, 423367003=12.0, 46992007=21.0, 4850=27.0, 87612001=43.0, filename=1.0, filename_int=1.0, 0=0.0, 1=0.0, 2=0.0, 3=0.0, 4=0.0, 5=0.0, 6=0.0, 7=0.0, 8=0.0, 9=0.0, 10=0.0, 11=0.0, 12=0.0, 13=0.0, 14=0.0, 15=0.0, 16=0.0, 17=0.0, 18=0.0, 19=0.0, 20=0.0, 21=0.0, 22=0.0, 23=0.0, 24=0.0, 25=0.0, 26=0.0, 27=0.0, 28=0.0, 29=0.0, 30=1.0),
 Row(10311=0.0, 105903003=3.0, 373873005=3.0, 385055001=0.0, 392521001=6.0, 410942007=3.0, 423367003=0.0, 46992007=0.0, 4850=6.0, 87612001=3.0, filename=10.0, filename_int=10.0, 0=0.0, 1=0.0, 2=0.0, 3=0.0, 4=0.0, 5=0.0, 6=0.0, 7=0.0, 8=0.0, 9=0.0, 10=0.0, 11=0.0, 12=0.0, 13=0.0, 14=0.0, 15=0.0, 16=0.0, 17=0.0, 18=0.0, 19=0.0, 20=0.0, 21=0.0, 22=0.0, 23=0.0, 24=0.0, 25=0.0, 26=0.0, 27=0.0, 28=0.0, 29=0.0, 30=1.0),
 Row(10311=32.0, 105903003=8.0, 373873005=8.0, 385055001=32.0, 392521001=4.0, 410942007=8.0, 423367003=15.0, 46992007=32.0, 4850=9.0, 87612001=9.0, filename=100.0, filename_int=100.0, 0=0.0, 1=0.0, 2=0.0, 3=0.0, 4=0.0, 5=0.0, 6=0.0, 7=0.0, 8=0.0, 9=0.0, 10=0.0, 11=0.0, 12=0.0, 13=0.0, 14=0.0, 15=0.0, 16=0.0, 17=0.0, 18=0.0, 19=0.0, 20=0.0, 21=0.0, 22=0.0, 23=0.0, 24=0.0, 25=0.0, 26=0.0, 27=0.0, 28=0.0, 29=0.0, 30=1.0),
 Row(10311=15.0, 105903003=7.0, 373873005=7.0, 385055001=15.0, 392521001=5.0, 410942007=7.0, 423367003=7.0, 46992007=15.0, 4850=12.0, 87612001=21.0, filename=10000.0, filename_int=10000.0, 0=0.0, 1=0.0, 2=0.0, 3=0.0, 4=0.0, 5=0.0, 6=0.0, 7=0.0, 8=0.0, 9=0.0, 10=0.0, 11=0.0, 12=0.0, 13=0.0, 14=0.0, 15=0.0, 16=0.0, 17=0.0, 18=0.0, 19=0.0, 20=0.0, 21=0.0, 22=0.0, 23=0.0, 24=0.0, 25=0.0, 26=0.0, 27=0.0, 28=0.0, 29=0.0, 30=1.0)]

spark_dff

returns this

spark_dff
Out[100]: DataFrame[10311: double, 105903003: double, 373873005: double, 385055001: double, 392521001: double, 410942007: double, 423367003: double, 46992007: double, 4850: double, 87612001: double, filename: double, filename_int: double, 0: double, 1: double, 2: double, 3: double, 4: double, 5: double, 6: double, 7: double, 8: double, 9: double, 10: double, 11: double, 12: double, 13: double, 14: double, 15: double, 16: double, 17: double, 18: double, 19: double, 20: double, 21: double, 22: double, 23: double, 24: double, 25: double, 26: double, 27: double, 28: double, 29: double, 30: double]

Now here is my problem

lz = ['10311','105903003','373873005','385055001']      #<------ (1)

from pyspark.ml.feature import VectorAssembler
vectorAssemblerZ = VectorAssembler(inputCols = lz , outputCol = 'zz')
vhouse_df = vectorAssemblerZ.transform(spark_dff)
vhouse_df = vhouse_df.select(['zz'])
vhouse_df.show(3)

this will return 

+-------------------+
|                 zz|
+-------------------+
|[21.0,5.0,5.0,21.0]|
|  [0.0,3.0,3.0,0.0]|
|[32.0,8.0,8.0,32.0]|
+-------------------+

This looks good an array with 4 values in it.

but when I change the line (1)

lz = ['1','2','3','4']

The result becomes different structure !!!!

+-----------------+
|               zz|
+-----------------+
|        (4,[],[])|
|    (4,[2],[1.0])|
|    (4,[3],[1.0])|
+-----------------+

Its a number then array, then another array !!!

I dont know why it is this structure

I made another change 

lz = ['10311','105903003','3','4']

The result is even weirder!!

+------------------+
|                zz|
+------------------+
|[21.0,5.0,0.0,0.0]|
|     (4,[1],[3.0])|
|[32.0,8.0,0.0,0.0]|
+------------------+

Why when I change the column the structure change and how I can I get this fixed?!

pyspark apache-spark python
1 Answers

These are two different types of vector representation called dense and sparse. For example, a dense vector [21.0,5.0,0.0,0.0] is same as sparse vector (4, [0,1], [21.0, 5.0]) where 4 represents size of the vector, [0,1] represents indices having non-zero elements and [21.0, 5.0] represents corresponding values. All other values are assumed to be zero.

The representation of the vector depends on the number of zeros in the vector. If a vector has more zeros it will be represented by a sparse vector whereas if a vector has more non-zero elements it will be represented by a dense vector.

There is no need to fix anything in this.

Check out this spark document for more clarification

user_58373750
Related