Create a version column based on hash column

user_12725687 0
Viewed 85

I want to add to myDF a version column where put the version of my Hash column. My DF:

   Hash    Code   P1  P2  P3  P4  P5... Pn
x-x-1-x-x    A    x   x   1   x   x
x-1-x-x-x    A    x   1   x   x   x
5-x-1-x-x    A    5   x   1   x   x
x-x-1-x-2    A    x   x   1   x   2
x-x-1-x-x    A    x   x   1   x   x
x-x-x-7-x    A    x   x   x   7   x
5-x-1-x-x    A    5   x   1   x   x

And i need:

Version    Hash    Code   P1  P2  P3  P4  P5... Pn
  1     x-x-1-x-x    A    x   x   1   x   x
  2     x-1-x-x-x    A    x   1   x   x   x
  3     5-x-1-x-x    A    5   x   1   x   x
  4     x-x-1-x-2    A    x   x   1   x   2
  1     x-x-1-x-x    A    x   x   1   x   x
  6     x-x-x-7-x    A    x   x   x   7   x
  3     5-x-1-x-x    A    5   x   1   x   x

I'm working with PySpark, don't know if it exists a method that could help me.

Edit: The version column that i need it should be made considering the Hash column. The process should be:

Read the Hash value, exists on previous rows? then put the version value that were assigned previously. Version value its like an ID, hash could be repeated, if it is, i need to know that version and put in on that row i'm reading. Unless i need to create a new version (id) last value version +1.

pyspark dataframe python
1 Answers
df = sqlContext.createDataFrame(
  [
     ('x-x-1-x-x','A', 'x', 'x', '1', 'x', 'x'),
     ('x-1-x-x-x','A', 'x', '1', '1', 'x', 'x'),
     ('5-x-x-x-x','A', '5', 'x', 'x', 'x', 'x'),
     ('x-x-1-x-2','A', 'x', 'x', '1', 'x', '2'),
     ('x-x-1-x-x','A', 'x', 'x', '1', 'x', 'x'),
     ('x-x-x-7-x','A', 'x', 'x', 'x', '7', 'x'),
     ('5-x-x-x-x','A', '5', 'x', '1', 'x', 'x')
  ]
  ,['Hash', 'Code', 'P1', 'P2', 'P3', 'P4', 'P5']
)




from pyspark.sql import functions as F

Hash_ids = df\
            .select("Hash")\
            .dropDuplicates()\
            .rdd.zipWithIndex()\
            .toDF()\
            .withColumn("Version", F.col('_2')+1)\
            .select(F.col("_1.*"), F.col("Version"))

df\
  .join(Hash_ids, 'Hash', 'inner')\
  .show()
user_116985620
Related