How random number behaves in spark

Question

What is the use of rand(INT) and what is happening when we multiply another number to that result rand(INT) * 10 or rand(INT) * 12

What benefit do we get if we multiply some number to rand(INT)

It seems that it produces distributed number as per the number we use for multiply. In my case it is 19. If we give 19 then all our random numbers are never be beyond 19. Is that so?

Also, it is mentioned that rand(seed) produces output in deterministic way. What does it mean, and why does the value in my dataset actually varies?

scala> val df = hc.sql("""select 8 , rand(123456), rand(123456)*19, floor(rand(123456)*19) as random_number ,rand(1)  from v-vb.omega """)
df: org.apache.spark.sql.DataFrame = [_c0: int, _c1: double, _c2: double, random_number: bigint, _c4: double]

scala> df.show(100,false)
 +---+--------------------+-------------------+-------------+--------------------+
 |_c0|_c1                 |_c2                |random_number|_c4                 |
 +---+--------------------+-------------------+-------------+--------------------+
 |8  |0.4953081723589211  |9.4108552748195    |9            |0.13385709732307427 |
|8  |0.8134447122366524  |15.455449532496395 |15           |0.5897562959687032  |
 |8  |0.37061329883387006 |7.0416526778435315 |7            |0.01540012100242305 |
 |8  |0.039605242829950704|0.7524996137690634 |0            |0.22569943461197162 |
 |8  |0.6789785261613072  |12.900591997064836 |12           |0.9207602095112212  |
 |8  |0.9696879210080743  |18.424070499153412 |18           |0.6222816020094926  |
 |8  |0.773136636564404   |14.689596094723676 |14           |0.1029837279488438  |
 |8  |0.7990411192888535  |15.181781266488215 |15           |0.6678762139023474  |
 |8  |0.8089896546054375  |15.370803437503312 |15           |0.06748208566157787 |
 |8  |0.0895536147884225  |1.7015186809800276 |1            |0.5215181769983375  |
 |8  |0.7385237395885733  |14.031951052182894 |14           |0.8895645473999635  |
|8  |0.059503458368902584|1.130565709009149  |1            |0.321509746474296   |
|8  |0.14662556746599353 |2.785885781853877  |2            |0.28823975483307396 |
|8  |0.28066416785509374 |5.332619189246781  |5            |0.45999786693699807 |
|8  |0.5563531082651644  |10.570709057038123 |10           |0.17320175842535657 |
|8  |0.6619862377687996  |12.577738517607193 |12           |1.152006730106292E-4|
|8  |0.9090845495301373  |17.272606441072607 |17           |0.7500451351287378  |

user_638764 · Answer

In Spark, rand(seed) and randn(seed) are not deterministic, which is an unresolved bug. Corresponding note was added to its source code via JIRA SPARK-13380:

  /**
   * Generate a random column with independent and identically distributed (i.i.d.) samples
   * from U[0.0, 1.0].
   *
   * @note The function is non-deterministic in general case.
   *
   * @group normal_funcs
   * @since 1.4.0
   */
  def rand(seed: Long): Column = withExpr { Rand(seed) }

user_2700344 · Answer

Deterministic means that function result is always the same for the same argument (Different invocations of function with the same argument produces the same result).

In your dataset it is definitely not deterministic because there are different random numbers produced with the same seed argument. If the documentation states it should be deterministic, then it is a bug in the documentation or bug in the function implementation.

Another question is why rand(12345)*19 is never beyond 19? This is because rand values are > 0 and < 1. This I believe is as per specification

How random number behaves in spark

2 Answers