1. Why x and y?

Make a key value pair like (word, 1)

Now your key is going to be word and value is going to be 1

When you do reduceByKey, it will add up all values for the same key

reduceByKey(lambda x, y: x + y) will group the rdd elements by the key which is the first element word, and sum up the values. In this particular statement, x is one element accumulating all values of the RDD and y is every other element for the same key/word. Reduce values by adding their values, for the same word or the same key. Might look something like:

# [('This', 1), ('is', 2), ('a', 3), ('random', 1), ('sample.', 2), ('And', 2), ('world', 1), ('count', 2), ('word', 1), ('sample,', 1), ('that', 1), ('it', 1)]

2. Let's breakdown your next question about wordCountsSorted = wordCounts2.map(lambda x: (x[1], x[0])).sortByKey()

Following line will swap the elements of the tuples. Essentially, element at position 0 to be moved to position 1, and element at position 1 to be moved to position 0.

reversed_tup = wordCounts2.map(lambda x: (x[1], x[0]))

Output will look like;

# [(1, 'This'), (2, 'is'), (3, 'a'), (1, 'random'), (2, 'sample.'), (2, 'And'), (1, 'world'), (2, 'count'), (1, 'word'), (1, 'sample,'), (1, 'that'), (1, 'it')]

Now when you do sortByKey, these tuples will be sorted using the key which is as mentioned above the first element of the tuple. So the rdd will be sorted by the count of the words.

wordCountsSorted = reversed_tup.sortByKey()
wordCountsSorted.collect()
# [(1, 'This'), (1, 'random'), (1, 'world'), (1, 'word'), (1, 'sample,'), (1, 'that'), (1, 'it'), (2, 'is'), (2, 'sample.'), (2, 'And'), (2, 'count'), (3, 'a')]

Hope this helps

First Lambda - lambda x: (x, 1)

During the first steps, the input would look below

['self', 'employment', 'building',
'self', 'employment', 'building',
'self', 'employment', 'building', 
'self', 'employment' ]

Here, your first lambda function takes each item of the above list and coverts to below items in the corresponding location.

data = [('self', 1), ('employment', 1), ('building', 1),
('self', 1), ('employment', 1), ('building', 1),
('self', 1), ('employment', 1), ('building', 1),
('self', 1), ('employment', 1) ]

This lambda is same as below

def convert_word_to_key_value(word):
    return (word, 1)

Example:

'self' is converted to ('self',)`

The goal of this lambda was to convert each word to look like a (key, value) tuple.

Second Lambda - lambda x, y: x + y

The goal of this lambda was to group words, i.e. accumulated the values with same keys.

Please notice that we are using reduceByKey which means to reduce the values with the same key names. So the x and y would represent key-values of two items and this operation happens only for items with same key

For example, if there are two tuples like below

('self', 1),('self', 1) would become ('self', 2)

Second example

('self', 2),('self', 1) would become ('self', 1)

Third example

('self', 12),('self', 15) would become ('self', 27)

Last part of your question

wordCountsSorted = wordCounts2.map(lambda x: (x[1], x[0])).sortByKey()

Firstly, here you are interchanging your key & values.

(key, value) becomes (value, key)

Example (say)

('self', 1231) becomes (1231, 'self')

Now you are sorting the items based on the key, in this new format which is 1231 or the grouped value of that key(i.e. word frequency).

@kikee1222,

@Sampath and @pissal provided you detailed explanation for all your questions and I think we're missing key point of one (x) parameter acts like an accumulator

What I don't understand is, what X and Y are in line 2. We only have one numeric input to the lamda function, which is the count column (all the 1's) from wordcounts, so why y?

wordCounts2 = wordCounts.reduceByKey(lambda x, y: x + y) # outputs [('self', 111), ('an', 178), ('internet', 26)

Here you can think of x as an accumulator and y count. Accumulator initializes with 0 and adds up with count for each key and produce the final count for the key

[('a', 1), ('b', 1), ('c',1), ('b',1), ('a',1), ('a',1)]

after applying reduceByKey ( lambda accum, count : accum + count)

you can think of iteration like this

#1 : accum = 0 , count = 1 =>(returns) 1
#2 : accum = 1, count = 1  => 2
#3 : accum = 2, count = 1  => 3

Hope this helps

x and y are temporary variables, Just like for loop in java. so, you can have any variable name. When you do sortByKey it sorts the results by key.