Using .filter and .collect on small practice RDD to print all words beginning with 'h' (pyspark beginner)

user_16923006 0
Viewed 85

I'm just starting to learn how to use pyspark on Jupyternotebook and a beginner coder.

To test the waters I created this array: word = ['hey', 'how', 'are', 'you']

Then this RDD: wordRDD = spark.sparkContext.parallelize([word])

I am now trying to use .filter and .collect on wordRDD to print all words beginning with 'h' but just can't reach the solution. Does anyone have any ideas?

This is as far as I have got:

word = ['hey', 'how', 'are', 'you']
wordRDD = spark.sparkContext.parallelize([word])

print(wordRDD.filter.map(lambda x=x: x in wordRDD if x[0] == "h").collect())
rdd jupyter-notebook pyspark lambda python
1 Answers

You are almost there, the function call is RDD.filter:

words = ['hey', 'how', 'are', 'you']
wordRDD = spark.sparkContext.parallelize(words)

filterRDD = wordRDD.filter(lambda word: word.startswith("h"))
print(filterRDD.collect())
user_60802760
Related