How to read a kinesis data stream on Amazon EMR?

user_10623162 0
Viewed 658

I have a kinesis datastream which contains a stream of tweets. How can I read this stream on pyspark (on emr)? I want to be able to perform operations on this datastream.

records = KinesisUtils.createStream(
    ssc, appName, streamName, endpointUrl, regionName,
    InitialPositionInStream.LATEST, 2, 
    StorageLevel.MEMORY_AND_DISK_2,
    awsAccessKeyId, awsSecretKey)

doing so I always get this error on jupyter:

'JavaPackage' object is not callable
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/streaming/kinesis.py", line 84, in createStream
    helper = ssc._jvm.org.apache.spark.streaming.kinesis.KinesisUtilsPythonHelper()
TypeError: 'JavaPackage' object is not callable
amazon-kinesis amazon-emr pyspark amazon-web-services
1 Answers

In a cell before your code add this line. It should install the dependency that ia missing in pyspark

%%configure
{ "conf": {"spark.jars.packages": "org.apache.spark:spark-streaming-kinesis-asl_2.11:2.4.0" }}
user_92777470
Related