MicroBatchExecution: Query all_tweets terminated with error java.lang.IndexOutOfBoundsException: at 0 deleting 7, pyspark

Viewed 34

I'm implementing a real-time streaming sentiment analysis (positive, negative, neutral) pipeline using Spark for English comments in Twitter including some keywords .I'm getting this error when I use this pyspark and tweepy code.

ERROR MicroBatchExecution: Query all_tweets [id = f4f91819-5806-4f04-9e01-7361686d8d7a, runId = bfa7ee98-b53b-45b4-9cce-32c6cfb68863] terminated with error
java.lang.IndexOutOfBoundsException: at 0 deleting 7

this is the code I'm using.

    spark = SparkSession.builder.appName("TwitterSentimentAnalysis").getOrCreate()
    lines = spark.readStream.format("socket").option("host", "0.0.0.0").option("port", 5555).load()
    words = preprocessing(lines)
    words = text_classification(words)
    words = words.repartition(1)
    query = words.writeStream.queryName("all_tweets")\
        .outputMode("append").format("parquet")\
        .option("path", "./parc")\
        .option("checkpointLocation", "./check")\
        .trigger(processingTime='60 seconds').start()
    query.awaitTermination()
0 Answers
Related