Kryo serialization failed: Buffer overflow

We read data present in hour format present in S3 through spark in scala.For example,

 sparkSession
.createDataset(sc
 .wholeTextFiles(("s3://<Bucket>/<key>/<yyyy>/<MM>/<dd>/<hh>/*"))
.values
.flatMap(x=> {x
.replace("\n", "")
 .replace("}{", "}}{{")
 .split("\\}\\{")}))

Doing the above slice and dice (like replace and split)to convert the pretty json data in form of json lines(one json record per json).

Now I am getting this error while running on EMR:

Job aborted due to stage failure: Task 1 in stage 11.0 failed 4 times, most recent failure: Lost task 1.3 in stage 11.0 (TID 43, ip-10-0-2-22.eu-west-1.compute.internal, executor 1): org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 1148334. To avoid this, increase spark.kryoserializer.buffer.max value.

I have tried increasing the value for kyro serializer buffer --conf spark.kryoserializer.buffer.max=2047m but still I am getting this error for reading data for some hour locations like hours 09,10 and for other hours it is reading fine.

I wanted to ask how to remove this error and whether I need to add something else in spark configurations like change number of partitions?Thanks