Spark Structured Stream - Kinesis as Data Source

I am trying to consume kinesis data stream records using psypark structured stream. I am trying to run this code in aws glue batch job. My goal is to use checkpoint and save checkpoints and data to s3. I am able to consume the data but it is giving only few records for every trigger whereas kinesis data stream has lot of records. I am using TRIM_HORIZON which is alias to earliest and trigger spark.writestream once so that it executes once and shuts down the cluster. When i run the job again, it picks latest offset from checkpoint and runs.

kinesis = spark.readStream.format('kinesis') \
        .option('streamName', kinesis_stream_name) \
        .option('endpointUrl', 'blaablaa')\
        .option('region', region) \
        .option('startingPosition', 'TRIM_HORIZON')\
        .option('maxOffsetsPerTrigger',100000)\
        .load()

// do some transformation here

TargetKinesisData = stream_data.writeStream.format("parquet").outputMode('append').option(
        "path", s3_target).option("checkpointLocation", checkpoint_location).trigger(once=True).start().awaitTermination()