Exporting cloudwatch logs results in limitExceededException

We are using step functions to periodically aggregate logs and forward logs to s3 buucket using lambda, limitExceededException is coming from Forward Logs Lambda and this happens randomly not all the time.

Since I haven't written code for this implementation so posting a reproducible example would be difficult.

Upon my investigation, I found out that this error is mentioned in https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateExportTask.html where Each account can only have one active (RUNNING or PENDING) export task at a time.

lambda code

async invokeForwardingTask(
    options: ForwardingTaskInvokingOptions
  ): Promise<void> {
    const { from, logGroupName, to } = options
    const date = dayjs.utc(from)
    const partitionPrefix = [
      `year=${date.format('YYYY')}`,
      `month=${date.format('MM')}`,
      `day=${date.format('DD')}`,
    ].join('/')
    const categoryPrefix = logGroupName.replace(/^\//, '')


    await this.cwl
      .createExportTask({
        logGroupName,
        destination: this.logDestination,
        from,
        to,
        destinationPrefix: `${categoryPrefix}/${partitionPrefix}`,
      })
      .promise()

step functions code

const logForwarding = new tasks.LambdaInvoke(this, 'Forward a log', {
      lambdaFunction: forwarder.lambdaFunction,
      inputPath: '$.systemLog',
      outputPath: '$.Payload',
      timeout: Duration.seconds(
        logForwardingRetryConfig.interval.toSeconds() +
          logForwardingRetryConfig.interval.toSeconds() *
            logForwardingRetryConfig.backOffRate *
            (logForwardingRetryConfig.maxAttempts - 1)
      ),
    })

    const oneMinWaiting = new sfn.Wait(this, 'Wait for a while', {
      time: sfn.WaitTime.duration(Duration.minutes(2)),
    })

    const choice = new sfn.Choice(this, 'No Logs?')
      .when(
        sfn.Condition.numberEquals('$.systemLog.length', 0),
        new sfn.Succeed(this, 'No more logs')
      )
      .otherwise(logForwarding)
    const definition = logAggregation.next(choice)
    choice.afterwards().next(oneMinWaiting).next(choice)
    logForwarding.addRetry(logForwardingRetryConfig)

    const stateMachine = new StepFunctions(this, 'StepFunctions', {
      definition,
    })

    // Every 1 AM in JST
    stateMachine.addEventsRule('cron(0 16 * * ? *)')