Error saving state: Failed to upload state: POST

I have Gitlab CICD pipeline to deploy my terraform changes, it uses Gitlab http backend to store state files. However momentarily it fails to save state files with below errors.

The errors occurred after the resources have been created, so re-running the same job sometimes resolves the issue but duplicates the resources.

I am not exactly sure about the cause of this error. Would appericiate help in this regard. TIA!

gitlab-ci.yml:

variables:
  TF_ROOT: ${CI_PROJECT_DIR}
  TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}
  TF_VAR_app_key: ${APP_KEY}
  TF_VAR_api_key: ${API_KEY}
  TF_VAR_gitlab_token: ${CI_JOB_TOKEN}
  TF_VAR_environment: ${ENV}
  TF_CLI_CONFIG_FILE: $CI_PROJECT_DIR/.terraformrc

default:
  image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest

cache:
  key: ${CI_PROJECT_NAME}
  paths:
    - ${TF_ROOT}/.terraform


before_script:
  - echo -e "credentials \"$CI_SERVER_HOST\" {\n  token = \"$CI_JOB_TOKEN\"\n}" > $TF_CLI_CONFIG_FILE
  - cd ${TF_ROOT}

stages:
  - prepare
  - validate
  - build
  - deploy

init:
  stage: prepare
  script:
    - gitlab-terraform -v
    - gitlab-terraform init

validate:
  stage: validate
  script:
    - gitlab-terraform validate

plan:
  stage: build
  script:
    - gitlab-terraform plan
    - gitlab-terraform plan-json
  artifacts:
    name: plan
    paths:
      - ${TF_ROOT}/plan.cache
    reports:
      terraform: ${TF_ROOT}/plan.json
  rules:
    - if: $CI_COMMIT_REF_NAME == "main" 
      when: on_success

apply:
  stage: deploy
  environment:
    name: $TF_VAR_environment
  script:
    - gitlab-terraform apply --auto-approve
  dependencies:
    - plan
  artifacts:
    name: errored_tfstate
    paths:
      - ${TF_ROOT}/errored.tfstate
    when: on_failure
  rules:
    - if: $CI_COMMIT_REF_NAME == "main" 
      when: manual
      allow_failure: false

Job output:

...
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
2021/11/07 13:14:21 [DEBUG] POST https://gitlab.operations.mcorg.com/api/v4/projects/16/terraform/state/resources/lock
2021/11/07 13:14:21 [DEBUG] GET https://gitlab.operations.mcorg.com/api/v4/projects/16/terraform/state/resources
module....: Creating...
...
...
2021/11/07 13:14:23 [DEBUG] GET https://gitlab.operations.mcorg.com/api/v4/projects/16/terraform/state/resources
2021/11/07 13:14:23 [DEBUG] POST https://gitlab.operations.mcorg.com/api/v4/projects/16/terraform/state/resources?ID=cc520c4e-21c1-b787-46bf-5b577a9bf594
2021/11/07 13:14:23 [DEBUG] POST https://gitlab.operations.mcorg.com/api/v4/projects/16/terraform/state/resources?ID=cc520c4e-21c1-b787-46bf-5b577a9bf594 (status: 500): retrying in 5s (2 left)
2021/11/07 13:14:29 [DEBUG] POST https://gitlab.operations.mcorg.com/api/v4/projects/16/terraform/state/resources?ID=cc520c4e-21c1-b787-46bf-5b577a9bf594 (status: 500): retrying in 10s (1 left)
╷
│ Error: Failed to save state
│ 
│ Error saving state: Failed to upload state: POST
│ https://gitlab.operations.mcorg.com/api/v4/projects/16/terraform/state/resources?ID=cc520c4e-21c1-b787-46bf-5b577a9bf594
│ giving up after 3 attempts
╵
╷
│ Error: Failed to persist state to backend
│ 
│ The error shown above has prevented Terraform from writing the updated
│ state to the configured backend. To allow for recovery, the state has been
│ written to the file "errored.tfstate" in the current working directory.
│ 
│ Running "terraform apply" again at this point will create a forked state,
│ making it harder to recover.
│ 
│ To retry writing this state, use the following command:
│     terraform state push errored.tfstate
│ 
╵
2021/11/07 13:14:40 [DEBUG] DELETE https://gitlab.operations.mcorg.com/api/v4/projects/16/terraform/state/resources/lock
...
Uploading artifacts as "archive" to coordinator... ok  id=248 responseStatus=201 Created token=****
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1