job fail in gitlab if multiple jobs

My pipeline has two stages: build and test.

The build has 2 jobs while the test has 4 jobs (3 related to one build and 1 related to the other build)

If I run the pipeline by commenting out the build and test related to one task, then the pipeline passes but if I am running both tasks together then it fails. I am not sure why the two tasks cannot be tested together.

Following is the YAML file I have:

stages:
  - build
  - test
  
build-driver-only:
  stage: build
  artifacts:
    paths:
      - components/nic/linux/driver/
    expire_in: 1h
  script:
    - echo "Build driver-only initiated"
    - chmod +x CI_testing/build_driver.sh
    - bash ./CI_testing/build_driver.sh DRIVER_ONLY
    - wait

build-driver-vfpf:
  stage: build
  artifacts:
    paths:
      - components/nic/linux/driver/
    expire_in: 1h
  script:
    - echo "Build driver-VFPF initiated"
    - chmod +x CI_testing/build_driver.sh
    - bash ./CI_testing/build_driver.sh VFPF
    - wait

test-drv-ping:
  stage: test
  needs:
    - job: build-driver-only
      artifacts: true
  script:
    - echo "Ping test for driver-only initiated"
    - chmod +x ./CI_testing/test_driver_only.sh
    - bash ./CI_testing/test_driver_only.sh PING
    - wait

test-drv-ping-ns:
  stage: test
  needs:
    - job: build-driver-only
      artifacts: true
  script:
    - echo "Ping with namespace test for driver-only initiated"
    - chmod +x ./CI_testing/test_driver_only.sh
    - bash ./CI_testing/test_driver_only.sh PING_NS
    - wait

test-drv-iperf3:
  stage: test
  needs:
    - job: build-driver-only
      artifacts: true
  script:
    - echo "iperf3 test for driver-only initiated"
    - chmod +x ./CI_testing/test_driver_only.sh
    - bash ./CI_testing/test_driver_only.sh IPERF3
    - wait

test-drv-vfpf:
  stage: test
  needs:
    - job: build-driver-vfpf
      artifacts: true
  script:
    - echo "iperf3 test for driver-only initiated"
    - chmod +x ./CI_testing/test_driver_only.sh
    - bash ./CI_testing/test_driver_only.sh VFPF
    - wait

The pipeline is failing if I attach the second build stage with the first one. I have tested everything manually and it works fine. Any idea, on how I can further isolate both tasks?