Gitlab CI: Failed building wheel for opencv-python

user_17083118 0
Viewed 523

I am working on CI/CD for my python/django project in gitlab.

I have an error -- Gitlab CI: Failed building wheel for opencv-python

Full gitlab ci log -- https://pastebin.com/pZdZ6ws2

I have an error on the build_pip stage: gitlab-ci.yaml

stages:
  - linter
  - build_pip
  - build
  - meta
  - code_quality
  - deploy

.except-tags:
  except:
    - tags

build_pip:build_dist:
  stage: build_pip
  # image: $CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/python:3.9-alpine
  image: python:3.9-alpine
  before_script:
    - apk update && apk add postgresql-dev gcc python3-dev musl-dev g++ jpeg-dev zlib-dev
    - pip install pip --upgrade
    - pip install -r requirements/production.txt --no-cache
  script:
    - python setup.py bdist_wheel
    - echo PIP_CI_JOB_ID=$CI_JOB_ID > PIP_CI_JOB_ID.env
  dependencies: []
  artifacts:
    expire_in: 1 hour
    paths:
      - dist/
      - version
    reports:
      dotenv: PIP_CI_JOB_ID.env
  extends:
    - .except-tags

meta:version:
  stage: meta
  needs:
    - job: build_pip:build_dist
      artifacts: true
  script:
    - cat version
  artifacts:
    expire_in: never
    paths:
      - version
  extends: .except-tags


build:build_api:
  stage: build
  image: registry.ml.bastion-tech.ru:8843/ansible/infrastructure/ansible_tools:2.9
  needs:
    - job: build_pip:build_dist
      artifacts: true
  before_script:
    - ansible-vault decrypt /ansible/infrastructure/secrets/ansible@infrastructure/id_rsa --vault-password-file=${ANSIBLE_VAULT_PASSWORD}
  script:
    - |
      ansible-playbook -i /ansible/infrastructure/inventories/ml.inventory \
      --vault-password-file=${ANSIBLE_VAULT_PASSWORD} \
      --private-key /ansible/infrastructure/secrets/ansible@infrastructure/id_rsa \
      -e ansible_ssh_user=deploy \
      -e smartconstructions_pip_ci_job_id=${PIP_CI_JOB_ID} \
      -e build=true -e smartconstructions_build_ref=${CI_COMMIT_BRANCH} \
      /ansible/infrastructure/ml_smartconstructions.yml
  tags:
    - linux-docker

deploy:deploy_api:
  stage: deploy
  image: registry.ml.bastion-tech.ru:8843/ansible/infrastructure/ansible_tools:2.9
  needs:
    - job: build_pip:build_dist
      artifacts: true
  when: manual
  only:
    - master
    - dev
  before_script:
    - ansible-vault decrypt /ansible/infrastructure/secrets/ansible@infrastructure/id_rsa --vault-password-file=${ANSIBLE_VAULT_PASSWORD}
  script:
    - |
      ansible-playbook -i /ansible/infrastructure/inventories/ml.inventory \
      --vault-password-file=${ANSIBLE_VAULT_PASSWORD} \
      --private-key /ansible/infrastructure/secrets/ansible@infrastructure/id_rsa \
      -e ansible_ssh_user=deploy \
      -e smartconstructions_pip_ci_job_id=${PIP_CI_JOB_ID} \
      -e run=true -e frontend_restart=true \
      /ansible/infrastructure/ml_smartconstructions.yml
  tags:
    - linux-docker

include:
  - local: .gitlab/ci/code-quality.yml

requirements/production.txt

djangorestframework==3.12.4
drf-extra-fields==3.1.1
djangorestframework-camel-case==1.2.0  # https://pypi.org/project/djangorestframework-camel-case/
Pillow==8.3.2
python-dateutil==2.8.2  # datetime formatting
psycopg2==2.9.1
opencv-python==4.5.3.56
drf-yasg==1.20.0
sentry-sdk==1.4.3
gunicorn==20.1.0
requests==2.26.0
yarl==1.7.0
googlemaps==4.5.3
django_redis==5.0.0
celery==5.2.0
channels==3.0.4
channels_redis==3.3.1
gitlab python
2 Answers

In your logs, we can see the following error:

gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/include/python3.9 -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.9/c/_cffi_backend.o
c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
  15 | #include <ffi.h>
    |          ^~~~~~~
compilation terminated.
error: command '/usr/bin/gcc' failed with exit code 1

Errors like those suggests you're missing header files.

In Alpine, the ffi.h file should be part of libffi-dev. Try this:

apk add libffi-dev
user_56072070

Maybe your use case would benefit from using opencv-python-headless? It's basically the same library, maintained by the same people, but with less dependencies to Xorg and GUI packages. This means less errors due to missing packages (i.e. less stuff to install via apk).

You could use the opencv-python in a development.txt file and change production to the headless version.

From the docs:

These packages are smaller than the two other packages above because they do not contain any GUI functionality (not compiled with Qt / other GUI components). This means that the packages avoid a heavy dependency chain to X11 libraries and you will have for example smaller Docker images as a result. You should always use these packages if you do not use cv2.imshow et al. or you are using some other package (such as PyQt) than OpenCV to create your GUI.

Option 3 - Headless main modules package: pip install opencv-python-headless

Option 4 - Headless full package (contains both main modules and contrib/extra modules): pip install opencv-contrib-python-headless (check contrib/extra modules listing from OpenCV documentation)

Other than this, is it really a good idea using an Alpine-based image + Python + Scientific tooling? It wouldn't surprise me if you'll get way more issues in runtime even this image manages to build, given how tangled some Python tooling is to GCC et al.

Try using 3.9-slim-bullseye, it's a debian-based image.

user_46148400
Related