OpenCV Docker multistage build - cannot install prebuilt source

I'm trying to build a Docker image including a very particular configuration of OpenCV with CUDA and GPU support.

The build succeeds, and if I make install it from the same context that built the image, it works with no problems.

The problem happens when I try to use a multi stage build, to avoid keeping all the dependencies needed to build OpenCV. Before you continue reading, what follows might actually be an XY problem, if you have a better solution on how to copy OpenCV build artifacts (including Python bindings!) in a Docker multistage build, that is my actual intent.

Now for my attempted solution and the struggle I have:

I run COPY --from=requirements /opencv /opencv and it works and it apparently copies everything in the right path (I checked the filesystem). But, when I run from the build folder make install, I get this CMake error:

CMake Error: The source directory "" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.
Makefile:2724: recipe for target 'cmake_check_build_system' failed
make: *** [cmake_check_build_system] Error 1

Again, the same command, from the same folder, but without multistage build, works.

Here is my Dockerfile:

# Stage 1: Build
FROM nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04 AS requirements
# Install dependencies
RUN echo "deb http://es.archive.ubuntu.com/ubuntu eoan main universe" | tee -a  /etc/apt/sources.list
RUN apt-get update && apt-get -y upgrade
RUN apt-get -y install build-essential cmake unzip pkg-config libjpeg-dev libpng-dev libtiff-dev libavcodec-dev \
                    libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libgtk-3-dev libatlas-base-dev \
                    gfortran python3-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libxvidcore-dev x264 \
                    libx264-dev libfaac-dev libmp3lame-dev libtheora-dev libfaac-dev libmp3lame-dev libvorbis-dev \
                    libjpeg-dev libpng-dev libtiff-dev git python3-pip libtbb-dev libprotobuf-dev protobuf-compiler \
                    libgoogle-glog-dev libgflags-dev libgphoto2-dev libeigen3-dev libhdf5-dev wget libtbb-dev gcc-8 g++-8 llvm \
                    python3-venv libgirepository1.0-dev
# Install my project requirements
WORKDIR /venv
RUN python3 -m venv /venv
ENV PATH="/venv/bin:$PATH"
ADD requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
# Build OpenCV
WORKDIR /opencv
RUN wget https://github.com/opencv/opencv/archive/4.4.0.zip && mv 4.4.0.zip opencv.zip && unzip opencv.zip && rm opencv.zip
RUN wget https://github.com/opencv/opencv_contrib/archive/4.4.0.zip && mv 4.4.0.zip opencv_contrib.zip && unzip opencv_contrib.zip && rm opencv_contrib.zip
WORKDIR /opencv/opencv-4.4.0/build
ENV SITE_PACKAGES /venv/lib/python3.7/site-packages
ENV EXTRA_MODULES /opencv/opencv_contrib-4.4.0/modules
ENV CUDA_ARCH 7.5
ADD docker/build_opencv.sh .
RUN ./build_opencv.sh
# Stage 2: runtime
FROM nvidia/cuda:10.2-cudnn7-runtime-ubuntu18.04
RUN apt-get update && apt-get -y upgrade
RUN apt-get -y install build-essential cmake python3-venv
# Install OpenCV
COPY --from=requirements /opencv /opencv
WORKDIR /opencv/opencv-4.4.0/build
RUN make install && ldconfig
# build fails here and the rest is specific to my project so I've ommitted it

The build_opencv.sh script has this options:

#!/bin/bash
cmake -D CMAKE_BUILD_TYPE=RELEASE \
                -D CMAKE_C_COMPILER=/usr/bin/gcc-8 \
                -D CMAKE_INSTALL_PREFIX=/usr/local \
                -D INSTALL_PYTHON_EXAMPLES=OFF \
                -D INSTALL_C_EXAMPLES=OFF \
                -D WITH_TBB=ON \
                -D WITH_CUDA=ON \
                -D BUILD_opencv_cudacodec=OFF \
                -D ENABLE_FAST_MATH=1 \
                -D CUDA_FAST_MATH=1 \
                -D WITH_CUBLAS=1 \
                -D WITH_V4L=ON \
                -D WITH_QT=OFF \
                -D WITH_OPENGL=ON \
                -D WITH_GSTREAMER=ON \
                -D OPENCV_GENERATE_PKGCONFIG=ON \
                -D OPENCV_PC_FILE_NAME=opencv.pc \
                -D OPENCV_ENABLE_NONFREE=ON \
                -D OPENCV_PYTHON3_INSTALL_PATH=$SITE_PACKAGES \
                -D OPENCV_EXTRA_MODULES_PATH=$EXTRA_MODULES \
                -D PYTHON_EXECUTABLE=/usr/bin/python3 \
                -D WITH_CUDNN=ON \
                -D OPENCV_DNN_CUDA=ON \
                -D CUDA_ARCH_BIN=$CUDA_ARCH \
                -D CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-10.2  \
                -D WITH_GTK_2_X=OFF \
                -D BUILD_EXAMPLES=OFF ..
make -j16

You need at least numpy in your requirements.txt file.

In order to reproduce the issue, a minimal setup would have this structure:

- docker
    - Dockerfile
    - build_opencv.sh
- requirements.txt

Build using from the root of the build context:

docker build -t opencvmultistage:latest -f docker/Dockerfile .

Am I doing something wrong? Maybe CMake has some weird cache that I'm not copying to the new image and makes the build fail?

For the sake of clarity, if I add make install in the build_opencv.sh script it works, but I have OpenCV installed in the build context and not the runtime, which is not what I pretend to do. make install runs in the same directory, and the same files should be present, so I don't really know what's going on.