Convert from Direct3D11CaptureFrame to OpenCV's cv::Mat

Question

I'm trying to convert from a Direct3D11CaptureFrame, obtained as in the example from Microsoft's Screen Capture documentation (and verified to be correct by saving it on disk), to a cv::Mat. I am converting the frame's surface into a ID3D11Texture2D in the following way, and it seems to work:

void processFrame(const winrt::Windows::Graphics::Capture::Direct3D11CaptureFrame &frame) {
       winrt::com_ptr access =
          frame.Surface().as();
      winrt::com_ptr frameSurface;
      winrt::check_hresult(access->GetInterface(winrt::guid_of(), frameSurface.put_void()));
      (...)
}

From this point, looking at opencv's d3d interop samples, I see I have two options. The first one consists in using cv::directx::convertFromD3D11Texture2D such as

void processFrame(const winrt::Windows::Graphics::Capture::Direct3D11CaptureFrame &frame) {
       winrt::com_ptr access =
          frame.Surface().as();
      winrt::com_ptr frameSurface;
      winrt::check_hresult(access->GetInterface(winrt::guid_of(), frameSurface.put_void()));

  cv::Mat img;
  // throws exception in the following line
  //
  cv::directx::convertFromD3D11Texture2D(frameSurface.get(), img);
}

But it throws an exception after calling the destructor of the input array (frameSurface.get()). This method is used for GPU work in the code examples, but I haven't seen it stated as exclusively for GPU in the documentation.

The second option goes through a couple more hoops:

void processFrame(const winrt::Windows::Graphics::Capture::Direct3D11CaptureFrame &frame, int height, int width) {
  auto access =
      frame.Surface().as();
  winrt::com_ptr frameSurface;
  winrt::check_hresult(
      access->GetInterface(winrt::guid_of(), frameSurface.put_void()));

  ID3D11Device *deviceD3D;
  frameSurface->GetDevice(&deviceD3D);
  ID3D11DeviceContext *m_pD3D11Ctx;
  deviceD3D->GetImmediateContext(&m_pD3D11Ctx);

  auto subResource = ::D3D11CalcSubresource(0, 0, 1);
  D3D11_MAPPED_SUBRESOURCE mappedTex;
  auto r = m_pD3D11Ctx->Map(frameSurface.get(), subResource, D3D11_MAP_WRITE_DISCARD, 0, &mappedTex);
  // FAILS here
  //
  if (FAILED(r)) {
    throw std::runtime_error("surface mapping failed!");
  }
  cv::Mat m(height, width, CV_8UC4, mappedTex.pData, mappedTex.RowPitch);
}

... but it fails when mapping the context to the subresource.

I have been tweaking everything that I could think of given my inexperience with DirectX and trying to see what I am missing, but I can't find a solution for any of the two options. Does anyone see my mistakes or has a better solution for my problem?

user_19224495 · Answer

1.you need new D3D11_TEXTURE2D_DESC and set: CPUAccessFlags = D3D11_CPU_ACCESS_READ

2.use that D3D11_TEXTURE2D_DESC to create a new CreateTexture2D

3.copy frameSurface to your new CreateTexture2D

void SimpleCapture::OnFrameArrived(
Direct3D11CaptureFramePool const& sender,
winrt::Windows::Foundation::IInspectable const&) {
    auto newSize = false;
    auto frame = sender.TryGetNextFrame();
    auto frameContentSize = frame.ContentSize();

    D3D11_TEXTURE2D_DESC desc;
    desc.MipLevels = 1;
    desc.ArraySize = 1;
    desc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
    desc.SampleDesc = { 1,0 };
    desc.Usage = D3D11_USAGE_STAGING;
    desc.BindFlags = 0;
    desc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
    desc.MiscFlags = 0;
    
    if (frameContentSize.Width != m_lastSize.Width ||
        frameContentSize.Height != m_lastSize.Height) {
        newSize = true;
        m_lastSize = frameContentSize;
    }
    desc.Width = m_lastSize.Width;
    desc.Height = m_lastSize.Height;

    ID3D11Texture2D* stagingTexture;
    d3dDevice.get()->CreateTexture2D(&desc, nullptr, &stagingTexture);
    
    auto frameSurface = GetDXGIInterfaceFromObject(frame.Surface());
    m_d3dContext->CopyResource(stagingTexture, frameSurface.get());

    D3D11_MAPPED_SUBRESOURCE mappedTex;
    m_d3dContext->Map(stagingTexture, 0, D3D11_MAP_READ, 0, &mappedTex);
    m_d3dContext->Unmap(stagingTexture, 0);

    cv::Mat c = cv::Mat(desc.Height, desc.Width, CV_8UC4, mappedTex.pData, mappedTex.RowPitch);
    cv::imshow("show", c);

    stagingTexture->Release();


    DXGI_PRESENT_PARAMETERS presentParameters = { 0 };

    if (newSize)
    {
        m_framePool.Recreate(
            m_device,
            DirectXPixelFormat::B8G8R8A8UIntNormalized,
            2,
            m_lastSize);
    }
}

Convert from Direct3D11CaptureFrame to OpenCV's cv::Mat

1 Answers