Switching optimization profiles in TensorRT: bindingIndex 0 is not in profile 1

user_4943329 0
Viewed 663

I know this question should really be directed at the NVIDIA support team, however they are incredibly unhelpful in their support (just have a look at this thread here, instead of actually reading the question, they just point you to the generic API docs page, which trust me has no helpful info on it). Rant aside, here's my question:

I am trying to run batch inference with TensorRT while specifying optimization profiles for additional performance.

My network has input shape (-1, 3, 112, 112) meaning it supports dynamic batch sizes. I am registering several optimization profiles as follows. You can assume that m_options.optBatchSizes = std::vector<unsigned int>{2, 4, 8};

// Specify the default optimization profile
    IOptimizationProfile* defaultProfile = builder->createOptimizationProfile();
    defaultProfile->setDimensions(inputName, OptProfileSelector::kMIN, Dims4(1, m_inputC, m_inputH, m_inputW));
    defaultProfile->setDimensions(inputName, OptProfileSelector::kOPT, Dims4(1, m_inputC, m_inputH, m_inputW));
    defaultProfile->setDimensions(inputName, OptProfileSelector::kMAX, Dims4(m_options.maxBatchSize, m_inputC, m_inputH, m_inputW));
    config->addOptimizationProfile(defaultProfile);

    // Specify all the optimization profiles.
    for (const auto& optBatchSize: m_options.optBatchSizes) {
        if (optBatchSize == 1) {
            continue;
        }

        if (optBatchSize > m_options.maxBatchSize) {
            throw std::runtime_error("optBatchSize cannot be greater than maxBatchSize!");
        }

        IOptimizationProfile* profile = builder->createOptimizationProfile();
        profile->setDimensions(inputName, OptProfileSelector::kMIN, Dims4(1, m_inputC, m_inputH, m_inputW));
        profile->setDimensions(inputName, OptProfileSelector::kOPT, Dims4(optBatchSize, m_inputC, m_inputH, m_inputW));
        profile->setDimensions(inputName, OptProfileSelector::kMAX, Dims4(m_options.maxBatchSize, m_inputC, m_inputH, m_inputW));
        config->addOptimizationProfile(profile);
    }

Later in my code when I am running inference, I am trying to switch the optimization profile based on the batch size. If the batch size matches one of the registered profiles, then I want to switch the profile. You can assume that m_optProfIndx is a std::unordered_map<int, int> mapping the batch size to the registered profileIndex.

        // Determine if the batch size is in our optimization profile
        auto it = m_optProfIndx.find(batchSize);
        if (it != m_optProfIndx.end()) {
//             Switch the optimization profile
            m_context->setOptimizationProfileAsync(it->second, m_cudaStream);
        }
        m_context->setBindingDimensions(0, inputDims);

When I try running my code, it crashed with error message:

IExecutionContext::setBindingDimensions: bindingIndex 0 is not in profile 1. Using bindingIndex = 2 instead.
1: [convolutionRunner.cpp::executeConv::458] Error Code 1: Cudnn (CUDNN_STATUS_EXECUTION_FAILED)
terminate called after throwing an instance of 'std::runtime_error'

It looks like it is crashing on the call to m_context->setBindingDimensions(0, inputDims);. For reference, inputDims is defined as follows: Dims4 inputDims = {static_cast<int32_t>(inputFaceChips.size()), m_inputC, m_inputH, m_inputW};, where inputFaceChips.size() is the current batch size.

tensorrt c++
1 Answers

The relevant part of the documentation is here:

In an engine built from multiple profiles, there are separate binding indices for each profile.

In your code snippet you are always attempting to set bindingIndex = 0, instead of querying the proper bindingIndex for the given profile as described in the docs:

Likewise, if using ICudaEngine::getBindingIndex(name) to get the index for a profile K beyond the first profile (K=0), append “[profile K]“ to the name used in the INetworkDefinition. For example, if the tensor was called “foo“ in the INetworkDefinition, then engine.getBindingIndex(“foo [profile 3]“) returns the binding index of Tensor “foo" in optimization profile 3.

This may not solve all of your issues (as I suspect this is a red herring), but will (hopefully) suppress the initial warning and allow for more directed debugging.

user_61622550
Related