OpenSSL's documentation is frustratingly terse, and I can't find a definitive answer to this question there. However, I have used OpenSSL myself in C++ programs, and in my experience, it does not appear to be necessary to use OPENSSL_malloc() for byte buffers used as unsigned char* parameters to an OpenSSL function. If the function says that it will use the unsigned char* parameter to read or write binary data from memory, I have found that you can safely use the pointer from std::vector<unsigned_char>::data() for that parameter. This also applies to void* input parameters. For example, EVP_DigestUpdate expects a parameter void* d from which it will read bytes and hash them, and you can use it to hash bytes from a std::vector like this:
std::vector<uint8_t> my_bytes; // Fill this with the data you want to hash
EVP_DigestUpdate(context, my_bytes.data(), my_bytes.size());
(note that context is an EVP_MD_CTX*, which needs to be initialized previously).
Similarly, EVP_DigestFinal_ex expects a parameter unsigned char* md in which it will write the message digest (hash). You can use a std::vector<unsigned char> instead, as long as you make sure to allocate a large enough vector for the hash you're using. The documentation suggests using EVP_MD_CTX_get_size() to find out the size of the buffer you need for a message digest, so I did this:
int digest_size = EVP_MD_CTX_get_size(context);
std::vector<unsigned char> hash(digest_size);
EVP_DigestFinal_ex(context, hash.get(), NULL);
I have run my programs through valgrind after using various OpenSSL EVP functions in this manner and no memory leaks or errors were detected, so I'm fairly confident that using memory from a C++ vector is not introducing subtle bugs into OpenSSL.
For the sake of completeness, I can also address your comment that "you cannot use other modern C++ functions to allocate" objects from OpenSSL that have their own special new and free functions. In fact, you can manage these OpenSSL objects using std::unique_ptr as long as you specify a custom deleter that calls the special OpenSSL free function. Just wrap the pointer returned from the TYPE_new() function in a unique_ptr, and make the custom deleter call the corresponding TYPE_free() function on that pointer.
Putting it together, here's a complete example of using EVP_Digest functions with only C++-managed memory:
template <typename OpenSSLType>
struct DeleterFor;
template <>
struct DeleterFor<EVP_MD_CTX> {
void operator()(EVP_MD_CTX* p) { EVP_MD_CTX_free(p); }
};
std::vector<uint8_t> hash_bytes(const std::vector<uint8_t>& input_buffer) {
std::unique_ptr<EVP_MD_CTX, DeleterFor<EVP_MD_CTX>> digest_context(EVP_MD_CTX_new());
if(EVP_MD_CTX_reset(digest_context.get()) != 1) {
throw std::runtime_error("Error in EVP_MT_CTX_reset");
}
if(EVP_DigestInit_ex(digest_context.get(), EVP_sha256(), NULL) != 1) {
throw std::runtime_error("Error in EVP_DigestInit_ex");
}
if(EVP_DigestUpdate(digest_context.get(), input_buffer.data(), input_buffer.size()) != 1) {
throw std::runtime_error("Error in EVP_DigestUpdate");
}
int digest_size = EVP_MD_CTX_get_size(digest_context.get());
std::vector<unsigned char> hash(digest_size);
if(EVP_DigestFinal_ex(digest_context.get(), hash.data(), NULL) != 1) {
throw std::runtime_error("Error in EVP_DigestFinal_ex");
}
return hash;
}