How can I know it is safe to call vkFreeMemory?

I'm trying to update a model in my Vulkan rendered scene (with a new mesh that has a different number of nodes, for example), and want to clear the data held in buffers, like so (inside a Model class, where device and vertices are members, device is a VkDevice and vertices.memory is a VkDeviceMemory):

            if (vertices.buffer)
            {
                vkDestroyBuffer(device, vertices.buffer, nullptr);
                vkFreeMemory(device, vertices.memory, nullptr);
                vertices.buffer = VK_NULL_HANDLE;
                vertices.memory = VK_NULL_HANDLE;
            }

But will get occasional errors like this, telling me that the memory is in use by a command buffer:

ERROR: [1214025378][VUID-vkFreeMemory-memory-00677] : Validation Error: [ VUID-vkFreeMemory-memory-00677 ] Object 0: handle = 0x23b81794958, type = VK_OBJECT_TYPE_DEVICE; | MessageID = 0x485c8ea2 | Cannot call vkFreeMemory on VkDeviceMemory 0x7e08100000002e4f[] that is currently in use by a command buffer. The Vulkan spec states: All submitted commands that refer to memory (via images or buffers) must have completed execution (https://vulkan.lunarg.com/doc/view/1.2.154.1/windows/1.2-extensions/vkspec.html#VUID-vkFreeMemory-memory-00677)

And eventually crash. My draw call uses a fence to wait for the queue to be completed - but I am clearly missing another synchronisation step before I try to vkFreeMemory. How can I make this vkFreeMemory safe(r)?