Can we use bert on an image with embeddings created with the help of CNN architecture?

I am trying to use bert for the images, the following steps I'm considering to do this approach:

1. Create an embedding of an image using VggNet (extracting avgpool layer from the network).
2. Using PCA for dimensionality reduction from 4096 to 768 on vector matrix (embeddings we’ve got from VggNet).
3. As this is a sequence of integers, passing it to the transformer encoder, bert.

Does it seems like a sensible thing to do?