I noticed that TensorFlow allocates a large amount of main memory (700MB+) when running certain operations in eager mode, even on tiny tensors (i.e. with shapes as small as [5, 5]). For example:
import tensorflow as tf # Total memory usage is ~141MB
fst, snd = tf.random.uniform([5, 5]), tf.random.uniform([5, 5]) # ...grows to ~697MB
tf.matmul(fst, snd) # ...and then to ~1349MB
tf.concat([fst, snd], 0) # Total memory use remains ~1349MB
This is running on TensorFlow 2.6.0-rc1, on an RTX 3070 on Windows 10. I observed the same behaviour when enabling memory growth with tf.config.experimental.set_memory_growth.
Why do the calls to tf.matmul and tf.random.uniform result in almost 700MB being allocated each time, whilst the subsequent tf.concat results in no change in the memory use?