What is the batchSize in TensorFlow's model.fit() function?

The batch size is the number of training examples that you use to perform one step of stochastic gradient descent (SGD).

What is SGD? SGD is gradient descent (GD), but, rather than using all of your training data to compute the gradient of your loss function with respect to the parameters of the network, you only use a subset of the training dataset. Hence the adjective "stochastic", because, by using only a subset of the training data, you will be approximating stochastically (i.e. you will introduce noise) the gradient that would be computed by using all of your training data, which would be considered the "actual" gradient of the loss function with respect to the parameters.

Why should I change this number? I have realized that if I increase it, training gets faster, and if I decrease it, it gets slower. But what exactly am I changing here? Why would I change it? What do I need to watch out for?

If the batch size is too small, e.g. 1, then you will be computing the gradient only with one training example. This can make your training loss to oscillate a lot, because, each time, you approximate the gradient with only one training example, which is often not representative of the whole training data. So, as a rule of thumb, the more training examples you use, the better you approximate the gradient (that would correspond to all training examples), so this can potentially lead to faster convergence. However, in practice, if you use many training examples, it can also be computationally expensive. For example, imagine your training data is composed of millions of training examples. In that case, to perform a single step of gradient descent, you would need to go through all these training examples, which can take a lot of time. So, you would potentially need to wait a lot of time only to see how the parameters of your model are updated. This may not be ideal.

To conclude, small batch sizes can make your training process oscillate, and this can make your loss function take a lot of time to reach a local minimum. However, a huge batch size may also not be desirable, because it can also take a lot of time.

Typical values of the batch size are 32, 64, and 128. Why? People just use these numbers because they empirically seem to be good compromises (in terms of convergence, training time, etc.) between tiny batch sizes and huge batch sizes.

It is really rather simple. Assume for example you have a training set of 50,000 samples and associated labels. In classical theory you would feed your model all 50,000 inputs (batch size=50,000) then adjust the network weights via back propagation. So for 50,000 samples you get only 1 iteration of the network weights ( 1 epoch). There are two problems with this. One is the time it takes to train the network. Lets say to get a high level of accuracy it takes 50 iterations(Epochs). In that case you have to feed your network 50 X 50,000 samples. This would take a significant amount of processing time. Second with a batch size of 50,000 all 50,000 samples reside in memory.If you are working with images for example that would take up an enormous amount of memory and probably lead to a resource exhaust error. Now lets take a different strategy. Divide your training set into groups of 1000 samples, So you will have 50 groups(batches). Now you feed your network the first batch of 1000 samples, then adjust the weights via back propagation (Epoch1). Then do this again for the next batch of 1000 samples and again do back propagation (Epoch2). Repeat this for all 50 batches. At the end you have feed the network 50,000 samples and you have adjusted the weights 50 times. Result is you train much faster and you only have 1000 samples resident in memory if you are fetching the samples in batches. There is a trade off here. For example if you set a batch size=1 training is going to take a very long time because your will do back propagation 50,000 times. So it is best to pick a moderate batch size. I generally select it to be between 30 to 80 which seems to work well even with largers images like 254 X 254 X 3 if you have 16G of GPU memory.