What is the meaning of a 'mini-batch' in deep learning?

Both are approaches to gradient descent. But in a batch gradient descent you process the entire training set in one iteration. Whereas, in a mini-batch gradient descent you process a small subset of the training set in each iteration.

Also compare stochastic gradient descent, where you process a single example from the training set in each iteration.

Another way to look at it: they are all examples of the same approach to gradient descent with a batch size of m and a training set of size n. For stochastic gradient descent, m=1. For batch gradient descent, m = n. For mini-batch, m=b and b < n, typically b is small compared to n.

Mini-batch adds the question of determining the right size for b, but finding the right b may greatly improve your results.

Excellent Explanation by @majid ghafouri but I just want to add more details to make sure you got this and why we are using it or which advantages can we gain using it:

Stochastic Gradient Descent performs updates according to the following iterative process. This type of learning, which performs updates a single example at a time is called online learning.

The Algorithm for it would looks like this:

procedure Online
  for several epochs of training do
     for each training example in the data do
         Calculate gradients of the loss
         Update the parameters according to this gradient
     end for
  end for
end procedure

In contrast, we can also think of a batch learning algorithm, which treats the entire data set as a single unit, calculates the gradients for this unit, then only performs update after making a full pass through the data. These two update strategies have trade-offs.

• Online training algorithms usually find a relatively good solution more quickly, as they don’t need to make a full pass through the data before performing an update.

• However, at the end of training, batch learning algorithms can be more stable, as they are not overly influenced by the most recently seen training examples.

The Algorithm for Batch would looks like this:

procedure Batch
  for several epochs of training do
    for each training example in the data do
        Calculate and accumulate gradients of the loss
    end for
    Update the parameters according to the accumulated gradient
  end for
end procedure

• Batch training algorithms are also more prone to falling into local optima; the randomness in online training algorithms often allows them to bounce out of local optima and find a better global solution.

Minibatching is a happy medium between these two strategies. Basically, minibatched training is similar to online training, but instead of processing a single training example at a time, we calculate the gradient for n training examples at a time. In the extreme case of n = 1, this is equivalent to standard online training, and in the other extreme where n equals the size of the data, this is equivalent to fully batched training.

As we increase the number of training examples, each parameter update becomes more informative and stable, but the amount of time to perform one update increases, so it is common to choose an n that allows for a good balance between the two. One other major advantage of minibatching is that by using a few tricks, it is actually possible to make the simultaneous processing of n training examples significantly faster than processing n different examples separately. Specifically, by taking multiple training examples and grouping similar operations together to be processed simultaneously, we can realize large gains in computational efficiency due to the fact that modern hardware (particularly GPUs, but also CPUs) have very efficient vector processing instructions that can be exploited with appropriately structured inputs.

the Explanation is taken from this Excellent paper, you can read further if you have time: