When should we use mutex and when should we use semaphore

All the above answers are of good quality,but this one's just to memorize.The name Mutex is derived from Mutually Exclusive hence you are motivated to think of a mutex lock as Mutual Exclusion between two as in only one at a time,and if I possessed it you can have it only after I release it.On the other hand such case doesn't exist for Semaphore is just like a traffic signal(which the word Semaphore also means).

I find the answer of @Peer Stritzinger the correct one.

I wanted to add to his answer the following quote from the book Programming with POSIX Threads by David R Butenhof. On page 52 of chapter 3 the author writes (emphasis mine):

You cannot lock a mutex when the calling thread already has that mutex locked. The result of attempting to do so may be an error return (EDEADLK), or it may be a self-deadlock, where the unfortunate thread waits forever. You cannot unlock a mutex that is unlocked, or that is locked by another thread. Locked mutexes are owned by the thread that locks them. If you need an "unowned" lock, use a semaphore. Section 6.6.6 discusses semaphores)

With this in mind, the following piece of code illustrates the danger of using a semaphore of size 1 as a replacement for a mutex.

sem = Semaphore(1)
counter = 0 // shared variable
----

Thread 1

for (i in 1..100):
  sem.lock()
  ++counter
  sem.unlock()
----

Thread 2

for (i in 1..100):
  sem.lock()
  ++counter
  sem.unlock()
----

Thread 3

sem.unlock()
thread.sleep(1.sec)
sem.lock()

If only for threads 1 and 2, the final value of counter should be 200. However, if by mistake that semaphore reference was leaked to another thread and called unlock, than you wouldn't get mutual exclusion. With a mutex, this behaviour would be impossible by definition.

Binary semaphore and Mutex are different. From OS perspective, a binary semaphore and counting semaphore are implemented in the same way and a binary semaphore can have a value 0 or 1.

Mutex -> Can only be used for one and only purpose of mutual exclusion for a critical section of code.

Semaphore -> Can be used to solve variety of problems. A binary semaphore can be used for signalling and also solve mutual exclusion problem. When initialized to 0, it solves signalling problem and when initialized to 1, it solves mutual exclusion problem.

When the number of resources are more and needs to be synchronized, we can use counting semaphore.

In my blog, I have discussed these topics in detail.

https://designpatterns-oo-cplusplus.blogspot.com/2015/07/synchronization-primitives-mutex-and.html