I'm writing some basic neural network methods - specifically the activation functions - and have hit the limits of my rubbish knowledge of math. I understand the respective ranges (-1/1) (0/1) etc, but the varying descriptions and implementations have me confused.
Specifically sigmoid, logistic, bipolar sigmoid, tanh, etc.
Does sigmoid simply describe the shape of the function irrespective of range? If so, then is tanh a 'sigmoid function'?
I have seen 'bipolar sigmoid' compared against 'tanh' in a paper, however I have seen both functions implemented (in various libraries) with the same code:
(( 2/ (1 + Exp(-2 * n))) - 1). Are they exactly the same thing?
Likewise, I have seen logistic and sigmoid activations implemented with the same code:
( 1/ (1 + Exp(-1 * n))). Are these also equivalent?
Lastly, does it even matter that much in practise? I see on wiki a plot of very similar sigmoid functions - could any of these be used? Some look like they may be considerably faster to compute than others.
