I was going though Ian's 2015 paper and came across this idea that the LSTM are linear in nature. In the same paragraph he mentions sigmoid are non-linear (which is true). As far as I know they contain sigmoid and tanh in its gates which helps in non linear mapping between input, hidden and cell state.