I have trained ResNext50 for activity recognition. The original architecture of network includes ReLU. The test accuracy I achieved with original architecture is %85. When I replace all ReLU activation functions with Mish, accuracy is going down dramatically to %71. By the way LeakyReLU shows similar test accuracy with ReLU.
I want to know that is there a way make this activation function replacement better to achieve high accuracy ? For example: I saw an example that using both ReLU and Mish in the same network. But I don't know how to bring them together carefully.