Training loss fluctuates and does not go down effectively

I am training the SSD efficientnet B0 on Pascal Voc 2012 dataset from scratch (It needs to be from scratch) with tensorflow object detection API, i have been training for about more than 60k step with batch size of 5 (More than that and i have OOM error), however the loss fluctuates alot and really struggle to drop even abit as can seen in training loss graph, has my training gone wrong or is it the optimizer that i set is not suitable? How would i solve it?

The optimizer configurations is as below:

optimizer {
    adam_optimizer: { 
        learning_rate: {
        manual_step_learning_rate {
            initial_learning_rate: .01
            schedule {
                step: 20000
                learning_rate: .001
            }
            schedule {
                step: 40000
                learning_rate: .0001
            }
            schedule {
                step: 60000
                learning_rate: .00001
        }
    }
}

Please help, thanks.