Override iOS BadgeNumber from Firebase

I have Firebase for my iOS app and as it is already clear from Stack Overflow that I cannot have Firebase +1 badge number on iOS for me. Instead I have to do my own calculation from within appdelegate myself.

However, I have put down below codes expecting that it will override whatever badge number sent in through Firebase, but they seem to have no effect.

func userNotificationCenter(_ center: UNUserNotificationCenter, didReceive response: UNNotificationResponse, withCompletionHandler completionHandler: @escaping () -> Void) {
    let userDefault: UserDefaults = UserDefaults()
    if var badgeCount = userDefault.value(forKey: "badgeCount") as? Int {
        badgeCount += 1
        UserDefaults.standard.setValue(badgeCount, forKey: "badgeCount")
        UIApplication.shared.applicationIconBadgeNumber = badgeCount
    }
}

As a further test, I have even tried to hardcode the below line to the function above and see if it will override the badge number sent in by Firebase:

UIApplication.shared.applicationIconBadgeNumber = 48

And it turns out it will not override the badge number sent through Firebase at all, the number '88' I sent through Firebase will become the new badge number upon receiving the push notification and it will not be overridden with '48'.

Do anyone have any clues what I have done wrong?

Thanks!