System dialog to change default dialer does not appear second time using RoleManager API

I have 2 options in app:

1. to set my app as default dialer app.
2. to remove my app from default dialer & set phone app as default dialer app.

Now issue is:

1. When clicked on SET option, intent is fired as below :

//For setting dialer 
boolean isRoleAvailable = roleManager.isRoleAvailable(RoleManager.ROLE_DIALER);
            if (isRoleAvailable){
                // check whether your app is already holding the default Dialer app role.
                boolean isRoleHeld = roleManager.isRoleHeld(RoleManager.ROLE_DIALER);
                if (!isRoleHeld){
                    Intent roleRequestIntent = roleManager.createRequestRoleIntent(RoleManager.ROLE_DIALER);
                    startActivityForResult(roleRequestIntent, REQUEST_CODE_SET_DEFAULT_DIALER);
                }
            }

Using this system dialog appears properly asking for changing default dialer app.

2. But when i want to remove my app from default dialer & instead set phone app as default dialer then 2nd time system dialog does not appear.

//For resetting
if (android.os.Build.VERSION.SDK_INT >= android.os.Build.VERSION_CODES.Q) {

               // check if the app is having permission to be as default Dialer app
               // check whether your app is already holding the default Dialer app role.
               boolean isRoleHeld = roleManager.isRoleHeld(RoleManager.ROLE_DIALER);
               if (isRoleHeld){
                       Intent roleRequestIntent = roleManager.createRequestRoleIntent(RoleManager.ROLE_DIALER);
                       startActivityForResult(roleRequestIntent, REQUEST_CODE_RESET_DEFAULT_PHONE_APP);
                    }
}