Android asks for USB access permission twice

I have an application designed to operate the USB serial device. The Manifest contains the corresponding filter:

<intent-filter>
    <action android:name="android.hardware.usb.action.USB_DEVICE_ATTACHED" />
</intent-filter>
<meta-data
    android:name="android.hardware.usb.action.USB_DEVICE_ATTACHED"
    android:resource="@xml/device_filter" />

When I plug the USB device, the system pop-up dialog asks "Open ... when this USB device is connected?". I can confirm or decline, and this part works fine. However, if I decline and then restart the application from the menu, my phone hangs out. In the stack trace, I see the error:

User has not given permission to device UsbDevice...

Therefore, I have to ask for the USB permission explicitly. I have tried several implementations and the simplest one looks like this (I use the usb_serial_for_android library: https://github.com/mik3y/usb-serial-for-android):

@Override
protected void onResume() {
....
    UsbManager manager = (UsbManager) getSystemService(Context.USB_SERVICE);
    ...
    List<UsbSerialDriver> availableDrivers = CustomProber.getCustomProber().findAllDrivers(manager);
    ...
    UsbSerialDriver driver = availableDrivers.get(0);
    if (!manager.hasPermission(driver.getDevice())) {
        manager.requestPermission(
            driver.getDevice(),
            PendingIntent.getBroadcast(this, 0, new Intent("com.android.example.USB_PERMISSION"), 0));
    return;
    }
....
}

The problem is that "manager.requestPermission" executes twice.

When I start the application from the menu, the pop-up dialog asks "Allow the app ... to access the USB device?" If I confirm, the same dialog appears again, but my reply doesn't matter. It seems that the permission was already granted, as if I decline the second request, the application continues to work and successfully communicate with the device. It's a rather annoying behavior :(

What do I miss?

P.S. I have also tried a "canonical" way described by Android Developers (https://developer.android.com/guide/topics/connectivity/usb/host). However, this BroadcastReceiver based implementation behaves in the same way. I have read the related posts, like this one Android asks for USB permission twice and communicated with the author. The problem is still actual.

UPDATE

Here is the Log:

...
...
I/storage permission: DENIED
V/PhoneWindow: DecorView setVisiblity: visibility = 4, Parent = null, this = DecorView@8856cab[]
D/WindowClient: Add to mViews: DecorView@8856cab[MainActivity], this = android.view.WindowManagerGlobal@5653205
...

The pop-up on storage access appeared. I pressed "allow"

I/storage permission: GRANTED
I/directory: EXISTS
I/usb permission: DENIED
V/PhoneWindow: DecorView setVisiblity: visibility = 0, Parent = ViewRoot{632025a com.ERG.erglogger/com.ERG.erglogger.MainActivity,ident = 0}, this = DecorView@8856cab[MainActivity]

The first pop-up on USB access appeared. I pressed "OK".

I/storage permission: GRANTED
I/directory: EXISTS
I/usb permission: DENIED
V/PhoneWindow: DecorView setVisiblity: visibility = 0, Parent = ViewRoot{632025a com.ERG.erglogger/com.ERG.erglogger.MainActivity,ident = 0}, this = DecorView@8856cab[MainActivity]

I pressed "OK", but the permission was not granted (manager.hasPermission(driver.getDevice()) is false). So, the second pop-up on USB access appeared, and after I pressed "OK", the permission was granted:

I/storage permission: GRANTED
I/directory: EXISTS
I/usb permission: GRANTED
D/CdcAcmSerialDriver: trying default interface logic
...