There is a way to use AlertDialog inside BottomSheetDialogFragment?

Instead of using AlertDialog inside of bottomsheetdialogfragment. Create one bottomsheetdialog which behaves like your requirement.

Please refer https://medium.com/glucosio-project/moving-from-dialogs-to-bottomsheetdialogs-on-android-15fb8d140295 How to use BottomSheetDialog?

the best way is to use BottomSheetDialogFragment as you said and to set it a custom view that you wants with only title, msg and buttons

BottomSheetDialog dialog = new BottomSheetDialog(YourActivity.this);
dialog.setContentView(YourView);

dialog.show();

The AlertDialog and the BottomSheetDialog extend AppCompatDialog both but have different implementation.
Since the layout is simple (just title, message and buttons), it is easier to use a BottomSheetDialog with a custom layout, rather than to use a AlertDialog and adapt all the behaviour and animations of the BottomSheet.

Just use the BottomSheetDialogFragment (which creates a BottomSheetDialog):

public class MyBottomSheetDialog extends BottomSheetDialogFragment {

  @Nullable @Override
  public View onCreateView(@NonNull LayoutInflater inflater, @Nullable ViewGroup container,
      @Nullable Bundle savedInstanceState) {
    super.onCreateView(inflater, container, savedInstanceState);
    //Use your custom layout
    View view = inflater.inflate(R.layout.yourLayout, container, false);
    return view;
  }    
}

And then

MyBottomSheetDialog myBottomSheetDialog = new MyBottomSheetDialog();
myBottomSheetDialog.show(getSupportFragmentManager(), "TAG");

I answer to my question: Simply no, you're forced to use a CustomView