RecyclerView not calling onScrolled at the bottom

I'm trying to create a RecyclerView with pagination and I have a problem with showing progress bar when I try to scroll down being already at the very end of the list. There's a callback RecyclerView.OnScrollListener which has a method onScrolled for handling scroll events, but it's not working when no actual scrolling has happened.

There's onScrollStateChanged method that works when I try to scroll down from the bottom, but my logic requires me to know direction of the gesture (up/down), so this method is not helpfull in my situation.

I currently trying to do it somewhat like this (which is obviously not working):

recyclerView.addOnScrollListener(new RecyclerView.OnScrollListener() {
        @Override
        public void onScrolled(RecyclerView recyclerView, int dx, int dy) {
            super.onScrolled(recyclerView, dx, dy);
            fetchDataFromServer();
        }
    });

EDIT: I tried to detect if the end is reached in onScrolled and it kinda worked, but sometimes when I try to fetch the data from the server I don't receive anything, so after such a request I'm still at the end of the list and I want to scroll down to try to update the list again, but I can't since onScrolled is not getting called.

Any ideas on how to fix that? Is there another callback that I can use?