Chromecast - Unable to control casted video on tv using tv remote

I am using google cast SDK for casting video from our android app to chromecast enable android tv. Using default media receiver on chromecast side, I can able to play, pause, seek video from our app using RemoteMediaClient. but when I try to play, pause, seek casted video using android tv's remote, I am unable to seek.