I need a way to prevent access to files in media items if a user is not authenticated.
If a user is not authenticated he/she should be forced to redirect to the login page. If a user is authenticated then we should let the user to access the media item file, like we should handover the request to Sitecore.
Also help me on changing the configuration settings to override the existing MediaRequestHandler.
I would suggest using the OOTB Sitecore Security feature to grant or deny access to any item including media files.
This way you can restrict access to the specific assets only, not to ALL media files as in case with a bespoke media handler overriding the standard one, and you will not create any technical dependency on your custom code going forward. The latter is seen to be critical from the future website maintenance and Sitecore upgrade perspective.
To begin with, create a secure folder in the Media Library where you will be uploading the protected files and remove the READ permission from the anonymous user (typically it is extranet\anonymous). The full list of access rights can be found here. Expect all child assets to inherit the access permission from the parent folder. Now if you upload an asset into the restricted folder and try to request it in the frontend, you will get the "Access denied" message which is correct.
Next step is to create a new user role that will be allowed to view the restricted media files or use your existing one for the logged in users, assign the READ rights to this role, then assign this role to the media folder and items you want to be accessible behind the login only.
Depending on your user experience on the website you can check whether the current context user can read a certain media item or not by calling item.Access.CanRead() method and then show or not a download/preview link for it or redirect to the login form.
