How to avoid catching OutOfMemoryError in java and how to resolve class java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception issue

You can't cast OutOfMemoryError into an Exception.

Check the OutOfMemoryError Doc, and you can see that it is not a child class of Exception but from Error.

And as the Error Doc, states:

An Error is a subclass of Throwable that indicates serious problems that a reasonable application should not try to catch. Most such errors are abnormal conditions.

So you should not try to catch OutOfMemoryError as you don't have much options available at that point to recover your application state. For sure however you CAN NOT cast a OutOfMemoryError into an Exception as this is not possible.

Errors are not Exceptions.

Assuming that you get the OutOfMemoryError when calling an external service, then you could check if there was an Error before casting the cause to an Exception

 catch (ExecutionException e) {
    if (e.getCause() instanceof Error) {
        logger.error("Caught Error");
    } else {
        Exception ex = (Exception)e.getCause();
        if(ex instanceof UncategorizedJmsException) {
            logger.error(e.getMessage());
        } else if(ex instanceof ApplicationException) {
            logger.error(e.getMessage());
        }
    }
}

I am going to try expanding on Panagiotis Bougioukos' answer a bit.

So you should not try to catch OutOfMemoryError as you don't have much options available

This is true. This is what the architects of Java recommend. In my almost 17 years developing software using Java, I have only needed to catch OutOfMemoryErroronce. It would nice if OP include details on the nature of the error to know how to solve this part of the problem. In my case, I was working on a feature that used Apache POI to open Excel spreadsheets. The application was for an Oil and Gas company who kept DECADES worth of data on Spreadsheets. Our application was limited due to the fact that the application could only run on 32-bit due to some legacy DLLs our Java application needed to interact with on other parts of the program. Due to this limitation, I decided that instead of asking our clients to break down these spreadsheets, I would catch these errors and attempt to open the files in old Excel format, rather than on the new format. This decision worked like a charm.

I am telling this story so that you can see how specific these circumstances were: new Excel with excessive data, an application with a 32-bit limitation, not wanting to bug down the clients with manual data manipulation, etc. For you, the key here is figuring out the real root cause(s) of the problem. Once you that, try to mitigate by fixing the issue outside the application. For example, maybe you are allocating not enough memory to run your code. After all, we don't know how much max memory your application needs or needed before this issue surfaced.

If you go through the list of issues and there are enough mitigating circumstances (like in my case) to justify catching the error AND your plan to make this "fix" in the code is solid, then proceed at your own risk. As it has been said already here, an OutOfMemoryError is a sign that something has gone horribly wrong. So, by catching the error, if your alternate plan to proceed running your application in some degraded state is not sound, it could lead to even major issues, like losing data.

Handling the error

In my opinion, you should not create convoluted "instanceof" interrogation on your exception object. If you need to handle specific exceptions (or errors), just handle them directly. For example, if you need to handle an out of memory error, simply do that in the try/catch

try {
    // execute your process
} catch (OutOfMemoryError error) {
    // execute alternate process
}

The body of the try should be as small as possible, the line of code that would result in the condition if you know it.