What happens if you specify max heap size greater than available RAM

Only if your -Xms (minimum) is larger than available memory will you get an immediate failure on initialization of the JVM

$>java -Xms100g            #JVM fails to start
Error occurred during initialization of VM Could not
reserve enough space for object heap

If your -Xmx (maximum) is larger than available memory your JVM does initialize since you are not using memory yet

$>java -Xmx100g            #JVM starts up fine
Usage: java [-options] class [args...]
...

If your -Xmx (maximum) is larger than the available memory (total memory to include any virtual memory) you will get a runtime failure if and only if your JVM processes actually tries to use more memory than the machine has.

OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5feb100000, 927465472, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 927465472 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /some/file/path/hs_err_pid25.log

It wont 'thrash' until it nears your -Xmx limit, but if that limit is above your available memory you will get the above memory allocation error and your program will terminate before thrashing is even considered. (And that is very dramatic!)