gcc -O optimiziation doesn't detect heap overflow with unused variable

I went to godbolt and it looks like there's no assembly instructions at all

Indeed, GCC optimized out all your code because you do not use in any way. If you change it slightly then it will stay in generated assembly and AddressSanitizer will find heap overflow.

The following code generates AddressSanitizer heap-buffer-overflow error as you expect:

#include <stdio.h>
#include <stdlib.h>

int main(void)
{
    char* x = malloc(10);
    char n = x[11];

    return n;
}

What happens to the code when you pass only -O without any level?

It's the same as -O1, "optimize a little bit"

What's the difference between this and -O0, that by the way, works well?

Since your code has no obvious side effects, it will be removed by the optimizer if enabled. This happens no matter if you invoke undefined behavior with x[11] or if you access a valid index x[0]. Writing to the x[0] item first to ensure it isn't indeterminate doesn't have any effect either.

-O0 explicitly disables optimizations, so you'll get some manner of machine code generated... maybe. The compiler doesn't have to generate anything predictable or meaningful in case your code contains undefined behavior.

There's generally no bounds-checking in C, it's the programmer's responsibility to handle it.

As for -fsanitize=address, add a side effect and it might kick in:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv) 
{
    char* x = malloc(10);
    char n = x[11];
    printf("%d\n", n);
}

Results in:

==1==ERROR: AddressSanitizer: heap-buffer-overflow on address-...

The docs answer both of your questions.

What happens to the code when you pass only -O without any level?

-O
-O1

Optimize. Optimizing compilation takes somewhat more time, and a lot more memory for a large function.

[...]

It enables optimizations.

What's the difference between this and -O0

-O0

Reduce compilation time and make debugging produce the expected results. This is the default.

As the default, it really does nothing at all.

In this mode, optimizations (or at least non-trivial ones) are effectively disabled.

Zero optimiziation doesn't detect heap overflow

On Compiler Explorer, you did enable optimizations (by using -O).

If you stop optimizing the variables away (by removing -O or by using -O0), you get the expected error.

<source>: In function 'main':
<source>:7:15: warning: unused variable 'n' [-Wunused-variable]
    7 |          char n = x[11];
      |               ^
<source>:7:15: warning: 'x[11]' is used uninitialized [-Wuninitialized]
    7 |          char n = x[11];
      |               ^

Program returned: 1
Program stderr

=================================================================
==1==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000001b at pc 0x0000004011a4 bp 0x7fffc6caef80 sp 0x7fffc6caef78
READ of size 1 at 0x60200000001b thread T0
    #0 0x4011a3 in main /app/example.c:7
    #1 0x7f88c4c140b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x240b2)
    #2 0x40109d in _start (/app/output.s+0x40109d)

0x60200000001b is located 1 bytes to the right of 10-byte region [0x602000000010,0x60200000001a)
allocated by thread T0 here:
    #0 0x7f88c4e9d1af in malloc (/opt/compiler-explorer/gcc-12.1.0/lib64/libasan.so.8+0xbb1af)
    #1 0x401167 in main /app/example.c:6
    #2 0x7f88c4c140b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x240b2)

[snip]

Demo on Compiler Explorer

The same would happen with optimizations if the program actually used n.