raise at ExceptAddr not working for x64-debugger

I use Delphi's raise at <address> feature to find the origin of a re-raised exception while debugging.

While this does work under 32 bit, the x64 debugger always shows the line where the exception is re-raised.

Minimal Example:

program Project1;

{$APPTYPE CONSOLE}

{$R *.res}

uses
  System.SysUtils;

begin
  try
    raise Exception.Create('Test'); // this line should be where the debugger breaks
  except
    var E := AcquireExceptionObject;
    raise E at ExceptAddr; // for x64, the debugger breaks here
  end;
end.

I tested this with Delphi 10.4 and 11.

Did I misunderstand the raise at incorrectly?

Is this a bug, or maybe settings related?