AArch64 execution stops after ERET when trying to switch to a lower exception level

I have started a bare-metal application for AArch64 for the purpose of education. It works fine, when I don't change the exception level to a lower one. But when I want to try to change the exception level from EL2 to EL1 the CPU seems to hang after the ERET instruction.

My current startup code:

#include <asm.h>

IMPORT_ASM(_cpu_el3_vec_tbl_set)
IMPORT_ASM(_cpu_el2_vec_tbl_set)
IMPORT_ASM(_cpu_el1_vec_tbl_set)

IMPORT_C(init)
IMPORT_C(main)

.text

ENTRY(_start)
    mrs x0, MPIDR_EL1
    and x0, x0, #0x3
    cmp x0, #0
    beq __elx

__wfe_cpu1_3:
    wfe
    b __wfe_cpu1_3
    
__elx:
__el3:
    mrs x0, CurrentEL
    and x0, x0, #0xC
    asr x0, x0, #2
    cmp x0, #3
    bne __el2
__el3_stack:
    ldr x0, =_stack_el3_e
    mov sp, x0
__el3_vector:
    bl _cpu_el3_vec_tbl_set
    msr SCTLR_EL2, xzr
    msr HCR_EL2, xzr
    mrs x0, SCR_EL3
    orr x0, x0, #(1<<10)
    orr x0, x0, #(1<<0)
    msr SCR_EL3, x0
    mov x0, #0b01001
    msr SPSR_EL3, x0
    adr x0, __el2
    msr ELR_EL3, x0
    eret
    
__el2:
    mrs x0, CurrentEL
    and x0, x0, #0xC
    asr x0, x0, #2
    cmp x0, #2
    bne __el1
__el2_stack:
    ldr x0, =_stack_el2_e
    mov sp, x0
__el2_vector:
    bl _cpu_el2_vec_tbl_set
    msr SCTLR_EL1, xzr
    mov x0, xzr
    orr x0, x0, #(1 << 31)
    msr HCR_EL2, x0
    /*orr x0, x0, #(7 << 6) */
    adr x0, __el1
    mov x1, xzr
    orr x1, x1, #(1 << 2)
    orr x1, x1, #(1 << 0)
    msr SPSR_EL2, x1
    msr ELR_EL2, x0
    eret
    
__el1:
__el1_stack:
    ldr x0, =_stack_el1_e
    mov sp, x0
__el1_vector:
    bl _cpu_el1_vec_tbl_set
    
    ldr x0, =_bss_s
    ldr x1, =_bss_e
    sub x1, x1, x0
    mov x2, #0x0
    cbz x1, __init
__bss:
    strb w2, [x0], #1
    sub x1, x1, #1
    cbnz x1, __bss
    
__init:
    bl init

__main:
    bl main
__main_wfe:
    wfe
    b __main_wfe

.end

When I comment out the section between label __elx to __el1 it works (the C function init and main will be executed). I have also tried to place the init (currently only contains a function, which prints out the current exception level over UART) before and after the ERET instruction. The result is, that the init call after ERET will be never executed.

No idea what's going wrong here.

EDIT:
I have also created a vector table with handlers for EL2 and EL1. But unfortunately no exception will be thrown.

UPDATE:
Okay, EL1 will be entered. But I have some strange issues with some parts of my C code when running in EL1:

First:
The execution from the C code is (subjectively) slower in EL1 than in EL2.

Second:
Some parts from the C code (my sprintf implementation) are not called. The strange thing here is with sprintf, it works without issues in EL2.

Is there maybe a wrong configuration of HCR_EL2 and/or SCTLR_EL1? Currently I initialize HCR_EL2 and SCTLR_EL1 with zero and set the RW bit (AArch64 state) in HCR_EL2.