Why save init_task struct address to sp_el0 in arm64 boot code __primary_switched?

Alright, crash course on stack management on arm64:

At EL1 you have two stack pointer registers that you can access with mrs/msr: sp_el1 and sp_el0. You also have a register just called sp that you can access in most other instructions like add, str, etc. And then there's another system register called spsel, which consists of a single bit that controls whether sp is an alias of sp_el1 or sp_el0. To illustrate:

movz x1, 0x1000
movz x2, 0x2000
msr sp_el0, x1
msr sp_el1, x2
msr spsel, 0
add x3, sp, 0x10
msr spsel, 1
add x4, sp, 0x20
// AT this point, x3 == 0x1010 and x4 == 0x2020

In addition, when you're running at EL1 and you eret to EL0, the stack pointer will always be sp_el0. But the reason for all of this is that when you take an exception to EL1 again, your stack pointer is always switched to sp_el1. This is done because every single general-purpose register holds userspace values at that point, and you need a way to save them away without clobbering any of them (or storing to userland memory).
So what kernels usually do is set up an exception stack in sp_el1 onto which registers can be spilled when taking an exception. When taking an exception from EL1 to EL1 (e.g. an IRQ), then it should usually be safe to store the stack pointer that was in use before the exception was taken, so it is possible to run the kernel itself entirely on sp_el1.
Most operating systems don't do this, however, and instead add another stack pointer, a "normal kernel stack" if you will. Then the exception flow will look something like this:

1. Take an exception to EL1, the hardware implicitly switches to sp_el1 and disables interrupts.
2. Spill all general-purpose registers to the exception stack.
3. Replace the value in sp_el0 with the address of the "normal kernel stack" pointer.
4. Switch to spsel, 0 and enable interrupts.

Because exception vectors are different depending on whether you came from a context running on sp_el0 or sp_el1, this allows you to confine "expected exceptions" to sp_el0, and if you ever take an exception while running on sp_el1, you assume you faulted in a critical section and panic.

Now for the code you've shown: all it does in the first four instructions is set up the exception and "normal" stack pointers. It seems to be running with spsel, 1.

Also note that str_l isn't an actual instruction, but a Linux-specific macro:

/*
 * @src: source register (32 or 64 bit wide)
 * @sym: name of the symbol
 * @tmp: mandatory 64-bit scratch register to calculate the address
 *       while <src> needs to be preserved.
 */
.macro  str_l, src, sym, tmp
adrp    \tmp, \sym
str \src, [\tmp, :lo12:\sym]
.endm

So the code it would generate is:

adrp x5, __fdt_pointer
str x21, [x5, :lo12:__fdt_pointer]

Which, as you can see, doesn't use the old value of x5.

In the EL0 mode you can only use sp_el0 as a stack pointer, but In the EL1, you could select which one is the stack pointer between sp_el1 and sp_el0 using lsb(least significat bit) in the PSTATE Register. In Current Linux, this value is one. so sp_el1 is used as sp in the EL1 mode(kernel Mode). and sp_el0 is used as a pointer to current task's task_struct for fast context switching. for this, sp_el0 is set to the init task as the "msr sp_el0 x5". and every time context switch occurs. there is this kind of code which set sp_el0 to current task_struct.

there is another case when entering into EL1 mode from EL0 using svc instruction or interrupt. in that case, sp_el0 is set to the current task's task_struct using __entry_task(task_struct) because sp_el0 is used as stack pointer in the EL0 mode, but in the EL1, sp_el0 is used for another purpose which is described above.

It is a reciprocation of where init_thread_union is located in the kernel source(v5.10).

Open the arch/arm64/kernel/vmlinux.lds.S file to find RW_DATA.

. = ALIGN(SEGMENT_ALIGN);
__initdata_end = .;
__init_end = .;

_data = .;
_sdata = .;
RW_DATA(L1_CACHE_BYTES, PAGE_SIZE, THREAD_ALIGN)

Tracking RW_DATA shows the following definitions.

#define RW_DATA(cacheline, pagealigned, inittask)           \
    . = ALIGN(PAGE_SIZE);                       \
    .data : AT(ADDR(.data) - LOAD_OFFSET) {             \
        INIT_TASK_DATA(inittask)                \
        NOSAVE_DATA                     \
        PAGE_ALIGNED_DATA(pagealigned)              \
        CACHELINE_ALIGNED_DATA(cacheline)           \
        READ_MOSTLY_DATA(cacheline)             \
        DATA_DATA                       \
        CONSTRUCTORS                        \
    }                               \
    BUG_TABLE                           \

Here, we track INIT_TASK_DATA once again.

#define INIT_TASK_DATA(align)                       \
    . = ALIGN(align);                       \
    __start_init_task = .;                      \
    init_thread_union = .;                      \
    init_stack = .;                         \
    KEEP(*(.data..init_task))                   \
    KEEP(*(.data..init_thread_info))                \
    . = __start_init_task + THREAD_SIZE;                \
    __end_init_task = .;

we can see that it is located in vmlinux.lds.S -> RW_DATA -> INIT_TASK_DATA -> init_thread_union.