How Docker can do $ su...
Usually it can't. A typical Docker container won't have a root password or any other password set, and it may not even have the su binary.
This isn't typically a problem since a Docker container only runs a single process, and you can explicitly set the user ID when you start the container with...
docker run -it --rm -u 10001:10001 alpine:3.16
The important thing about Unix user and group IDs is their numeric value. For example, there's nothing special about the user name root, but if the current user has the numeric user ID 0 then it has special privileges. Similarly, this command sets both the user and group IDs to 10001; it can write files if they have user- or group-write permission and are owned by exactly that numeric user or group ID. There's no requirement that a user or group "exist" per se.
Particularly in a Docker context, this can come up routinely with bind-mounted host directories. The container's numeric user ID needs to match the numeric user ID that owns the files on the host, but there's no requirement to "create the user".
Is it safe to use $ docker run -u or USER variable in Dockerfile with a nonexistent user?
Yes. The two things that are important are (a) whether or not the numeric user ID is 0 and (b) whether the numeric user ID matches the ownership of files in the container or mounted volumes. If the numeric user ID doesn't match then the user won't be able to overwrite files.
It's possible some application code might try to find out the current user name and get confused when it doesn't exist. If you're using bash as a debugging shell it might print a "no such user" complaint as part of the prompt, but this is totally cosmetic.
A more specific statement: it's safe to do something like
docker run -u $(id -u):$(id -g) -v "$PWD:/data" ...
to both set the numeric user and group ID of the container to match the host, and to mount the current directory into the container somewhere. There's no requirement to "create the user" before you do, and it's not an especially good practice to build an image that has a specific host user ID built in.