I wanted to ask for clarification on exactly what happens during runtime.

What actually happens is that when you call start(), the JVM typically¹ makes syscalls to the operating system to do the following:

1. Allocate memory segments for the new thread stack. (Typically two segments are allocated: one segment for the thread stack, and a second read-only segment that is used to detect stack overflow.)

2. Create a new native thread².

When the native thread is created, it must wait (along with all of the other currently ready-to-run threads) for the OS's thread scheduler to schedule it to a physical core. Generally speaking the OS's thread scheduler respects priorities, but scheduling between threads of the same priority is typically not "fair"; i.e. it is not guaranteed to be "first come, first served".

So, at some point, the OS will schedule the new native thread to run. When that occurs, the thread will execute some native code that gets hold of the Runnable reference and call its run() method. The same code will deal with any uncaught exceptions from the run() method.

The precise details will be JVM specific and OS specific, and you don't really need to know to them.

Is it the case that the main thread goes through all 100 iterations of the for loop creating these threads, and then the JVM arbitrarily decides later when each of these Thread objects starts?

Not necessarily. It might do, or it might not.

What actually happens will depend on how the OS's native code scheduler deals with a newly created native thread. And that will depend on various factors which are difficult to predict. For example, the behavior of other threads, and other applications, and so on.

Basically, there are no guarantees³ that the child threads will start executing in any particular order, or that the main thread will or won't complete the loop before any of the child threads starts.

^{1 - This is typical for a JVM which provides a 1 to 1 mapping between Java threads and native threads. This is the way that most current generation JVMs behave, but it is not the only implementation model.
2 - A native thread is a thread supported by the operating system. See Java Threading Models for more information and Native POSIX Thread Library for an example.
3 - Under some platforms and load conditions, you may be able to observe patterns of behavior, but you are liable to find that the behavior is different on other platforms, etc.}

The Answer by Stephen C is spot-on correct, and informative. I want to add two separate points: executors, and virtual threads.

Executors

Your code says:

new TestThread()).start()

In modern Java, we rarely need to address the Thread class directly. Instead, we use the Executors framework added to Java 5. An ExecutorService relieves us of the burden of juggling threads.

Write your tasks as Runnable/Callable objects to be submitted to an executor service implementation of your choosing.

Here is a revised version of your code, rejiggered for an executor service.

Notice how I chose an ExecutorService implementation that uses a backing pool of up to 4 threads, newFixedThreadPool. For brief tasks such as yours, we might have chosen newCachedThreadPool() for an unbounded thread pool. To run your tasks sequentially, we could have used newSingleThreadExecutor().

Minor note: I changed from getting the thread's name to getting its id number. A thread does not always have a name.

package work.basil.demo.threads;

import java.time.Instant;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

public class App2
{
    public static void main ( String[] args )
    {
        System.out.println( "INFO - The main method of our demo is starting. " + Instant.now() );

        Runnable task = ( ) -> System.out.println( "Thread instance: " + Thread.currentThread().getId() + " | " + Instant.now() );
        ExecutorService executorService = Executors.newFixedThreadPool( 4 );
        for ( int i = 0 ; i < 100 ; i++ )
        {
            executorService.submit( task );
        }

        executorService.shutdown();
        try { executorService.awaitTermination( 30 , TimeUnit.SECONDS ); } catch ( InterruptedException e ) { e.printStackTrace(); }

        System.out.println( "INFO - The main method of our demo is ending. " + Instant.now() );
    }
}

When run.

INFO - The main method of our demo is starting. 2021-06-06T06:57:31.423358Z
Thread instance: 17 | 2021-06-06T06:57:31.437343Z
Thread instance: 16 | 2021-06-06T06:57:31.435732Z
Thread instance: 14 | 2021-06-06T06:57:31.432704Z
Thread instance: 15 | 2021-06-06T06:57:31.433343Z
Thread instance: 14 | 2021-06-06T06:57:31.445160Z
Thread instance: 16 | 2021-06-06T06:57:31.445206Z
…
Thread instance: 16 | 2021-06-06T06:57:31.449733Z
Thread instance: 16 | 2021-06-06T06:57:31.449776Z
Thread instance: 14 | 2021-06-06T06:57:31.448232Z
Thread instance: 14 | 2021-06-06T06:57:31.449871Z
Thread instance: 14 | 2021-06-06T06:57:31.449923Z
Thread instance: 15 | 2021-06-06T06:57:31.447659Z
Thread instance: 14 | 2021-06-06T06:57:31.449964Z
Thread instance: 16 | 2021-06-06T06:57:31.449816Z
Thread instance: 17 | 2021-06-06T06:57:31.449404Z
INFO - The main method of our demo is ending. 2021-06-06T06:57:31.450433Z

You asked:

the JVM arbitrarily decides later when each of these Thread objects starts

When we call ExecutorService#submit, the executor service takes care of actually running that task, as the name suggests. The executor service tries to run the task soon, depending on a thread being available. How long it takes for your task to actually begin its work will vary, depending on the state of the JVM, the algorithms used by the host OS in scheduling threads for execution on the CPU cores, and how burdened is the host hardware.

You can track the progress of the execution by capturing the Future object returned by the submit method, and interrogate if the task is done or cancelled.

You may choose to specify a delay before running the task if using a ScheduledExecutorService.

Virtual threads

Understand that the native threads discussed in that Answer by Stephen C are relatively “heavy”/“expensive”. Each thread is relatively slow to create, typically start with a large amount of stack (usually a meg), and are somewhat slow when switching for some execution time on a CPU core because of going between the OS kernel space and userland.

If Project Loom is successful, Java will gain an additional kind of thread, virtual threads, also known as fibers. These lightweight virtual threads are managed within the JVM without directly involving the host OS.

Many virtual threads are mapped to actually run on a few number of native/platform threads. The virtual threads have a smaller stack by default that can grow and shrink(!) as needed, unlike native threads. Virtual threads when blocking (such as waiting for file system, network access, database response, etc.) are automatically “parked”, that is, set aside so that another virtual thread can be switch into place on the native thread for immediate execution. This switching will be very fast as it can be managed by the JVM with involving the host OS.

These characteristics mean that virtual threads are very “cheap”, taking minimal amounts of both memory and CPU. Even millions of threads may be practical on conventional computer hardware.

Furthermore, virtual threads eliminate the need for thread-pooling. This means no threat of leftover ThreadLocal values leaking between uses of a recycled thread.

Virtual threads are appropriate for running code that blocks. This applies to most run-of-the-mill business apps commonly developed in Java. However, for true CPU-bound tasks such as video encoding, stick with conventional threading.

Changing your code to use virtual threads is as easy as changing your implementation of ExecutorService.

ExecutorService executorService = Executors.newVirtualThreadExecutor() ;

For more information, see the more recent video presentations and interviews by Ron Pressler and others from Oracle working on Project Loom. Experimental builds are available now based on early-access Java 17 if you want to learn more.