Why test performance become worse when worker count increasing?

I have 4 kinds of test cases with the same steps for each type. When I use 1 worker, the case duration looks better. When I tried to increase the worker number to 2, 4, and 6, the case duration became longer. Some case duration is doubled. Here is the average duration table. My question is why the perf become worse when worker count increased. It is not related to the CPU core. I run the case on a 10-core windows PC. Performance compare