Is there a way to control which thread a configuration method uses in TestNG?

I recently switched to using ThreadLocal in my test suite, and while it has worked great for parallel tests, it has now broken non-parallel tests. I have @BeforeClass/@BeforeMethod methods that handle creating a new instance of WebDriver based on the test class's driver instance policy. However, when they are creating a new instance of WebDriver it is tied to the thread of "Test Worker" which is only used in configuration methods, like @BeforeClass and @BeforeMethod. This doesn't happen when running with parallelization enabled. Here's a very simplified example:

public class DriverWrapper () {
   private static ThreadLocal<WebDriver> driver = new ThreadLocal<>();
 
   public DriverWrapper() {}

   public void newInstance(){
      driver.set(new ChromeDriver());
   }

   public WebDriver getDriverInstance(){
      return driver.get();
   }
}

abstract public class AbstractTest {
   DriverPolicy policy;
   DriverWrapper driver;
 
   public AbstractTest(DriverPolicy policy){
      this.policy = policy;
   }
 
   @BeforeClass
   public void setUpClass() {
      if (policy == DriverPolicy.NEW_INSTANCE_PER_CLASS) {
         driver = new DriverWrapper();
         driver.newInstance();
      }
   }   

   @BeforeMethod
   public void setUp() {
      if (policy == DriverPolicy.NEW_INSTANCE_PER_METHOD) {
         driver = new DriverWrapper();
         driver.newInstance();
      }
   }
}

public class SomeTest {
   public SomeTest() {
      super(DriverPolicy.NEW_INSTANCE_PER_CLASS);
   }

   @Test
   public void doSomeThings() {
      SomePageFactoryClass somePageFactoryClass = new SomePageFactoryClass(driver.getDriverInstance());
      // it does some pagefactory stuff 
      somePageFactoryClass.login(); 
      // login() tries to do some WebDrivery stuff
      // It's here that I hit the NPE because the thread ID that got assigned to the WebDriver in
      // DriverWrapper is different from the thread ID in @Test, so it's just null.
   }
}

My testng.xml that runs these tests only encounters this issue when parallel is disabled. As soon as I enable parallel, @BeforeClass and @BeforeMethod methods are running on the same thread as their class's @Test methods.

I've come up with a messy work around where I'm just creating the new instances of DriverWrapper in each individual @Test method, but it feels wasteful when I have the code already sitting in AbstractTest.

Is there something obvious I'm missing? Maybe I should just not use ThreadLocal for non-parallel tests?