Best practise of test database configuration for pytest, SQLAlchemy + FastAPI

I have built one of my services using FastAPI(0.75.1), SQLAlchemy(1.4.34) with pytest(7.1.1) and factory boy (3.2.1) for unit-testing.

I have managed to set up testsession mechanism with test database. I have been writing unit-tests for some time actively using factories, but the further I go, the more I face the problem of parallel sessions. My current TestSession setup looks like this

TestSessionLocal = orm.scoped_session(
    sessionmaker(
        autoflush=False, bind=engine
    )
)

And this is what I am importing when declaring factories

class FooFactory(factory.alchemy.SQLAlchemyModelFactory):
    bar = factory.Faker('last_name')

    class Meta:
        model = Foo
        sqlalchemy_session = TestSessionLocal
        sqlalchemy_session_persistence = 'commit'

With time I get more and more errors because of parallel test sessions that can't either create and object or cannot tear them down.

My question is - am I correct to assume that I need to decrease the number of testsessions running throughout all tests to one? So, it should be the only session that manages factories, in-test changes and changes that must occur on the part of application running in the test mode? In that case - what is the best way to make sure that at least inside unit tests I am using only one session to create objects via factories and to do all other stuff? Should I go after fixture that set up test session and then fixtures that set up factories working on that session?