symfony extra user info to be persisted in session after authentication

I am trying to use a property called lastLoginOnReference in my User entity. This property is not linked to any column in my corresponding db table as I don't want to store it.

I use this property in order to display to the user some activity logs since the last time he logs in.

I also have a lastLoginOn property, also in the User entity, which has a dedicated column in my user table.

what am I doing at authentication is this one (in my GoogleAuthenticator class):

public function getUser($credentials, UserProviderInterface $userProvider)
    {
        $googleUser = $this->getGoogleClient()
            ->fetchUserFromToken($credentials);

        $email = $googleUser->getEmail();
        $user = $this->em->getRepository(User::class)
                    ->findOneBy(['email' => $email]);

        if ($user)  {
            $user->setLastLoginOnReference($user->getLastLoginOn());
            $user->setLastLoginOn(new DateTime('now'));

        } else {
            (not important)
        }                  

        $this->em->persist($user);
        $this->em->flush();

        return $user;
    }

so I store in my lastLoginOnReference property the value from the lastLoginOn User property.

I update the lastLoginOn User property in the User repository and it works.

But when in my controller I use

$this->getUser();

getLastLoginOnReference returns null

I read the documentation and found out the user is apparently refreshed between every request. So I suspect my user object to be refreshed and as the lastLoginOnReference not in db, it is set to null

Then I have implemented the Serializable interface in my User entity, specifying this property to be serialized as well but it doesn't change anything.

login/logout are working as expected, and I am using Symfony 5.1