ssh_config AddKeysToAgent yes not working on Fedora

I've tried a multitude of things to get this working but despite having the AddKeysToAgent variable set to yes in my ssh_config the keys are not getting added.

This is my ssh-config:

Host * 
    AddKeysToAgent yes 
Host remote 
    HostName /*hostname for remote here*/
    User dcaglar2
    IdentityFile ~/.ssh/personal_laptop
    IdentitiesOnly yes 

Host git 
    HostName github.com
    User git  
    IdentityFile ~/.ssh/git
    IdentitiesOnly yes 

and running

ssh-add -l 

returns

The agent has no identites.

I've checked the man pages ssh, ssh_config, but wasn't able to find anything. I know that I can add a line to my .bashrc as a substitute but I just want to know what's wrong at this point.