Bash script to limit a directory size by deleting files accessed last

I had previously used a simple find command to delete tar files not accessed in the last x days (in this example, 3 days):

find /PATH/TO/FILES -type f -name "*.tar" -atime +3 -exec rm {} \;

I now need to improve this script by deleting in order of access date and my bash writing skills are a bit rusty. Here's what I need it to do:

1. check the size of a directory /PATH/TO/FILES
2. if size in 1) is greater than X size, get a list of the files by access date
3. delete files in order until size is less than X

The benefit here is for cache and backup directories, I will only delete what I need to to keep it within a limit, whereas the simplified method might go over size limit if one day is particularly large. I'm guessing I need to use stat and a bash for loop?