select non-duplicated records

I have a table with about 50 millions records.

the table structure is something like below and both callerid and call_start fields are indexed.

id -- callerid -- call_start

I want to select all records that their call_start is greater than '2013-12-22' and callerid is not duplicated before '2013-12-22' in whole table.

I used something like this:

SELECT DISTINCT 
  ca.`callerid` 
FROM
  call_archives AS ca 
WHERE ca.`call_start` >= '2013-12-22' 
  AND ca.`callerid` NOT IN 
  (SELECT DISTINCT 
    ca.`callerid` 
  FROM
    call_archives AS ca 
  WHERE ca.`call_start` < '2013-12-21')

but this is extremely slow, any suggestion is really appreciated.