Build factories for self referential associations in rails

I have a typical requirement, I have to address user object as follows

user.referrer and user.referrers.

Basically, user can refer more than one person and one person should be referred by one particular user. So I build associations as follows. They are working great.

class User < ActiveRecord::Base
    attr_accessible :account_number, :display_name, :referrer_id

    has_many :referrers, :class_name => "User", :foreign_key => "referrer_id"
    belongs_to :referrer, :class_name => "User"
end

Now I would like to test assoications in Rspec. I am using factory girl so any one help me to build factories.

I tried as follows but end up with an errors

factory :user do
  gender :male
  name "super test"
  .....
  .....

  factory :referrer do
  end
  association :referrer
end