Building a gem from outside of the directory

I have an ordinary gem directory structure in directory path_to_foo:

path_to_foo
|-bin
 |-some_file_name
|-lib
 |-foo.rb
|-foo.gemspec

When I move to that directory and build the gem, it succeeds:

path_to_foo$ gem build foo.gemspec 

but if try to do that from outside of the directory, it fails:

~$ gem build path_to_foo/foo.gemspec
WARNING:  See http://guides.rubygems.org/specification-reference/ for help
ERROR:  While executing gem ... (Gem::InvalidSpecificationException)
["bin/some_file_name"] are not files

How can I build a gem from a gemspec from outside of the directory? I think I probably need to add something to the gemspec file.