How to require all files in a directory, in a gem?

I know that I can require all files in a directory using

Dir['path/to/files/**/*.rb'].each { |file| require file }

However, I'm trying to do that for a gem setup file, like so:

class MyGem
  module NS; end
end

Dir['lib/my_gem/files/**/*.rb'].each { |file| require file }

(where all the files in lib/my_gem/files are namespaced under MyGem::NS). With a directory structure that looks like this:

lib                              
|-- my_gem                        
|   |-- files              
|   |   |-- file1.rb       
|   |   |-- file2.rb         
|   |   |-- file3.rb         
|   |   `-- ... (many more)      
`-- my_gem.rb                     
my_gem.gemspec

And, well, the Dir command doesn't seem to work when I require my gem in another file, since it seems to look for the file paths at run time in my project root (of whatever project I've required my gem from).

How can I require all the files in lib/my_gem/files in lib/my_gem.rb?