TFS "Assign To" is showing too many users

I am connected to a TFS server, and when I create a work item (Bug, Task, etc.), the "Assign To" shows WAY too many users. I cannot find a way to limit the users shown that will affect anyone else who connects to the server.

For example, I have under the "Assign To" dropdown:

Administrator
LOCALE SERVICE
User 1
User 2
User 3
User 4
User 5

I want the "Assign To" dropdown to only show:

User 1
User 3
User 5

How would I do this, so that it changes on the server, and if anyone new connects to the server, they can only see the three users on the "Assign To" dropdown?