The problem

When you use --limit in your ansible-playbook command, you cannot play any tasks on hosts out of this limit. This includes gathering facts (i.e. play the setup module automatically or explicitly) and set_fact (i.e. create/update a fact manually for the host). With ansible default settings (memory facts cache), you will not be able to query any hostvars on those hosts inside your playbook because there are no keys for their inventory_hostname in that dict.

Facts cache to the rescue

A solution is to enable a non ephemeral facts cache in ansible.cfg. The default cache goes to memory and dies at the end of the playbook.

Enable the cache

As a start, you can enable and store the cache in json files on disk with the following settings in ansible.cfg:

[defaults]
fact_caching = jsonfile
fact_caching_connection = /path/to/cache/folder

For more info about this feature and all the possible cache backends, you can look at the comments for the relevant parameters in the default ansible.cfg file and check the cache plugin documentation

Populate the cache

Once you have a non ephemeral cache, you can gather facts and set_fact for all relevant hosts. If you only need facts queried from the host, you can easily do this with an ad-hoc command:

ansible -i your/inventory my_hosts -m setup

In your case, it is a bit more complicated, as you want to push to the cache the result of a task. You will need to create a first playbook that you will run on all hosts you need to use later. I'll call it init_facts_and_tokens.yml:

---
- name: Gather hosts facts and initialize tokens
  hosts: my_hosts
  # You can uncomment the line below to be explicit but this is on by default
  # gather_facts: true

  tasks:
    - name: Slurp token file token
      slurp:
        src: /var/mytoken
      register: token_file

    - name: Register token in facts cache
      set_fact:
        token: "{{ token_file.content | b64decode | replace('\n', '') }}"

and your run it on all hosts with

ansible-playbook -i your/inventory init_facts_and_tokens.yml

Use the cache

Now that the cache is populated, you can play your other playbook, with a limit or not, and call hostvars for hosts out of the play/limit. If their facts have been correctly cached, you will get the values queried during the last facts gathering or set_fact

In such a scenario, you can probably disable facts gathering on all your plays to save some execution time on your final playbook. And if for any reason (e.g. changing a network interface, adding an lvm volume.....) you need to refresh the facts during the play, you can simply run the setup module. For any other fact not queried from you host, any use of set_fact for the given variable will create/refresh its value

---
- name: Do the jobs with cache facts
  hosts: my_hosts
  gather_facts: false
 
  vars:
    # We will use toker from first server in group,
    # even if out of the limit
    this_play_token: "{{ hostvars[groups['my_hosts'][0]].token }}"
  
  tasks:
    - name: Show token
      debug:
        msg: "The token for this play is {{ this_play_token }}"

    # Examples to illustrate above explanations
    - name: This task can use cache
      debug:
        msg: "OS of somehost.com is: {{ hostvars['somehost.com'].ansible_os_familly }}"

    - name: This task would change target, possibly obsoleting gathered facts
      debug:
        msg: "Warning I might change system facts !"

    - name: Refresh facts cache
      setup:

    - name: Back to normal activity
      debug:
        msg: "I use latest gathered facts from cache"
    
    - name: This would refresh the cached token for current host
      set_fact:
        token: "Not so valid token"

You can now launch this second playbook with a limit or not. It will still be able to read facts (queried or user set) for any host even out of the play

# run on all hosts
ansible-playbook -i your/inventory final_playbook.yml

# run only on third host of my_hosts group
ansible-playbook -i your/inventory --limit my_host[2] final_playbook.yml