Is it possible to move pool specifications into a template in ADO YAML?

We are not able to only specify Agent pool in Yaml template.

To achieve your requirement, you can define the pool name as variable in YAML template. Then you can use the variable as Pool name in main YAML.

Here is an example:

poolname.yml:

variables:
   poolname: agentpoolname

main.yml

variables:
   - template:  pool.yml

pool:
  name: $(poolname)

steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'