Preventing bots from accessing website hosted on GCloud

user_6486437 0
Viewed 77

I am trying to prevent all search engines bots from accessing my website.

I read that these permissions can be specified to robots.txt file but I am not sure where to access this file in order to edit it on Google Cloud.

I developed my web server using Python with Gcloud.

Note that I read the following topic http://stocksonfire.in/seo/edit-robots-txt-google-cloud-solved/ However, I did not find any VM instance in my resources. Do I need to create one first?

edit: This is my app.yaml file after appling @Dac Saunders suggestion

runtime: python
env: flex
entrypoint: gunicorn -b :$PORT main:app

runtime_config:
  python_version: 3

handlers:
- url: /robots\.txt
  static_files: robots.txt
  upload: robots\.txt
robots.txt google-cloud-platform google-app-engine python
1 Answers

My robots.txt(with python appengine) looks like this. 

User-agent: *
Disallow: /q?

Sitemap: /sitemap.xml.gz

I placed it in <MYAPP>/static/robots.txt. Then I mapped it in app.yaml like this:

handlers:
- url: /(robots\.txt|favicon\.ico)
  static_files: static/\1
  upload: static/.*

This way my robots.txt becomes accessible in the root of my domain.

user_1082070
Related