How to prevent a development staging website, hosted on Azure, from being indexed by search engines

user_1937348 0
Viewed 1440

Specific to Web Apps hosted on Microsoft Azure, is there a way to prevent the mydomain.azurewebsites.net URL from being indexed by search engines? I'm planning to use a web app as a staging website, and don't want it to accidentally get indexed.

I know I could add a robots.txt file to the project with everything set to no-index, but I don't want to ever accidentally publish it to the production site (or alternatively, forget to publish it to the staging website).

Is there a setting in Azure that will prevent the ".azurewebsites.net" domain from being indexed? Or if the robots.txt file is the only way, how do you keep it organized so that the right robots.txt file is published to staging and production, using ASP.NET Core.

azure-web-app-service azure asp.net
3 Answers

You can publish robots.txt to your staging server once. This can be done via FTP or via your SCM site. Once you publish this file, web publish will not remove additional files on the server (including your robots.txt file) unless you select "Remove additional files at destination" in your web publish settings.

So the robots.txt file will hang around forever on your staging server unless you remove it. Then you do not need to include robots.txt in your project or solution, and not risk accidentally publishing it to your production environment.

user_75091680

Restrict access based on hostname and request IP

Unless you need your staging slot to be accessible to a wide range of dynamic IPs, you could consider using the URL Rewrite module and adding rule[s] to your web app config to disallow traffic except for a few known IPs, but make those rules conditional on the HOST header matching the staging host (mydomain.azurewebsites.net), so they can never apply on the production hostname.

The details in the question here show a similar type of setup.

user_9479860
user_16589060
Related