I am working on a project based on nextjs and strapi cms. This is deployed to azure app services which pulls the docker image from a azure container registry. Originally it has a front door and a front door and CDN profiler resources. Front door has been defined with basic WAF managed rules which is as follows.
- Cross-site scripting
- Java attacks
- Local file inclusion
- PHP injection attacks
- Remote command execution
- Remote file inclusion
- Session fixation
- SQL injection protection
- Protocol attackers
This site also does not include a robots.txt file. However when I do a live URL test on the site through https://search.google.com/test/mobile-friendly it says that URL is not available to Google. However this sites has been able to index through Bing search console.
In azure app services, are there any default configurations which block googlebot crawling through the site. Or is there any other resources that affect for this.
Following are the main resources that has been used while hosting this. Was not able find any specific rule that might be blocking googlebot. Azure app service, front door, front door WAF policy, Front door and CDN profiles, Container registry
Also I noticed that the app service which is hosting the cms is allowing the googlebot to crawl through the site, but front end is not allowing this. It would be a great help if someone can guide me on the steps that I need to follow in this case. As I am also somewhat new to azure, was not able to find the exact reason for this.
Update : I tried adding the robots.txt file to the site and surprisingly then the google was able to reach that URL and crawl through. However, I was under the impression that although the site do not include robots.txt file the site should be able to crawled by Google. If someone can explain the reason for this cause, it would be a great help.