Since Google Maps API changed to "Credits" my question is: Do crawlers/bots (from search engines or watchdogs like i.e. "uptrends") consume Maps APIs Credits?
If so, is there a possibility to "exclude" them like in Google-Analytics?
I would like to avoid a lot of costs for "nothing".
Thanks in advance Tobi
My experience is that YES, requests to the Maps API do create a transaction that is chargeable even when a bot or crawler accesses the page. I too had to take a close look at this when Google changed to the $200 credit system when the free quota method was retired. My website is making approximately 3,000 map requests a day. However, Google Analytics shows that my page views per day that require a map request are only 700! To avoid this costing me dearly I was able to switch to using Embed Maps that have no limit and are free.
Analysis of my server access log revealed that indeed, most of the page views are from crawlers and bots. This is consistent with reporting I've seen including one that says 52% of website visits are from crawlers and bots. On my website the crawler/bot percentage is closer to 75%. That's because I have around 50,000 pages. The human traffic concentrates on a small percentage of the popular pages while the crawlers/bots visit them all.
Your second question asks if there is a way to exclude bots like the option in Google Analytics provides. No, it is not provided as an option in the Maps API. You can detect the presence of a bot in your code and suppress the api request like in this PHP Stop Bot solution. It's a rudimentary start to filtering them out. It could be enhanced by using a bot/crawler detector like this one: https://github.com/JayBizzle/Crawler-Detect
I would note that there are good bots and bad bots. You certainly want your page content indexed by Google and Bing with the maps present. Don't blindly exclude all bots/crawlers.
I'm heavily into this on a project that requires chargeable Map and Places API requests. I might use a service provided by https://51degrees.com/ for bot detection. They have free and paid solutions targeted at device detection. In the process, their APIs identify human or not!
Good luck.
In order to get answer to your question I can suggest having a look at robots.txt file of Google Maps JavaScript API
http://maps.googleapis.com/robots.txt
Currently it has the following rules
User-agent: *
Disallow: /maps/api/js/AuthenticationService.Authenticate
Disallow: /maps/api/js/DirectionsService.Route
Disallow: /maps/api/js/DistanceMatrixService.GetDistanceMatrix
Disallow: /maps/api/js/ElevationService.GetElevationForLine
Disallow: /maps/api/js/GeocodeService.Search
Disallow: /maps/api/js/QuotaService.RecordEvent
According to this file the quota service is disabled for crawlers, so they shouldn't consume your quota.
The file has now been changed and reads
User-agent: AdsBot-Google
Disallow: /maps/api/js/
Allow: /maps/api/js
Disallow: /maps/api/place/js/
Disallow: /maps/api/staticmap
Disallow: /maps/api/streetview
User-agent: *
Disallow: /maps/api/js/
Allow: /maps/api/js
Disallow: /maps/api/place/js/
Disallow: /maps/api/staticmap
Disallow: /maps/api/streetview
The line Allow: /maps/api/js means that bots will in fact consume resources when crawling. In Particular, Google will charge you for calls generated by the Google bot (a bit shady if you ask me).
Answering from the response received by Google Maps team (October 2020)
Googlebot and Google Adsbot usage can indeed generate legit requests to the Maps APIs, however, Google Maps doesn't charge for these bots.
However, Google Maps can't prevent other bots like Bing Bots from making legitimate calls and increasing the usage.
Solution to this is to detect HTTP User-Agent that contains "Googlebot", "AdsBot-Google","SemrushBot" or "Bingbot", "etc. and block them from calling Maps API requests.
For example: Ex.:
User-agent: Bingbot
Disallow: /maps/api/js/AuthenticationService.Authenticate
Disallow: /maps/api/js/DirectionsService.Route
Disallow: /maps/api/js/DistanceMatrixService.GetDistanceMatrix
Disallow: /maps/api/js/ElevationService.GetElevationForLine
Disallow: /maps/api/js/GeocodeService.Search
Disallow: /maps/api/js/QuotaService.RecordEvent
Disallow: /maps/api/js/AutocompletionService.*
Disallow: /maps/api/js/PlaceService.*
Disallow: /maps/api/place/js/AutocompletionService.*
Disallow: /maps/api/place/js/PlaceService.*
Disallow: /maps/api/js/ViewportInfoService.*
Disallow: /maps/api/js/GeoPhotoService.*
The robots.txt file of Google Maps JavaScript API has changed. See:
http://maps.googleapis.com/robots.txt
It is now:
User-agent: AdsBot-Google
Allow: /maps/api/js?
Disallow: /maps/api/js/
Disallow: /maps/api/place/js/
Disallow: /maps/api/staticmap?
Disallow: /maps/api/streetview?
Disallow: /maps/api/streetview/
User-agent: *
Allow: /maps/api/js?
Disallow: /maps/api/js/
Disallow: /maps/api/place/js/
Disallow: /maps/api/staticmap?
Disallow: /maps/api/streetview?
Disallow: /maps/api/streetview/
So I guess calls to maps/api/js? are allowed will contribute to the api quota.