The User Agent tells the server who and what type of user is requesting the pages. Many sites use bot or spider detection tools so they don’t waste resources to people trying to seal content or do malicious things. In most cases they allow Google to access freely without any limitations.
Action: Choose which User Agent you prefer us to use. We can can request pages as either GoogleBot or with our own named bot names Hreflang Builder.
Default Setting: GoogleBot
By default we are using GoogleBot, by there are cases where the sites specifically block requests that are not the real GoogleBot. In this case we suggest using the Hreflang Builder bot spider ID and make sure you add “HreflangBuilder/0.1 +https://www.hreflangbuilder.com/bot.html” to your whitelist.
By whitelisting our bot, those cases where your DevOps team will block everyone from crawling their sites, except our Hreflang Builder allowing us to do the diagnostic checks.