Select User Agent

The User Agent tells the server who and what type of user is requesting the pages. Many sites use bot or spider detection tools so they don’t waste resources on people trying to seal content, do malicious things or waste resources.

By default we use HREFLang Builder’s user agent to indicate it is our crawler that is testing pages. In most cases they allow Google to access freely without any limitations.

Action: Choose which User-Agent you prefer us to use. We can request pages as either GoogleBot or with our own named bot named Hreflang Builder.

Default Setting: HREFLang Builder

Using GoogleBot – in most cases the server will allow Google into the site without any limitations but others are strict and only allow the “real googlebot” into the site and will block tools outside of Google’s IP range.

By default, we are using HREFLang Builder so we suggest adding HREFLang Builder’s bot to your approved list of tools. Make sure you add “HreflangBuilder/0.1 +https://www.hreflangbuilder.com/bot.html” to your whitelist.

By whitelisting our bot, those cases where your DevOps team will block everyone from crawling their sites, except our Hreflang Builder allowing us to do the diagnostic checks.