If you use a CDN like Cloudflare of Akamai or any other sort of network security product for the website you should ask the DevOps team to add us to the Whitelist of User Agents and IPs that are allowed to access the site.
Adding User Agent
If you plan to use URL validation you should whitelist the applications IP’s and/or the User Agent
User Agent: “HreflangBuilder/0.1 +https://www.hreflangbuilder.com/bot.html”
Adding IP’s
IP1: 54.156.174.125
IP2: 34.237.255.154
IP3: 52.6.109.245
If there are specific times of day or crawl rates that you require please let us know so we can enable them.
