I’m getting Google Crawler Errors. What should I do?

Review the following guidelines if you’re experiencing crawler errors in the Google Webmaster Tools Health dashboard.

• Monitor the performance and availability of your website directly to rule out any errors produced by your own web server. There are a number of free and paid services which you can use:
  • Pingdom
  • StatusCake
  • Monitor.Us
  • updown
• Do not block Google crawler IPs in the Cloudflare Threat Control Panel (you can verify whether an address belongs to Google using the method they mention in this article). Blocking IPs in their range will prevent Google from crawling your website.
• Do not block the United States as a country in the Cloudflare Threat Control Panel. Many search engines are based in the United States and this setting could prevent them from crawling your website.
• Do not block Google’s User-Agents in your .htaccess, server configuration, robots.txt, or web application. Google may use a variety of User-Agents to crawl your website. Please review this list of common User-Agents Google uses when crawling your website. When making modifications, you can test your robots.txt file on the, “Blocked URLs page“ under the, “Health” section in Google Webmaster Tools.
• Do not allow crawling of files in the /cdn-cgi/ directory. This path is used internally by Cloudflare and Google will encounter errors when crawling it.

Troubleshooting steps for the most commonly reported errors are below. For anything not mentioned here, please follow the steps in the additional troubleshooting section and then open a support ticket.

Google detected a significant increase in the number of URLs we were blocked from crawling due to authorization permission errors.

HTTP errors in the 400 range are the most common type of error users will see in Google Webmaster Tools Health Dashboard. Errors in this range indicate a request failure. Cloudflare acts as a messenger and delivers these errors from your web server to Google. These errors may be caused by something as simple as a missing page on your web server or a malformed link in your HTML. The solution will depend upon the problem encountered. Once you have identified and corrected the problem, Google will make additional attempts to crawl your website and errors will clear automatically.

HTTP errors in the 500 range, indicate that Cloudflare or your web server experienced an internal error. Unfortunately, the Google Webmaster Tools Health Dashboard provides a limited diagnostic view into these kinds of errors. If your web server is producing errors in the 500 range, it can be challenging to isolate these errors from errors that Cloudflare may be producing. The best solution is to monitor the health of your web server to help correlate dates of any outages with dates of errors.

File a Cloudflare Support Ticket if you are experiencing increased page download times after enabling Cloudflare. Please include the diagnostic output that the process below will generate in your inquiry.

1.    Login to your Google Webmaster Tools account and navigate to the Health section of the affected domain.

2.    Click, Fetch as Google in the left hand navigation.

3.    Type, cdn-cgi/trace in lowercase in the form field and then click FETCH.  

4.    Click, Success under the Fetch Status column. 

5.    Copy all of the text in the grey area and submit this to Cloudflare Customer Support.

Over the last 24 hours, Googlebot encountered a number errors while attempting to retrieve DNS information for your site. The overall error rate for DNS queries for your site is percent.

Most DNS failures will be handled gracefully by browsers and your visitors will not be impacted. Steps for further troubleshooting will vary depending on how your Cloudflare service is configured. Cloudflare can be enabled either through usage of a CNAME or through zone delegation. If you are using Cloudflare through a hosting provider, you are likely using a CNAME setup.

To verify which configuration you’re using, open a terminal and execute the following command. Replace www.example.com with your domain registered with Cloudflare.

dig +short SOA www.example.com

If you get output from the command similar to the text below, you are using a CNAME setup. Please proceed to the CNAME section.

example.com.cdn.cloudflare.net.

If you get output from the command similar to the text below, you are using Cloudflare through zone delegation. Please proceed to the Delegation section.

josh.ns.cloudflare.com. dns.cloudflare.com. 2013050901 10000 2400 604800 3600

In a CNAME setup, your hosting provider is primarily responsible for your DNS service. In order to investigate DNS errors, you will need to contact their technical support team. Provide them with the date that Google encountered DNS errors to help with their investigation. Additionally, you should view the Cloudflare System Status page and observe any network outages on the date the errors were encountered by Google.

At all plan levels, Cloudflare will provide at least two authoritative name server records for your domain. Browsers and operating system APIs handle failure of all but one of these nameservers gracefully without exposing an error to your visitors. If Google reports DNS errors to you often, please file a Cloudflare Support Ticket and provide the date and time that Google observed the errors.

Follow the following steps below to export crawler errors as a .csv file from your Google Webmaster Tools Dashboard. Include this file when reporting errors to Cloudflare Technical Support.

1.    Login to your Google Webmaster Tools account and navigate to the Health section of the affected domain.

2.    Click, Crawl Errors in the left hand navigation.

3.    Click, Download to export the list of errors as a .csv file. 

4.    Attach the downloaded .csv file to the Cloudflare Support Ticket.