Sometimes you may need to decrease or increase Google crawl rate when experiencing problems with your server.
How to increase Google crawling rate
Before we answer the How to increase Google crawling rate question, we should explain what determines how often or with what frequency Google crawls your web site.
* Relative importance of your site,
* relative importance of your site,
* and relative importance of your site.
The more important your web site is, the more often Google will crawl your web site. This is pretty obvious. A large content-rich portal needs to be crawled by Google more often because such portal is likely to add new content often while a low-quality outdated web site would be a waste of time if Google crawled it frequently.
Relative importance of your web site is measured by two metrics:
* Number of links to your web site
* Number of web pages within your web site
Links: The first measure is quite obvious. If Google does not find enough links pointing to your web site, then Google does not think your web site is very important. The more quality links you have pointing to your web site, the higher Google values your site, and crawls it with higher crawl frequency.
How do we define a "quality" link? Links from bad neighborhoods can hurt you. A quality link is the one that is from a page with relatively high page rank and yet better from a site with similar content. Links without the nofollow attribute work the best. See here for more information about the nofollow tag: nofollow.
Pages: If your site has only a few pages, Google will not crawl them very often. Google is likely to increase crawl rate or frequency when it finds large and constantly growing sites.
How can I increase Google crawl rate
Now that you know what it takes to increase Google crawl rate, you are probably going to write a few more pages and build some links. There is something more you should know.
If you have a Google Webmasters Tools account (http://www.google.com/webmasters/tools), you can increase Google crawl rate through the account in the Dashboard - Settings screen. This however applies to web sites that have been running for some time only. If your web site is brand new, Google assigns some default
Note that you can’t force Googlebot to visit you more often – what you can do is to invite it to come. Possible measures to take to increase the crawl rate may include:
1. Update your content often and regularly (and ping Google once you do) – well, an obvious one, so not much to describe here; in a word, try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).
2. Make sure your server works correctly: mind the uptime and Google Webmaster tools reports of the unreached pages. Two tools I can recommend here are Pingdom and Mon.itor.us.
3. Mind your page load time: note that the crawl works on a budget – if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.
4. Check the site internal link structure: make sure there is no duplicate content returned via different URLs: again, the more time the crawler spends figuring your duplicate content, the fewer useful and unique pages it will manage to visit.
5. Get more back links from regularly crawled sites.
6. Adjust the crawl speed via Google Webmaster tools.
7. Add a sitemap (though it’s up for a debate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it).
8. Make sure your server returns the correct header response. Does it handle your error pages properly? Don’t make the bot figure out what has happened: explain it clearly.
9. Make sure you have unique title and meta tags for each of your pages.
10. Monitor Google crawl rate for your site and see what works and what not:
Read more: http://www.searchenginejournal.com/10-ways-to-increase-your-site-crawl-rate/7159/#ixzz18Rs7Lsfy
Sign up here with your email
ConversionConversion EmoticonEmoticon