Most website owners have probably heard (and may have been worried) about some important Changes to Google’s Search System. “Caffeine” was originally announced in August 2009 and then released to a single data center for testing in November.
Understandably, the original announcement caused significant interest, speculation and even concern amongst the SEO community and webmasters in general – which has remained fairly constant ever since. Well yesterday, Google announced that Caffeine has finally been released across all data centres worldwide (yes, including Australia).
Google has also provided a much clearer description than ever before about what Caffeine is (and what it isn’t) and also what impact this very significant change is likely to have on your website.
So what is Caffeine exactly?
Caffeine is a change to Google’s indexing engine, and will not have any direct affect on rankings.
Previously Google spiders crawled websites in batches, and at the completion of the crawl, sent all the collected data off to be processed and indexed. It was not until this indexing was complete that the page would become available via search, so it could be a matter of hours (or even days) from when a page was crawled to when it appeared in the index.
Caffeine allows the indexing to be carried out much more quickly. Now, as soon as a page is crawled the data is passed on to be indexed – so pages may appear in search results, indexed within minutes or even seconds of being crawled for the first time. This means that brand new pages can theoretically be indexed almost as soon as they are published, and changes to already indexed pages are likely to be picked up (and affect rankings) much faster as well.
Remember though, that not all pages on your site will be crawled (and therefore indexed) every day. Google’s spider (Googlebot) tends to crawl a subset of your site each time it visits. Larger sites with lots of regularly updated content, and lots of backlinks are likely to be crawled much more regularly (and thoroughly) than small sites that are not updated very often.
What about Rankings?
Whilst Google have been at pains to declare that Caffeine will not directly affect Rankings, they have also announced that they can now start to accumulate extra data about your web pages, and this data will be used to help provide better search results.
As a part of the existing indexing process, Google already keeps track of such things as backlinks and anchor text. The new types of data they may start tracking is not clear, however, Caffeine provides Google with a lot more flexibility in what information they can start to associate with your site, so we can expect this to affect rankings over time.
In the short term, Caffeine will probably not affect your sites rankings at all. However, I expect that as they start to associate more data with your web pages, this could start to have a significant impact on rankings in the future.
That’s a whole LOT of Data!
Caffeine lets Google index web pages on an unprecedented scale. In fact, every second Caffeine is now processing hundreds of thousands of pages in parallel. Caffeine already takes up nearly 100 million gigabytes of storage in a single database and adds new information at a rate of hundreds of thousands of gigabytes per day.
What do I need to do NOW?
As long as you are pro-active in managing your site, the good news is that you probably don’t need to do much differently. However, anything that you can do to help increase the crawl rate of your website is going to help with your indexing, and ultimately, the more pages you have indexed (and the faster you can get them crawled), the more likely you are to be found via search. Therefore:
- It is more important than ever to get Links – Google relies on links to find new sites/pages and new content to crawl. Getting backlinks from authority sites and having a good internal link structure is very important.
- It is more important than ever to create new, fresh content – Googlebot loves nothing more than fresh, original, content. Regularly adding new content to your site is the best way to tempt Googlebot back (and to have it keep coming back). If you’ve not already considered it, adding a blog to your website, is one of the quickest and easiest ways for you to create new content (and Googlebot LOVES Blogs).
- It is more important than ever to have (and update) a sitemap – Googlebot uses your sitemap.xml file to help navigate your site. If you regularly update you sitemap, Googlebot knows what is new, what has been changed, and finds it easier to crawl and index your site.
If you are currently NOT doing any of these things, then you need to start an SEO Campaign as soon as possible. Caffeine is likely to give sites that are actively being optimised for search (SEO) an added advantage over those that are not.
Crawl Rate Optimisation
Although the activities associated with optimising the crawl rate of your site are similar to optimisation strategies associated with general SEO, there are some specific additional actions that can help you to better understand how search engines crawl your site – and take steps to improving your crawl rate, and with this latest Caffeine release, anything you can do to improve your crawl rate simply has to have a positive impact on how your site is indexed and ranked.
Please feel free to contact me if you’d like to find out more about how the crawl rate of your website could be better optimised.
About Darryl
