Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index.

We use a huge set of computers to fetch (or “crawl”) billions of pages on the web. The program that does the fetching is called Googlebot (also known as a robot, bot, or spider). Googlebot uses an algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site.

Google’s crawl process begins with a list of web page URLs, generated from previous crawl processes, and augmented with Sitemap data provided by webmasters. As Googlebot visits each of these websites it detects links on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.

Google doesn’t accept payment to crawl a site more frequently, and we keep the search side of our business separate from our revenue-generating AdWords service.

Author: HassanTech

Related posts:

1. A new Google penalty? What can you do against it? Webmasters in online forums reported an unusual Google behavior. New web pages of some sites are indexed with a longer delay. Is it a new...
2. How long does it take to get top rankings on Google? Many people who start a website think that it is possible to get high rankings on Google within a few days. Unfortunately, this is not...
3. What to do when your Google rankings have dropped Has your website lost its rankings in Google? Did your rankings drop or did your website vanish completely from Google? In that case, your website...
4. How to create trust for high Google rankings Trust is a major factor in Google’s ranking algorithm. Some websites can get high rankings on Google for nearly any content they publish while other...
5. How Google analyzes the top keywords on your website Source : Axandra.com Google recently filed a patent application that deals with the keywords that Google finds on your web pages. Google’s new patent filing...
6. The effect of your URL structure on your Google rankings Does Google care about the position of a web page on your server? Does it make a difference if a web page is in the...
7. It’s official: Google uses page speed as a ranking factor Last Friday, Google announced that they started to use site speed as one of the 200 signals that influence the position of a website in...
8. How the bounce rate of your website can affect your Google rankings Bounce rate is the percentage of single-page visits (i.e. visits in which the person left your site from the entrance page). Bounce rate is a...