What Is Indexing and Crawling

Crawling generally means taking out content without any sort of processing. Crawlers are basically dumb processors that supply the content from Web Servers to other processes. Web crawlers are computer programs that are used to browse the web in a predefined manner. These are basically    used to create a copy of all the visited pages for processing at the later stages by a search engine. Google generally uses a lot of computers to crawl many web pages. The program that performs crawling is known as Googlebot.

Indexing on the other way means to make sense out of the retrieved contents and then the processed result is stored in a document index. It is used to collect and store data so as to get a fast and accurate information retrieval. It uses an indexing algorithm which basically compares the related pages to each other according to their page rank and then these pages are arranged in a index starting from highest page rank. The two most important factors that affect the page rank are Keywords and Back links.

Some Factors that affect Crawling and Indexing are Site architecture, Links, content, and lastly domain importance. Domain importance plays an important role in Crawling process because more the number of sites that are linked to you. More will be the chances of your site to be crawled. Now we move towards indexing, generally 3 types of indexing are known and these are main, supplemental, and crawled but dropped. As the name suggests main indexing refers to main page results, supplemental indexing means that your results are not among the top 10 results whereas last type refers to that point when your page rank is dropped.