ZA-WWW, 2011 Conference

Font Size: 
A comparative analysis of search engine indexing time
Herbert Zuze, Melius Weideman

Last modified: 2011-08-27

Abstract


Internet usage is dramatically increasing daily, as is Web development which is enhanced by the emergence of new Web technologies. Social networks are continuously dominating our Web interactions with Facebook, Twitter, LinkedIn and MySpace drawing a considerable number of Internet users. However, the dominance of the Web as the epicentre and pool of both important and useless information have effected competition in industries like e-business.

The Internet has enhanced interactions and relationships between Small to Medium-sized Enterprises (SMMEs) and their customers as the Internet offer many more choices to the customers. However, it takes some time for a website to be indexed and appear on the search engine result page. The indexing process involves the reading and recording of the weight-carrying words in a search format to an index file by a crawler. This results in the webpage being discovered by users on a search page of a search engine.

A user can also submit a website manually for indexing. Crawlers visit a website, record all the words on the pages, and note links to other sites. They track links from one page to another and indexes everything they come across on their way. Nevertheless, they do not see images, Flash movies, JavaScript, frames, password-protected pages or directories. The index file is updated regularly, either by human editors or by crawlers. Search engines create a map of the Web by indexing webpages according to keywords and then by building those into a database that links page content to keywords and URLs.

After an extensive literature survey conducted, empirical evidence indicated that the indexing period for webpages is not fixed but varies according to the crawlers’ visitation.  In a recent two phase experiment done on five websites’ homepages to determine their indexing time of Google, Yahoo! and Bing, the experiment proved that there is a relationship between keyword density and indexing time as far as Yahoo! and Bing is concerned. Yahoo! and Bing favour sites with high keyword density when indexing, compared to the ones with a low keyword density.

During the first experiment of the three search engines, the shortest indexing waiting time was five days and the longest was 33 days. On the second experiment, the shortest waiting time was 19 days and the longest waiting time was 29 days. However, according to this study, a period of approximately 15 days is a reasonable average waiting time.

It is clear as mentioned by the scholars and the experiment done that there are different views in respect to the time it could take for a webpage to be indexed. This was characterised by time ranging from a day to three months if the correct procedure of submission is followed. It is possible that the Google sandbox effect plays a role in these experiments, and this was also investigated.


Full Text: PDF