Google

Have u ever experienced the use of Google? definitely nobody hasn't if you are reading this site. This article explains the mystery behind this powerful serch engine in the Internet and we try to digest the features provided by Google, talking about both its strengths and weakness.

What's Google About?

With the continuous growth of the Internet, people are now allowing to do everything online. You can look for a fine restuarant for your wife's birthday tonight, and you can even look for your first love from here. Actually, you can now do everything that you haven't thought of. As these need drives on, search engines are now more advertising oriented, changing from academic to commercial of its nature. Therefore, search engine has play an important role in information retrieval in your daily life. Enpowered with the search engine, people can look for any information as need. Google is a search engine designated to cope with these challenging task of 1. handling fast crawling for gathering web document from this dynamic Internet. 2. storing indexing information efficiently for fast retrieval. 3. improving search quality for your query relevancy. 4. handling query with fast response time. The following articles give you a general overview of what Google is about?

It's features

In order to cope with different challenging tasks, Google serach engine has two important features to improve your query relevancy. First, it makes use the link structure of the Web to calculate a quality ranking for each web page, and this ranking is called the PageRank of the page. Second, Google uses link to improve the search quality of the results.

How to calculate Page Rank?

In Google, the page rank score is the used for ranking for every query. If you got a highest score, then you rank high in the query. The Page rank of a web page is calculated by forming a citation (link) graph of the web page in the Internet. This page rank gives an objective measure of its citation importance that represents well with what people's subjective idea of pages' importance in the Internet. With this intuitive idea, page rank can be considered as a model of user behavior of the Internet community. The page rank of the page is calculated as follows: We assume page A has pages T1....Tn which point to it (i.e., are citations). The parameter d is a damping factor which can be set between 0 and 1. We usually set d to 0.85. Also C(A) is defined as the number of links going out of page A. The PageRank of a page A is given as follows: PR(A) = (1-d) + d(PR(T1)/C(T1) + .... + PR(Tn)/C(Tn))

FAQ of Page Rank

What is Page Rank?

The Page Rank is the score ranking by the Google search engine. The score value is from 0 to 10. The highest score you get, the high rank you get in the search result.

Is Page Rank for the whole site?

No!. Google calculates each page with a page rank.

How Page Rank (PR) is determined?

Basically, the PR is calculated based on the number of incoming links and the quality of the incoming links. The higher number of incoming links and the fewer number of incoming links, the more PR is passed to your page.

You can refer the calculation of Page Rank here

Is Page Rank fairs?

Theoretically, the Page Rank used by Google is a voting the system. More pages pointing to your site, it is just like people vote your page. The more vote you get, the highest rank you get.

Does the number of outbound links affect my PR?

No!. the number of outbound links only affect the PR received to the pointed page.

Does regular update of my content boost up my PR?

Not directly, the PR is affected by the incoming links to your page. But regular update of your site may introduce more incoming traffic and links, so increasing your PR.

How can i know the PR of my page?

You can download the Google toolbar. The PR is shown on a gauge on the toolbar. But it is not considered a reliable indicator of the current PR.

Is Anchor Text important?

In the Google search engine, it treats the text of the link in a special way for imporving search quality. In most search engines, it uses the text of link within the page for search result, more, Google associates this link to the page in which it points to. First, the third party provide more accurate descriptions to the page where it is pointed to. Second, anchors may exist for pages that are not text-based, like images, programs etc. so it provides more information for the crawler to crawl. Strengthened with this feature, Google can analysis more accurate of the web page's content from other third party's anchor text.

An Architecture Overview

In Google, there are three main part in its architecture. First, there is a web crawler to download web page from the Internet. This is done by the URL Server that sends a list of URL to the crawler for fetching. The fetched page are then stored in the Store Server which compresses and stores the web page into a repository. For every web page, there is an docID associated with. | URLServer | --> | Crawlers | --> | Store Server | --> | Repository | Second, there are an indexer and sorter then performs the indexing function. The indexer reads the document and then parses them. Each document is converted into a set of word occurences called "hits". The hit records the word's properties like position, font size etc. The indexer then distributes these hits into a set of "barrels", creating a partially sorted forward index. The indexer also parse out the information from the link forming the anchor files which contains the information of which page connects to which page. The sorter then takes the barrels and resorted them to generate the inverted index with the wordID. | Repository | --> | Indexer | --> | Anchor, Barrels (Forward Index) | | Barrels | --> | Sorter | --> | Inverted Index | Third, the URLresolver then responsible for converting the URL to absolute URL and finally to docID, forming a pairs of docID with link associated with. This link database is then used by calculating the PageRank of each page. | Anchor | --> | URLresolver | --> | Links | --> | PageRank | Finally, there is a program called DumpLexicon, it takes this inverted list together with the lexicon produced by the index and generates a new lexicon for the query. The searcher uses the lexicon built by the DumpLexicon and the PageRank to answer the query.

Crawl the web!

A web crawler (also known as a web spider or ant) is a program which browses the World Wide Web in an automated manner. Google uses a distributed crawler for fetching web document from the Internet. A single URLserver serves a list of URLs to a number of crawlers (around 3 at the same time). The crawlers download the web page and then send them to a store server. The store server then compresses and stores the page in the repository.

What does Repository store?

In Google, the respository contains full HTML of every web page fetched by the crawler. Google uses the zlib (RFC 1950) for compressing the data with the trade off of compression ratio. For every web page, it is stored as a packet with the following format. | docid | encode | url-lenght | page-len | url | page | The respository plays an important role for the origin of the information sources in the search engine. Data can be re-built from this database.

Your searching keyword - Lexicon

Lexicon is a word of Greek origin meaning vocabulary, and this is what your query in the search engine. In Google, the lexicon is a database of words used for search query from the user. This database contains around 14 million of words. For every lexicon, there is a wordID in associated with it. | wordID | lexicon |

The breakdown of your page - Hit lists

After the web page has been crawled into the repository, the indexer parses through every web page and break down it into a logical structure called hit lists. A hit list respresents a list of word occurence in the web document fetching from the Internet, it records the position, font and capitalization information. There are two main types of hits: fancy hits and plain hits. Fancy hits include hit occurring in a URL, title, anchor text, or meta tag. Plain hits include everything else. In the barrel, the indexer builds up a forward indexer for every web page with a list of hits associated with it. With the docID, the search engine can query what information of that page composed of with ease. Example: a list of doc in the barrel with forward index | docID | | wordID | no-of-hit | hit, hit, hit | | wordID | no-of-hit | hit, hit, hit, hit| | null wordID | | docID | | wordID | no-of-hit | hit, hit, hit | | wordID | no-of-hit | hit, hit, hit | | wordID | no-of-hit | hit, hit, hit | | null wordID |

Building an Inverted Index!

The inverted index are from the same barrels, but it is built by the sorter. For every lexicon, there is a pointer pointed into barrel that the wordID falls into and the wordID points to a list of docID, and finally to a list of hit lists. | Lexicon | --> | wordID | --> | docID | --> | hit list | With this inverted index, Google can locate the document from the lexicon submitted by the searching query.

How does the Ranking System work?

When you submit a query to Google, its search engine does the following 1. parse your query and extract the lexicon out of your query. 2. map your lexicon to the wordID. 3. seek the start of the document list for every word in your query. 4. scan through the doclists until a document that matches all the search terms. 5. compute the rank of each document. 6. if we are not at the end of the doclists, go back to step 4. 7. sort the documents that match the query and returns. however, once a certain amount of the documents are found, say 40000, Google goes to the final step immediately, such that the upper bound of the response time of the serch engine can be guaranteed. Let the things be simple, we quote an example here. For example, we will search for a single word query, say "happy", the search engine scan the doclists pointed by wordID:happy. when the search engine get the word "happy" in the document, it is weighted with a factor according to its type, like title, anchor text etc. This method is to avoid that no particular factor can have much influence to the result. Finally the above score is combined with the PageRank calculated from the Links to form the final result. For multiple word query, it is much more complicated that what we think, beside each type-weighted factor, it has a word proximity weighted factor in consideration. Actually, these parameters are the real mystery behind Google, and it is expected changing all the time to adapt changes.