How the Search Engines Work

By Jeff

In a previous post, I talked about why every business needs a web presence.

In this post, I’ll describe the way search engines work. This will help illuminate the methods of getting your website found on the major search engines.

Crawling the Net

The search engines use small programs called “spiders” to crawl the web. This is how they find all the billions of pages that exist on the internet.

The internal workings of these spiders are not the subject of this post, as complex and technical mathematics is not a subject most people like to talk about.

So how, in a nutshell, do the spiders find web pages? I can give you the answer in one word:

links.

One of the main reasons that html is the best format for creating web content is that it is very easy for the spiders to parse, or read. This is how the search engines are able to return the title of a page, a description, and other information to people who input a query.

When the spiders find an html link tag, they are directed to the new site. They have found another page.

Indexing the Multitude

The second thing that the search engines do is to index every page they find into gigantic databases. This allows them to deliver search results very quickly.

One of the side-effects of our modern technology is that people expect to have their answers instantly. Waiting more than second for search engines to respond will quickly drive people to the competition.

So how are all these pages indexed?

There are a number of factors, the main ones being keywords, pagerank, and link popularity. Only the insiders who work for the search engines know the inner workings of the algorithms, so it’s impossible to say with exact certainty how they work, but these are the three biggies.

Of the three, link popularity and keywords are the most important. Pagerank has its uses, but is at best only a general indicator for comparisons.

What this Information Means to Your Website

Getting ranked in the search engines means getting traffic. How do you get ranked?

First, you have to get found. This means that the search engines must find your pages. To find your pages, there must be links to your pages.

Then the search engines must index your pages. There are a number of guidelines to getting indexed faster than if you just wait around for them to get to you, which I will cover in a separate post.

To get ranked in the results pages, you need to be indexed for certain keywords. This is why keyword research is so crucial. Anyone can get ranked for nonsense words like “kjsbdfksj,” but no one is going to be looking for that. If anyone is looking for it, I’ll bet you they aren’t interested in buying anything.

The other big piece of the pie is link popularity. Other things equal, the site with more links pointing to it gets ranked higher.