Search engines imply they search the web using software robots called spiders. They index either the purported contents of sites they find, or they index the actual contents of sites they find.
Advertising in Search Sites
Many well known search engines lie about search results. The search results at the top are there because advertisers paid for them to have a good position, or because they are owned by the search site. Sometimes these paid sites are listed as Featured Sites or Spnsored Sites. Basically, you can't trust the results. They are trying to match users to advertisers, rather than find the result you want.
MSN list advertised sites first, then their own sites, then more advertised sites from Ask Jeeves. AOL has been using Overture, which provides advertisoing sites. AltaVista, Ask Jeeves, Lycos, Sprinks, and Yahoo show advertised results, without being real clear about they being advertising.
Many previous innovative search sites no longer rely upon their own search processes. Many smaller search sites use Inktomi or Overture (formerly GoTo) to do their searches.
Metasearch engines (which check multiple search sites) like MetaCrawler, which were once good ideas, are now almost no use. They generally mix up advertising and real results making it much harder to tell which is which.
Google is still probably the best of the whole bunch. It uses its own spiders and page rank system (based on the number of links to a site). It does have advertising, however the advertising is clearly labelled, and is in a different colour to search results.
Featuring in Search Results
- Relevant Domain Name
- Your domain name should indicate your business. A large company can afford to just use, say ibm.com A small business needs to use something specific, like airliebeachhotdogs. Make sure you have a specific permanent IP address for your site, not a shared IP address.
- Title
- Brief, readable and relevant title tags, specific to each and every page on your site. Start with your best search term. If you have access to server search logs (a business should ensure this) check what search terms are most common for your site. Make sure you have pages aimed at these terms.
- MetaTags
- I like to use an accurate and relevant set of keywords and also do a decent description metatag, usually along the lines of my title tag. Google doesn't use the keywords tag at all due to false keywords on many sites, however other search engines still use it.
- Content
- I prefer detailed text content and lots of it in every page. Many web sites have sparse content, and are basically a poster. I don't think these rate well with search engines or viewers. I like to have key terms in headings high on the page. Summarising the page in the first sentence (if you can manage it) seems a great idea. I don't like having any sort of scripting code high up in a page. I leave the page appearance to an external cascading style sheet, to avoid a slow loading page full of tag soup. I think search engine spiders work better with clean, lightly tagged sites.
- Site Map
- Site maps or detailed index pages are good for both viewers and for search engines. Link related pages to each other.
Using Search Engines
I'd suggest anyone wanting better results find one or two search engines they are happy about, and learn how to drive them properly. Learn about specifying enough terms, using quotes, adding Boolean restrictions, using task specific phrases. Remember also that sometimes it is quicker to phone, or to visit a library.
One of the best places to learn more is to visit Danny Sullivan's SearchEngineWatch, and keep up to date with what is changing.