Understanding

WWW

Search Tools

The first type arranges internet resources in some sort of classifictory schemes: alphabetical, chronological, geographical, subject-oriented, or a combination thereof. Here are some typical examples.

The main function of these general listings is for easy browsing. Most of them provide searching as well. They require a great deal of human effort in terms of collecting, arranging, html coding and annotating of resources.

The second type, which we are concentrating on today, attempts to collect and index resources in a more automatic fashion. It does not require extensive human intervention. Searching, instead of browsing, is the main feature of this type of tools.

These search engines/tools have two components: collection and search. The collection (also known as automated robot Wanderer, Spider, Harvest and Pursuit) part roams internet sites, mostly www, gopher and ftp sites, brings back resources, sorts, indexes and creates a database out of them. The metaphorical definition of Lycos, one of the most popular search engines, sheds some interesting light on the collecting activity:

Lycos comes from Lycosidae, a cosmopolitan family of relatively large active ground spiders that catch their prey by pursuit, rather than in a web. They are noted for their running speed, and are especially active at night.

The search component concerns the end user. It is the interface between the human searcher and the indexed database of resources.

There are several factors that determine the success of a search engine, chief among which are: the size, content and currency of the database, the speed of searching, the availability of search features, the interface design and ease of use.

We are going to introduce some of the major search engines, comparing their features in terms of the success factors mentioned above, and pointing out their database characteristics. In doing so, we hope you will become more familiar with those tools, more aware of their tricks and features, and find what you are looking for with more comfort and ease.

• One of the pioneers in WWW searching
• Best of the Web '94 Award - Best Navigational Aid

• Size of database: unknown, claims to serve 3 million URLs to 2 million users a month.
• Four types of search databases:
  1. citation hypertext
  2. citation addresses (URL)
  3. HTML titles
  4. HTML addresses
• Boolean: AND, OR
• Search Limit (maximum: 50,000)
• Instructions and Examples

• The definition of the types of search databases difficult to understand
• No content of webpages in database
• Database is rather small, not updated for quite a while.
• This search engine is included here mainly for historical interest. Not recommended for daily use.

• Purchased by America Online

• Database size
• Fast
• Indexed Document/Content Search
• Breadth-first Search
• Boolean: AND, OR
• Ranked
• Stopword ignored, no penalty
• Convert regular Plural to singular (classics or classic, but not children to child)
• Technical Description and a paper

• Excellent for quick searching for a top level webpage.
• No need to change the number of maximum return results (You can retrieve "more" later).
• Only titles of webpage accompany the search results displayed.
• Difficult to achieve precision in searching
• Broad coverage of servers but less in-depth information

• Now Lycos, Inc. a subsidiary of CMG Information Services, Inc.
• Acquired Point Communications (Review and ratings of web sites)
• Many Business Partners, including Microsoft

• Database size: Claims 91% of web sites, or 10 million webpages. Another Count
• FAQ
• Indexed documents include: title, headings and subheadings, 100 most significant words, first 20 lines of the document and its size in bytes and number of words.
• Lycos Search Language
• Fast and flexible
• Simple search (by default "OR" "Loose Match" and "Standard Results")
• Search options can be changed
• Search results ranking includes adjacency and number of occurrence

• Adjust search options before doing a multiple term search
• Add a . to stop stemming.
• Search results display can be very confusing.
• Database not purged for a while, contains much out-of-date information.
• Search term match options range from "loose" to "strong" but no indication what those options mean.
• Each time the homepage is accessed, the ad page changes, even if you use the back arrow/button.

• It has commercial databases, called Infoseek Professional

• Help on Infoseek Guide
• Includes WWW pages, Gopher/FTP sites and 2 weeks of newsgroups.
• Search results ranked according to relevancy.
• Free search limited to 100 hits.
• Proximity, unordered proximity, and "require" operators. No boolean operators.
• Case Sensitive
• Cross-References in search results (Similar pages)
• A comparision

• Put search terms in [ ] to achieve the equivalent of "AND"
• Use Capitals where applicable
• No database size information available
• No truncation
• Many small rules to achieve precison

• Search engine used by Yahoo.

• A sophisticated search engine
• Database Size
• Simple Search: exact phrase, word (boolean "AND/OR")
• Search Results: Go to; Sample matches in context; Find similar pages. As well as summaries. Also, dates, matches, size.
• Power Search: Search only parts of web pages (Subject, Title, Hyperlink etc), combine searches using Boolean (AND, OR, NOT) and proximity (NEAR, FOLLOWED BY) operators.
• Weighted Search: Rank by accurrence count or presence/absence.

• Space after a word stops stemming (truncation implied?) "rcsd" and "rcsd " make a big difference.
• Excellent Online help
• Weighted Search is not very intuitive.
• Server frequently refuses connection.

• Find out excite for Web Servers

• Database size: 1.5 million pages. Read about their way of Measuring the size of a database
• Concept search and Keyword search.
• Online Handbook
• Confidence ranking and site sorting
• Similar page searching

• Concept searching is not well-defined, though an interesting idea.
• Not easy to achieve accuracy in searching.
• Finding similar pages often results in dubious links.
• Summaries do not seem to be much different from the first paragraphs.

• Scalable

• Database size: 2.8 million documents. An interesting comparison of ways of counting.
• Fast (due to its scalability?)
• Simple interface.
• Ranked

• Prefix search terms with + to achieve "AND".
• Very few search features. Hard to archieve precision.
• Database may have a large number of documents, but lack broad coverage of web servers.
• Not enough online documentation.

• Read about its Hardware, Software, Wetware and Bragware

• Database size: 8 billion words found in over 16 million Web pages.
• A full-text index of over 13,000 news groups updated in real-time.
• Simple and Advanced Queries search either the webpages or the news groups.
• Output in three formats.

• Excellent online help for both simple queries and advanced queries
• By far the largest, fastest, and most sophisticated search engine.
• You do need to spend some time to learn all the bells and whistles.

• URL: http://cage.cs.colostate.edu:1969/
• FAQ
• HELP

• URL: http://www.infomkt.ibm.com/
• FAQ
• Free but requires login.

• URL: http://metacrawler.cs.washington.edu:8080/index.html
• HELP

• URL: http://www.designlab.ukans.edu/ProFusion.html
• HELP