Search engines

Patrick Chappatte

Search Engines
Google
Directories
Invisible Web
Topic-Specific Engines
Wikipedia

Search Engines

There are two basic ways that search engines get their listings.

1. Authors may submit their own Web pages.

2. Spiders and crawlers (automatic software) scan the Web, newsgroups, and other Internet sources to see how many times a word appears on a page or how many other pages are linked to it.

Not all search engines are created equal. The way they collect and display results can make a big difference to the speed of your search. Each of these sites provides a novel approach to scanning the Web.

• Best search tools This chart compares search engines, meta-search engines, and subject directories.

• Topix.net News headlines from more than 9,000 online sources, sorted geographically and by thousands of categories.

• A9 is a new search site from an Amazon.com subsidiary. A9 takes Google’s search results and adds some useful personalized features.

• Teoma (search engine that supposedly rivals Google)

• The Kartoo search engine is all about aesthetics and personalization. Instead of presenting search results as a simple list in order of relevance, Kartoo displays them as a loosely organized flow chart. Roll your mouse over a particular Web site, and lines appear connecting that site to the sites to which it links. The results can also be personalized to reflect past searches—if you’re willing to let Kartoo track such things. This engine isn’t recommended for those with low-speed Internet access, however, as the site features some amusing little animations that could slow down your search.

• Vivisimo is a bit like a mix between Google and the old Yahoo directory approach. Results are displayed in the typical list format but are also displayed along the left hand side of the page in clusters. A search for the word “oatmeal,” for example, will organize the results in sub-categories, such as recipes, soap, peach, bread, and Quaker. Those who appreciate clusters might want to download the Vivisimo tool bar. Once installed, the toolbar resides in your Web browser for instant access. Each listing also includes information on how that site is ranked by mainstream search engines such as Google, Lycos, MSN, and Looksmart. Another favorite feature is Vivisimo’s preview pane: Simply click on the “preview” link that accompanies any entry, and that Web page will appear within the Vivisimo results page.

• The Librarians’ Index to the Internet, or LII, has a motto: “Information you can trust.” Its searchable directory contains only Web sites that have been personally approved by at least two librarians. Each listing is also annotated by one of the people who felt it was worth including. While you might not have heard of the LII, the search tool began as the bookmark file of a librarian back in 1990. Today, it includes more than 15,000 online resources. If you're a real infohound, you might want to sign up for the LII newsletter that profiles dozens of new finds every week.

Jump to top

Google

• If you’re gaga for Google, try supple Soople. It makes using Google's most advanced features so easy that even your mother could use them.

• Google Local Beta pinpoints online local resources on maps.

• Google Scholar searches articles and citations from scholarly literature. This could help you with research papers.

Other Google options

Windows-users: The free Google Desktop download puts a fast, Google-like searching ability in your PC. It will index and search all your Word, Excel, PowerPoint, Access, and text files as well as Outlook e-mails. You can use it often to find stuff hidden on your hard drive. Picasa 2.0, another free Google download, is an image viewer/organizer that finds all the images on your hard drive and catalogs them into easily-changed “albums.” In addition, it’ll let you edit the images easily without needing PhotoShop.

Jump to top

Directories
A directory is a database of Web sites collected and sorted into categories by humans. Yahoo! and other directories group Web pages into subject categories, such as archaeology or the Olympics. The most efficient way to use a directory is to start with a topic and drill (click) down through various categories until you find what you’re looking for. This is a great way to find entire Web sites on subjects such as Shakespeare or Siamese cats. It’s not a good way to find your grandfather’s Web page or a current theory of black holes that is one small part of a scientist’s research paper. Since directory listings are compiled by people rather than by automatic crawlers and spiders, a Web site will probably not be listed unless its owner has registered it with the directory. Compare with search engine.

Jump to top

Invisible Web
Direct Search, maintained by renowned librarian Gary Price, offers a way into the so-called Invisible Web. Most popular search engines use automated software to scan the Internet, but these Web crawlers can't access all kinds of files. As Price and colleague Chris Sherman explain in their book, The Invisible Web, crawlers can’t find some files because they simply aren’t linked from other accessible files. There’s no visible trail leading to these documents. Others files, such as the animated ones made with Macromedia’s Flash software, can’t be easily read by machines so their contents can’t be properly indexed. Many more pages are effectively locked away in databases: Even if the database is free to the public, the Web crawler simply has no way to access the pages within because of the way the interface is set up. Direct Search provides access to many of those otherwise hidden pages.

Jump to top

Topic-Specific Engines
A great way to make your search more efficient is to start with an engine tailored to your topic. The sites listed below show just how eclectic those engines can be.

• Bartleby.com is a terrific compendium of quotes from source literature—Oxford Shakespeare, King James Bible—and tomes like Roget’s and Bartlett’s.

• Census Bureau’s American Factfinder allows you to get detailed demographic information about any location in the U.S., from state down to neighborhood.

• Reporter's Desktop provides one-stop shopping for search engines, people finders, dictionaries, government links and more.

• Scirus.com is a great starting point for scientific information. An international scientific advisory board oversees Scirus to ensure it meets the needs of scientists. According the company that runs Scirus, Elsevier B.V., the engine searches 167 million science-specific Web pages. That includes a wealth of scientific papers saved in special formats that are overlooked by other search engines. Eliminating non-science sites can make many searches far more efficient. If you're searching for information on oxygen, for example, you probably aren’t interested in a link to Oprah’s Web site by the same name.

• Topix.net, created by some of the founders of the Open Directory Project, calls itself the Internet’s largest news site. It pulls together articles from more than 7,000 media outlets, ranging from small-town papers to the BBC. The selection on most of the site is very U.S. centric, however. You can only personalize the site for local news from American cities.

• Go to UrbanLegends.about.com to check out a suspicious e-mail; former journalist David Emery does a really good job posting items soon after they break online.

• Run by Barbara and David Mikkelson, snopes.com is much bigger than UrbanLegends.about.com. What’s New is the best way to keep abreast of urban legends across a wide range of cultural, political and business arenas.

Jump to top

Andrew Lih's Thoughts on Wikipedia
Andrew Lih is director of technology at the Journalism and Media Studies Centre at the University of Hong Kong.

Open content
Wikipedia is an Internet-based, volunteer-contributed encyclopedia that has become a popular reference. It has thousands of international contributors and is the largest example of an open content wiki. (The Hawaiian word for “quick,” WikiWiki, is the basis for the name.) The goal of Wikipedia was to create an encyclopedia that could be shared and copied freely while encouraging people to change and improve the content. Each and every article has an “Edit this page” button, allowing anyone, even anonymous passersby, to add or delete any content on the page. What would surely seem to create chaos has actually produced surprisingly credible content which has been evaluated and revised by the thousands of visitors to the site.

The project was started by Jimmy Wales, head of Internet startup Bomis.com, after his original concept of a strictly controlled, Ph.D-edited free encyclopedia ran out of money and resources after two years and only a few hundred articles. Not wanting the content to stagnate, he put them on a wiki Web site in January 2001 and invited visitors to edit or add to the collection. It became a runaway success. In the first year it gained a loyal following, generating over 20,000 articles and spawning over a dozen language translations. After two years, it had 100,000 articles. At the three year mark, it exceeded 200,000 articles in English and 500,000 articles in 50 languages. Every day, nearly 2,000 articles are added in various languages. 

Keeping it social and neutral
What could possibly allow this completely open editing system to work? Because wikis enable users to track the status of articles, review individual changes and discuss issues, they function as social software, thus facilitating communication and collaboration with other users. A wiki also tracks and stores every version ever edited, so no operation is ever permanently destructive. With regard to malicious contributors, in a wiki it takes much more effort to vandalize a page than to revert an article back to an acceptable version. While it may take five or ten seconds to deface one article, it can be quickly undone by others with just one click of a button. This crucial asymmetry tips the balance in favor of productive and cooperative members of the wiki community, allowing quality content to emerge.

However, technology is not enough on its own. Wales created an editorial policy of maintaining a neutral point of view (NPOV) as the guiding principle. “NPOV is an absolute non-negotiable requirement of everything that we do,” he says. According to Wikipedia's guidelines, “The neutral point of view attempts to present ideas and facts in such a fashion that both supporters and opponents can agree.” Inspired by this policy, the grassroots project has confronted the same great issues facing modern newsrooms — sticking to the facts, attributing sources, maintaining balance and applying rules uniformly, such as when to use the word “terrorist,” or evaluating what constitutes a cult or a religion.

So far, the effort has created numerous reference-quality articles as wide ranging as the Hutton Inquiry, algorithms, social history of the piano, origins of the American Civil War and severe acute respiratory syndrome. As its quality has improved, news publications have increasingly cited Wikipedia on subjects such as Wahhabism, crony capitalism, folk metal, British honours system, Abdul Qadeer Khan and extinct animals. It has even been used in litigation. In July 2003 a Wikipedia article on profanity was cited in a motion to dismiss a case in a Colorado court.

Teaching
Instructors at the University of Hong Kong's Journalism and Media Studies Centre use the Wikipedia in undergraduate and graduate journalism classes to teach the skill of writing dispassionately for an international audience. By collaborating online with others, students not only interact with each other when writing, but get advice and corrections from complete strangers around the world within minutes of making contributions to the Wikipedia.

Future
Wikis are just starting to receive recognition for generating credible collaborative content. Perhaps the toughest part of Wikipedia's future is how to manage its own success. While Wikipedia has recorded impressive accomplishments, its articles have a mixed degree of quality because they are, by design, always in flux and always editable. That reason alone makes people wary of its content. But first-time visitors are typically impressed with what the community has developed, considering the decentralized nature of the effort and the usefulness of its content.

Jump to top