Search engines use software spiders to crawl the web and index static web pages that do not require password access. Every word of each web page found has the potential to be indexed. Unscrupulous web masters might embed hidden or deceptive text within their web pages to boost their placement in search results. View the source of this example or try this search using the word Costco and look for the 'Soulmate Trading Company'
Google adds link popularity to its indexing my checking to see how many other web pages link to the page it is idexing. This has lead to instances of 'Google Bombing' -- example. Articles about the concept of Google Bombing
With the proliferation of weblogs, it has become easier then ever for everyone to publish on the web. Because search engines cannot distinguish based on content value, they are indexing more and more and more and more junk. example
Search engines are not able to index web content that is located within databases [for technical reasons] and web pages that contain protected information.
Academic directories, sometimes known as Virtual Libraries, provide searchable lists of reviewed web sites. The reviewers for these directories are usually experts in their subject area and typically have advanced degrees. Web sites will only be listed within Virtual Libraries if the content within the site is considered outstanding.