It is very difficult to predict what sites or kinds of sites or portions of sites will or won't be part of the Invisible Web. There are several factors involved:
Which sites replicate some of their content in static pages (hybrid of visible and invisible in some combination)?
Which replicate it all (visible in search engines if you construct a search matching the page)?
Which replicate none and must be searched directly (totally invisible)?
You often don't know if a page has a ? in its URL until after you've somehow found it (excluded by policy).
Search engines can change their policies on what the exclude and include.
Barker, Joe. Invisible Web: What it is, Why it exists, How to find it, and Its inherent ambiguity. Finding Information on the Internet: A Tutorial. 28 August 2003. UC Berkeley - Teaching Library Internet Workshops. 17 Nov 2003. http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InivisibleWeb.html
Posted by iachan on Monday, 17 November 2003 @ 9:57 PM