The Dark Web: 500x Larger Than The World Wide Web

The World Wide Web is big. Really big. As of July of 2008, Google found 1 trillion (that's 1,000,000,000,000) unique URLs on the web at once. The search engine has only indexed a fraction of those web pages (the last count I found was 25 billion in 2006).

But that's nothing compared to the "Deep Web" - a part of the Internet that is not easily accessible by search engines (for example, dynamically generated content that exists only momentarily). People have estimated that the Deep Web is several orders of magnitude larger than the "surface Web". There is, however, another part of the Deep Web that is more sinister: the dark side of the Internet used by criminals.

Andy Beckett of The Guardian wrote:

The modern internet is often thought of as a miracle of openness – its global reach, its outflanking of censors, its seemingly all-seeing search engines. "Many many users think that when they search on Google they're getting all the web pages," says Anand Rajaraman, co-founder of Kosmix, one of a new generation of post-Google search engine companies. But Rajaraman knows different. "I think it's a very small fraction of the deep web which search engines are bringing to the surface. I don't know, to be honest, what fraction. No one has a really good estimate of how big the deep web is. Five hundred times as big as the surface web is the only estimate I know."

"The darkweb"; "the deep web"; beneath "the surface web" – the metaphors alone make the internet feel suddenly more unfathomable and mysterious. Other terms circulate among those in the know: "darknet", "invisible web", "dark address space", "murky address space", "dirty address space". Not all these phrases mean the same thing. While a "darknet" is an online network such as Freenet that is concealed from non-users, with all the potential for transgressive behaviour that implies, much of "the deep web", spooky as it sounds, consists of unremarkable consumer and research data that is beyond the reach of search engines. "Dark address space" often refers to internet addresses that, for purely technical reasons, have simply stopped working. [...]

Michael K Bergman, an American academic and entrepreneur, is one of the foremost authorities on this other internet. In the late 90s he undertook research to try to gauge its scale. "I remember saying to my staff, 'It's probably two or three times bigger than the regular web,"' he remembers. "But the vastness of the deep web . . . completely took my breath away. We kept turning over rocks and discovering things."

In 2001 he published a paper on the deep web that is still regularly cited today. "The deep web is currently 400 to 550 times larger than the commonly defined world wide web," he wrote. "The deep web is the fastest growing category of new information on the internet … The value of deep web content is immeasurable … internet searches are searching only 0.03% … of the [total web] pages available."

Link