Once upon a time, I had the domain searchmee.com – I started playing around with writing web spiders around 1996, which is how I was able to create FindAnISP.com and gather the data. Partly just to see what was out there, I wrote a spider that searches the Web smartly as opposed to trying to index 100% of everything. This is based on the old 80/20 rule – that 80% of what people are looking for you’ll be able to find with 20% of the effort. I was doing this spidering from dialup in 1996 🙂 Needless to say, it took a while and kind of annoyed the ISP since I was online 24 hours a day, but clearly not just playing games to hold an idle connection open.
The early output of this was the “top 100” web sites by country or by category. Since I refused to censor the results, that got me kicked off of homestead.com for violating the TOS – this will shock you, but people look for porn on the web! So with like 120 different categories of web sites, having one page of the “top sex related web sites” got me banned – that’s when I decided it was time to have my own web server, and not be subject to the whims of a minimum wage clerk killing my entire web site because of a single complaint. Since I wasn’t paying for the service, I had no leverage and they had no interest in keeping me around. *deleted*
I got to the point where my results were comporable to Google’s as long as you were looking for something fairly general. The ability of mySQL to deal with doing a text search on millions of rows is however, not so swift. When I dropped paying the $450 a month for a “real” web server in a data center, I archived the data to my windoze box, and pulled the web site down, since the web server is running on a box with something like 35 GB total space.
Now that I have 800+ GB free, I may start playing around some more with this. It’s interesting some of the things you can find when you can see things like which web servers are located in the same facility with some government agency, for instance. I now have about a 29 Mb/sec download speed, so I can gather data a bit faster than my ~50k bps modem did.
The down side is this can destabilize the streaming radio web site – when you have a 15 GB table, doing anything to it takes a very long time and has a way of making the server run out of memory. I have no delusions of creating something better than Google or even approaching it.
You want random? Here are snippets of the first 15 records
| http://hollykirschke.blogspot.com/ | BlogThis! L’Afrique: From East to West My summer internship in Nouakchott, Mauritania Thursday, June 23, 2005 Hey Salam Alakum everyone!I am still alive here in the desert. I have seen a lot of different colors of sand and learned some important things lately, namely:1, whiping your butt–hum everyo |
| http://www.conferencebaden-baden.com/ | We will find Conference Venue and Meeting Venues in Baden-Baden. Also Video Conferencing Facilities Baden-Baden, IT and computer training venues Baden-Baden, Exhibition Venues Baden-Baden, Team Building Activities and convention Venues Baden-Baden. We also have details of all Corporate Hospitality |
| http://www.conferencehungary.com/ | We will find Conference Venue and Meeting Venues in Hungary. Also Video Conferencing Facilities Hungary, IT and computer training venues Hungary, Exhibition Venues Hungary, Team Building Activities and convention Venues Hungary. We also have details of all Corporate Hospitality and Event management |
| http://www.governor.state.tx.us/divisions/stategrants/ | Skip to main content. HomeRick Perry Gov. Rick Perry Duties & Requirements First Lady Anita Perry Governor’s Mansion Priorities Education Jobs Health & Safety Transportation Tax Reform Other Divisions Appointments Budget, Planning & Policy Criminal Justice Disabilities Committee Economi |
| http://edition.cnn.com/2006/TECH/internet/04/06/gardening.blogs/ | U.S. Edition Member Center: Sign In | Register Home Page Asia Europe U.S. World World Business Technology Science & Space Entertainment World Sport Travel Weather Special Reports Video CNN Pipeline What’s On Art of Life Business Traveller Future Summit Principal Voices Quest Revealed Talk Asia |
| http://www.palm.com/us/products/palmpilot/ | http://www.palmOne.com United States home stores products software business/solutions support ownerresources products handhelds Palm Z22 Tungsten E2 Tungsten E2 GPS Navigation Pak Palm TX mobile managers smartphones Treo 650 Treo 700w accessories overview cases & belt clips palm® persona premium c |
| http://www.imood.com/users/Crescendo | get an account! login internet’s mood mood indicators help contact us bookstore moodylinks imood forums imood home imood news Today: Sat Apr 29, 2006 log in to imood Want to let everyone see how you’re feeling? Come get an imood account right now! Name Crescendo Base Mood disillusioned Personal Moo |
| http://www.doctorcruise.net/forms/quotstd.htm | Request a Cruise Quote Thanks for your interest! We will not use the information you provide for any other reason than to generate your quote, and to follow up with you. Click Here to read our privacy policy. All of our quotes are custom-prepared individually. This allows us to escape the confines |
| http://www.peterme.com/archives/000662.html | peterme.com Links, thoughts, and essays from Peter Merholz. â–’ Urban walking – musin’ on cruisin’ | Main | Heading to North and South Carolina â–’ December 13, 2005 Notes from User Interfaces for Physical Spaces Yesterday I attended User Interfaces for Physical Spaces, a one-day workshop and field tri |
| http://www.oralteenpics.com/ | oral teen pics great hot horny girls gives head like a vacum cleaner click on the thumb to go to that paricular paysite Glory hole Sweetloads Wet Cream Pies Deep Oral Girls Cumshots n Facials Deep Oral Guys free Adult links Little Teen Bang :: hot petite teens get nailed by huge hard cocks in pussy |
| http://www.northnorfolkconservatives.com/ | Local Conservatives Working for all the Residents of North Norfolk Welcome to our site. Here you can find the latest news, information about us, what we are doing and how we can help you. We are keen to know how we can help you. If you have any comments then please feel free to e-mail us at info@no |
| http://www.injectionlocator.com/ | We subscribe to the HONcode principles. Verify here. ARE YOU ONE OF THE MILLIONS OF PEOPLE AFFECTED BY DIABETES? Are you using Insulin Injections to control your blood glucose levels? â–’â–’.. The â–’Injection Locatorâ–’ was especially developed for you! Diabetes itself is very well manageable when it is d |
| http://s77.photobucket.com/albums/j74/KurtEvans/ | home|login log in | join now | tools and tips | recent current album: KurtEvans – 56 pics Enjoy the latest stars pics – here 1 2 Url Tag Img Url Tag Img Url Tag Img Url Tag Img Url Tag Img Url Tag Img Url Tag Img Url Tag Img Url Tag Img Url Tag Img Url Tag Img Url Tag Img Url Tag Img Url Tag Img Ur |
| http://www.freewebz.com/eucharist/ms1.htm | !–webbot bot=”HTMLMarkup” endspan –> THE AMAZING SECRET OF THE SOULS IN PURGATORY [An interview with Maria Simma written by Sister Emmanuel of Medjugorje] This Story Fills A Void: One day, I read with great interest a book about the souls in Purgatory. It struck me so much because it related very |
| http://www.fp247.com/247_galleries/Whipped__Ass/3514.html | Return to Free Porn 24 7 Go to Whipped Ass now hot Adult Pictures – Whipped Ass whipped ass – free whippedass – whipped ass pic – lesbian whipped ass – Whipped Ass Pictures – Whipped Ass – Whipped Ass Whipped Ass – whipped ass video – whipped ass – Whipped Ass – ass basement whipped – whipped ass v
Here was the count by category last time I turned this thing loose, which seems to be 2007
It would probably be interesting just to see how few of the web sites from 2007 still exist – facebook, youtube, twitter and google have pretty much killed off the idea of a “personal web site”
There are only two real contenders in web servers (at least in 2007)
Apache: 2,618,433
Microsoft IIS: 728,249
Third place:
Netscape 19,104
Does your spider honor the “NoBot” tags?
As a matter of fact, yes!
In fact, I store the robots.txt files for future reference – I think I have a bit over a million of them