{"id":8002,"date":"2013-08-17T13:54:14","date_gmt":"2013-08-17T17:54:14","guid":{"rendered":"http:\/\/streamingradioguide.com\/startingover\/?p=8002"},"modified":"2013-08-17T13:54:14","modified_gmt":"2013-08-17T17:54:14","slug":"searchmee","status":"publish","type":"post","link":"https:\/\/streamingradioguide.com\/startingover\/?p=8002","title":{"rendered":"Searchmee!"},"content":{"rendered":"

Once upon a time, I had the domain searchmee.com – I started playing around with writing web spiders around 1996, which is how I was able to create FindAnISP.com and gather the data. Partly just to see what was out there, I wrote a spider that searches the Web smartly as opposed to trying to index 100% of everything. This is based on the old 80\/20 rule – that 80% of what people are looking for you’ll be able to find with 20% of the effort. I was doing this spidering from dialup <\/strong>in 1996 \ud83d\ude42 Needless to say, it took a while and kind of annoyed the ISP since I was online 24 hours a day, but clearly not just playing games to hold an idle connection open.<\/p>\n

The early output of this was the “top 100” web sites by country or by category. Since I refused to censor the results, that got me kicked off of homestead.com for violating the TOS – this will shock you, but people look for porn on the web! So with like 120 different categories of web sites, having one page of the “top sex related web sites” got me banned – that’s when I decided it was time to have my own web server, and not be subject to the whims of a minimum wage clerk killing my entire web site because of a single complaint. Since I wasn’t paying for the service, I had no leverage and they had no interest in keeping me around. *deleted*<\/p>\n

I got to the point where my results were comporable to Google’s as long as you were looking for something fairly general. The ability of mySQL to deal with doing a text search on millions of rows is however, not so swift. When I dropped paying the $450 a month for a “real” web server in a data center, I archived the data to my windoze box, and pulled the web site down, since the web server is running on a box with something like 35 GB total space.<\/p>\n

Now that I have 800+ GB free, I may start playing around some more with this. It’s interesting some of the things you can find when you can see things like which web servers are located in the same facility with some government agency, for instance. I now have about a 29 Mb\/sec download speed, so I can gather data a bit faster than my ~50k bps modem did.<\/p>\n

The down side is this can destabilize the streaming radio web site – when you have a 15 GB table, doing anything to it takes a very long time and has a way of making the server run out of memory. I have no delusions of creating something better than Google or even approaching it.<\/p>\n","protected":false},"excerpt":{"rendered":"

Once upon a time, I had the domain searchmee.com – I started playing around with writing web spiders around 1996, which is how I was able to create FindAnISP.com and gather the data. Partly just to see what was out … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[19],"tags":[],"class_list":["post-8002","post","type-post","status-publish","format-standard","hentry","category-about-the-guide"],"_links":{"self":[{"href":"https:\/\/streamingradioguide.com\/startingover\/index.php?rest_route=\/wp\/v2\/posts\/8002","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/streamingradioguide.com\/startingover\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/streamingradioguide.com\/startingover\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/streamingradioguide.com\/startingover\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/streamingradioguide.com\/startingover\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=8002"}],"version-history":[{"count":3,"href":"https:\/\/streamingradioguide.com\/startingover\/index.php?rest_route=\/wp\/v2\/posts\/8002\/revisions"}],"predecessor-version":[{"id":8005,"href":"https:\/\/streamingradioguide.com\/startingover\/index.php?rest_route=\/wp\/v2\/posts\/8002\/revisions\/8005"}],"wp:attachment":[{"href":"https:\/\/streamingradioguide.com\/startingover\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=8002"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/streamingradioguide.com\/startingover\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=8002"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/streamingradioguide.com\/startingover\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=8002"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}