Hinky’s Proxy Obsession

Apparently, nearly every open proxy in Mexico goes through this box.  Maybe that’s why they named it after a volcano.

Yes, I admit it.  Since yesterday I’ve been resurrecting dead proxies right and left, entertaining myself by watching the HTTP headers fly by (I am easily amused).  And going through the list I have noticed a few things.

For one, the “Proxies of 2008″ are dying off, which is to be expected.  There are only about 60 or so left.

Second, this month, August 2009, has been a great month for proxies!  Twelve full pages worth.  August has also been a great month for Mexican proxies, with most of them discovered on the 15th.  Most of them are going through the server in the title of this article.  It’s a shame I never bothered to put the VIA header in the database.  There’s a lot of good information there, lost forever.  Oh, well.

August 2008 was also a good month for proxies.  It was when we hit our first 1,000,000 boxes.  And if anyone besides me remembers back then, September 2008 was extremely dry.  Things did not pick up again until February of 2009.

Thirdly, the sites I mentioned yesterday as running “TeamViewer” all turned out to be a single IP address with 200 open ports (and probably more, since I started dropping any port less than 80 over a year ago).  And although they were running TeamViewer yesterday, all of them are open proxies today (so far, 40 out of the 200 open ports are now reporting as Transparent proxies in the list – I just haven’t hit them all yet).  Which means they’ll probably be dead tomorrow.

Here is the whois information on that particular IP:

Note they assigned a netname to a single IP address.  This seems unusual to me, but it could be a common practice (in fact this ISP does it a lot, if you look here and search for “SXTY”).  This probably at least partially explains the “here today, gone tomorrow” nature of Chinese proxies.  But the tie-in with “This site is running TeamViewer” is still a head-scratcher.  WTF is up with that?

I did solve my issue with the proxy judges that don’t return an Http-Referer header.  They still all return the User-Agent header, so I use that as well.  Still, I have a nagging suspicion that some proxy judges are lying to me.  I would use my own (yes, I have one of my very own design), but I have found that a lot of servers in foreign countires have difficulties resolving mrhinkydink.com.  Perhaps it may be useful as a backup of last resort for High Anon proxies (all the judges I use identify Transparent and Anonymous proxies faithfully).

Web site/false proxy detection (“Offline/WEB” in the database) is now rock solid.  I used to depend on the headers returned, but you will often get an “HTTP 200 OK” result for a login page instead of the expected (some RFC dweebs would say “correct”)  ”HTTP 403 Forbidden” result, followed by an “HTTP 302 Object Moved” to the login page.  You can’t expect Web developers to play by the rules, since they’re morons.

Other Tidbits

I keep checking the Israeli obfuscator site manually, although it has been removed from the code.  The name still resolves but the site times out and the nmap still shows port 80 as “filtered”.

I added “Cameroon’s favorite proxy list” under the page title, just for the Hell of it.

We added another 100,000 proxies to the database, hitting the 2.1 million mark this week.  And there’s more than 107,000 proxies in the “gold” table (address/ports that are or were open proxies since March 2008).

Something isn’t right with the code.  Right now there are 1140 proxies, but only 19 pages.  At fifty per page there should be 23 pages.  Come to think of it, this probably explains the missing proxies from 2008.

Unfortunately, I haven’t looked at the forum spammer’s reporting site since I started mirroring it.

Remember the Canadian hospital proxies?  They’re all gone now.  Someone figured it out and fixed it.  Good for them!

The code is getting much better at detecting non-proxies with the Referer hack, but as my code gets better I keep stumbling across crud in other people’s code.

Since twiddling the CoDeeN detection, I have been making small resurrection runs on the “gold” database, pulling 500 “dead” proxies at a time and checking the results. Interestingly, I’m getting somewhere between 1% and 2% worth of good hits, which is very typical of even the crappiest proxy lists found in the wild (with the possible exception of ProxyCemetary, which must be the worst proxy list of all time).

I discovered that two of my proxy judges (so far) don’t bother to return the Http-Referer header, which is fine for Transparent and Anonymous proxies, but sucks ass when it comes to High Anon servers. So, out they go. All of the other judges seem fine.

I’m not sure why you would bother not returning that field, although most of these pages don’t know they’re proxy judges.

Then there’s shit like this. Don’t worry, it won’t bite.

If you Google the IP, it’s in proxy lists everywhere, over 1500 hits worth.

How the Hell does that get into proxy lists? Maybe it is a proxy if you have a login (I haven’t tried brute forcing a login), but it’s useless if you don’t.

Then there’s “This site is running TeamViewer.” This is usually on a variety of ports on the same IP address. Pure crap. There are hundreds of them. Check out this nonsense from their “Security Statement“:

International top corporations from all kinds of industries (including such highly sensitive sectors as banks and other financial institutions) are successfully using TeamViewer.

And their IP addresses are on proxy lists?

DOH!

The latest javascript-obfuscated proxy list has apparently died, which is kind of a shame because it was good for a few proxies a day while it was up.

Plus, de-obfsucating it was good for lulz.

I’m not quite certain when that happened, but the other day we had a power blip and as a result the 3PM harvest didn’t run.  I noticed it was gone after I ran the script manually.  Every page timed out.

I did quite of bit of Googling to see if it was really down or whether they had simpky blocked my IP (it happens, and when it does I just use a proxy).  It looks like it was heavily promoted from December of  ’08 to May of this year.  In fact a few “respectable” (relatively speaking) sites have praised (trolled?) it since the beginning of the year.  The domain name still resolves, guerrilladns.com says it’s the only DNS name on that IP, and nmap says port 80 is “filtered” but I’m getting nothing from it.

So out it goes.  One positive side effect of it going down was I found another site with a good list.  This site also had a very nice proxy-harvesting shell script on it.  In fact he hits the same sites I hit.  And it’s enlightening to see how someone else would do it.  I get a little down and dirty when it comes to sites that require cookies, but he had a much more elegant approach.

My vacation is over.  The List has been running on full auto for two weeks now.  The purge is working right and the resurrection kicks in as expected when the list gets too short.  However, I still need to work on the Referer hack.

Which is good because I’ll need something to do at work this week.

lulz

No sooner do I declare the Google Hack played out, it turns me into a liar.

Shortly after my last post, I ran the Hack one more time before putting it out to pasture. A few minutes into the run, it came up with one of “those” sites. It seems to have every kind of scam on the Net (lose weight, make money from Google/Craigslist, payday loans, etc.) and some type of blackhat SEO angle. There are dozens of text files listing “users” of various online forums, the accounts they use, URLs for their profiles, and…

… the proxies they use to post.

I ran a few Google searches on some of these names and they are absolutely everywhere. There’s hardly an online forums that hasn’t been hit by these accounts so it’s definitely a forum or comment SPAM operation of some sort.

I’m not sure if this is good news or bad news. I’ve done a few runs against what they’ve got and so far it appears they don’t have anything that isn’t already in my database, although I am getting a few new TIMEOUTs and port CLOSED results. In other words, dead crap.

Anyway, there is so much stuff here that I’m going to mirror the site and bang at it at my leisure. It’s well-indexed by Google, so it’s no Big Secret, but the last time I ran across a site like this it didn’t last more than a couple of months. And the information is so… varied… that an offline study is the best way to mine it.

Check back in to see how it goes!

OK, listen up. Blog SPAM like this is VERBOTEN:

I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.

Give me a break. I’m not that stupid.

If you’re going to linkbait me, at least try to stay on topic, like this guy, who is obviously a master linkbaiter.