Hinky’s Proxy Obsession

You learn something new every day.  If not, you’re not doing it right.

Consider this Web page (it won’t bite – I think).

Isn’t this the most special thing you’ve ever seen?  It seems this Webmaster (or, more likely, Web “developer”) has decided to collect proxy data and he probably doesn’t even realize it.

This kind of shit happens all the time.  Some Web clown finds something kewl on SourceForge and decides he’s going to put it on every stinking two-bit PHP/MySQL site he touches.  In a week or so, he gets bored with it and forgets about it completely.  But he doesn’t remove it.  Ever. It runs and runs, collecting data forever.

This is not necessarily a Bad Thing™.  I’d never run it and I wouldn’t recommend that you run it, but these people are supplying us with some very valuable (and up to date) proxy information.

This is exactly how the proxy judges I use came into being.  A cool page somebody found,  ten fucking years ago. Copied, pasted, and deployed damn near everywhere, but not like this beauty.

This thing is BIG.  It is HUGE.  Do a Google search on it and you will get 60,000+ hits.  A lot of those hits are articles about an SQL injection bug or two in this little gem, but most will be hits on sites that are running it for site statistics.

Some of those sites aren’t places you want to click on, so tread carefully.  Some are just plain deadwebs that nobody ever hits, except for your occasional wandering GoogleBot.

I considered scraping these sites, but there’s hardly any point.  I can almost guarantee that any open proxies in all that data is already in my database.  Still, for the active sites running this crap, the proxy data is probably fresher than 99% of the proxy lists on the Web.  The trick would be finding those high traffic sites.

In case you’re wondering, I didn’t mention the name of the software on purpose.

I have been going over the emails generated by the Top Ten List’s cron jobs and it’s getting me thinking.  So I’m just going to mull over a few ideas here.

A successful catch should update the “Last Checked” timestamp and the speed value in the database.  That’s easy enough.

Since I’m going for a faster proxy than usual, a “Timeout” doesn’t really count.  It may still reply given the full 45 seconds I normally use.  Do nothing with those, just wait for the next garbage collect (list purge).

Then there’s those “connection refused” results.  Update timestamp, zero speed, and mark as “Offline”.  It will fall off the list next time it’s published.

I’m getting 403s (“Authentication Required”) and 407s (“Proxy Authentication Required”) here and there, which, regardless of the timeout, indicates a worthless proxy – unless you like to pass the time hacking passwords.  I don’t.  So in those cases, update the “Last Checked” timestamp, set the speed to zero, and mark the proxy “Offline”.  The next time the list is generated, it’s gone.

I get an occasional 404 (“Not Found”).  This means nothing, since a lot of foreign (to me) proxies have a hard time resolving my domain name the first time anyway.   Do nothing at all with those since a purge will eventually take care of them.

I could reset the time requirement to the list’s default of 45 seconds, but that would take away from the intrinsic value of the Top Ten.  After all, they’re supposed to be good and fast.

Also, limit the search to proxies whose “Last Checked” value is less than curdate(), that is, proxies that haven’t been checked today.  That would cause the list to be searched more aggressively.  But the problem there is purge days, since they will have all been checked “today”.  And, “today’s” proxies would never make it to the Top Ten.

There’s probably a way around that, but I’d have to think about it for a while.

These are tiny little changes you’ll never notice, but it will make the list a bit more reliable.  And we want reliable proxies!

I have a silly expansion on a couple of my UT99 servers that forwards player conversation to Twitter.  I realize that Makes No Sense, but I was bored and needed to practice my Scriddie Skillz.  It’s been running since August and has probably spit out about a quarter of a million tweets since then.

UT99 players are not that talkative, so I have a script that injects conversation into the game.  That is where 99.9% of the tweets come from.  With all that mindless jabbering, I hit Twitter’s hourly limits whenever the UT99 servers are busy.

Yesterday I decided to leverage the list to get around that.  Once every 15 minutes, a script queries the database and gets a proxy that:

• Is not in the USA
• Is not a CoDeeN proxy
• Responds in less than 20 seconds

Then, the Twitter scripts use that proxy to send tweets.  It has been working well.  I check the proxies against My Own Personal Proxy Judge for functionality only (it doesn’t care about Transparent, Anonymous, etc.).  As a result I get a private list of working proxies.

I’m thinking I can leverage this private list as “Hinky’s Top Ten List”.  It should be fairly dynamic, since it finds a working server every 15 minutes.  Of course, since the list is only refreshed once every 120 minutes, it could get stale quickly, but I’m mulling that problem over and it should be a simple hack to keep it fresh.  Some may balk that it has no U.S. proxies, but fuck them anyway.  They won’t get pissed if I don’t mention it.

One drawback: it may get fairly repetitive.  Especially immediately following a purge, which tends to happen about twice a week.

Look for a link soon.  It will probably be along the lines of the CoDeeN lists, which are simple text files.

The damned cable went out last night, so the list hasn’t been updated in over six hours. Rather than play catch-up, I’m just going to go with the flow.