I did a kernel upgrade this weekend on the UT99 servers and as a result had to stop The List for a few hours.
After everything was up and running I manually refreshed The List but I accidentally ran an old script – one that still had CoDeeN servers listed. As soon as I caught that I fixed it, but it was up for about a half an hour before I noticed.
At noon (EST) today everything should be back in sync.
I bumped into an unintended side effect of using the Http-Referer as a definitive proxy marker. It seems it has killed off most of the CoDeeN proxies, which do not return an Http-Referer.
Of course, they do, but the only thing I use to detect CoDeeN servers is the infamous “(Not Really) Welcome” page they shove in your face when you first connect. After ten seconds they give you the page you requested (which, being a proxy judge page, does have the Referer), but by then I’ve closed the connection and moved onto the next proxy anyway.
And here’s an interesting tidbit about CoDeeN: they keep track of who they think you are by looking at your IP address and User-Agent. If they have seen your IP+UA before within a certain timeout period, you don’t get the welcome page.
For some reason they seem to think the IP+UA is a unique identifier (probably a hash of some sort). Since I can’t change my IP, I guarantee the User-Agent is always unique by stuffing it with random numbers. Microsoft has made this a piece of cake by adding the “.NET CLR [1-3].[0-9].xxxxx.[0-9]” extensions to the User-Agent field of Internet Explorer. Since there are no less than three versions of dot-NET compatibility (this week) plus multiple versions of IE and Windows platforms there’s a lot of room for randomness in those numbers.
This also gives you a modicum of plausible deniability with CoDeeN when they call the cops on you, since it’s highly improbable than any of those User-Agent values are actually valid (and they obviously don’t check).
Since there are often multiple ports on any CoDeeN server and servers at the same location with different IP addresses probably use the same backend IP+UA database, you have to change the User-Agent every time you test or you will get an invalid result (that is, it will not be an obvious CoDeeN server).
Even though no one gives a rat’s ass about CoDeeN servers, I thought it was important enough to fix. You really should be aware of them, especially if you want to avoid them. That’s why it’s important to detect them properly.
Still, I can’t help wondering if there’s some server room Trevor at PlanetLab scratching his head over all those weird MSIE User-Agents that have shown up in the database over the last year…
I know almost no one cares about the CoDeeN text files, but starting with the 4PM (EST) run they will be randomized, thanks to “order by rand()“.
Currently they’re listed in the reverse order of when they were discovered (like the rest of the list), which isn’t very random.
When I moved the CoDeeN proxies to a standalone text file, there were about 300 total.
Today, there are 50! FIFTY!
I thought perhaps it was something I did, so I ran a resurrection on them all. They’re pretty easy to identify in the database even when they’re down because most of the DNS names have either “planet” or “lab” (or both) in them. Sure enough, they’re showing up as CLOSED, meaning the address is definitely there but nothing’s listening.
It could be they’re cracking down on abusers (recall my problem with my ISP and the Polish CoDeeN operator from a few weeks back).
Whatever the reason, they’re going fast.
I split off the USA-based CoDeeN servers into a separate file.
I’ll admit I did this mostly for my own benefit.
I found it odd that the number of US servers was less than half the total count (42.5% at the present time). For some reason I expected a bigger chunk.
I suppose the next step would be to split off a file with non-USA servers. It would only take a couple of minutes but I’m feeling lazy today.
Changes applied. Page rewritten. CoDeeN purged.
“Undefined” is gone, due to the new junk filter. This does not mean the junk is gone for good. There is still one particularly nasty piece of junk to catch: “proxies” that mimic proxy judges. You will know them when you see them. It’s very difficult to tell whether a “proxy” has returned your judge page or it’s own judge page, which is the only thing it serves.
This is very popular in Japan, for some reason. China seems to be jumping on the bandwagon as well. I think there is a simple way out – request two pages instead of one: the judge and (say) Google’s home page. The downside is that will double the amount of time required for testing and verification.
Be that as it may, Mr. Hinky Dink still has the highest percentage of active proxies of any list anywhere, junk or no junk!
360+ CoDeeNs have been reclaimed and the file is on the server. The page doesn’t reflect this at the moment and the servers are still in the Main List. I plan to take them out of the list and keep them stashed away in the text file (updated and tested, of course). The CoDeeN file will be updated every other hour, just like the Main List. It’s randomized each time, so don’t depend on a hash to detect changes. It’s a very static list, but some servers may drop in/out over time.
Speed, country of origin, and all the ancillary data is not in the text file. That is not the point anway.
Remember, the main idea is using it with the >SwitchProxy tool for FireFox, but if you have other uses (like starting a proxy list with servers that actually work), then go for it. Don’t do something silly like uploading the list to a proxy forum because they don’t generally like CoDeeN proxies (in fact they despise them) and the 312x ports are a dead giveaway.
I had finished rewriting the code and was starting to get the CoDeeNs back when apparently the power blinked at home. Since I have my cable modem, switch, both UT servers, and the domain controller on uninterruptible power supplies, the connection stayed up.
Of course, none of the boxes involved in this project were protected. Maybe Santa will send me another UPS for Christmas.
There won’t be another run today until 6PM.
At least it waited until I finished coding.
No big surprise there.
The junk filter worked flawlessly. However, I never intended it to take out the CoDeeN proxies. Some would say that’s no great loss because they are, in fact, junk. But I’ve grown somewhat fond of them, so they will be back, but not in the main list.
I have been using the SwitchProxy Tool for Firefox for quite some time. It’s very handy for testing proxies, although it does some silly things now and then (for instance, when you select “None” it clears whatever settings you originally had in the browser), but one of its main features is it lets you use a text-based list of addresses and ports that it will cycle through either sequentially or randomly.
This is not very useful for testing, but if you have a big list of known good proxies it works very well. The problem is getting that big list in the first place. The CoDeeN list works great for this since there are so many of them and they’re all – with some exceptions – “fast enough”.
So, I’m going to split off the CoDeeNs and make them available on the left side menu as a text link. You can then add this link to SwitchProxy and browse through multiple CoDeeN servers.
From the SwitchProxy toolbar, select Add->Anonymous->Next and you’ll see the interface. Just plop in the link, decide how often you want to switch, and you’re ready to rock’n'roll. I haven’t decided on a name for the link yet, but it will probably be:
http://www.mrhinkydink.com/codeen.txt
Original, no? Don’t get excited because it’s not there yet. I have to resurrect them from the database first (since they got junked by the junk filter) and hack the code around.
Stay tuned.
