Early this morning all of the updates just STOPPED.
The page was getting refreshed on schedule but nothing new was getting added. When I got home after slaving away all day in the salt mines, the VM that runs the project was choking on its own puke. It was starved for memory and unresponsive. I had to hit the Virtual Big Red Button to get it back. I rebooted it and took a nap.
When I woke up it was doing it again but it wasn’t dead yet.
I killed a lot of garbage database processes. More popped up so I killed those. Then more, until it went back to normal. Then I ran the process manually to see WTF was going on.
As it happens, my ISP has decided to be “helpful”. They have re-hacked their DNS servers to return their own search page whenever a DNS lookup SERVFAILs. This makes my scripts go nuts.
I have one or two Web sites that disappeared after the shutdown of ESTdomains in November. I keep them in the mix because I’m hoping against hope they’ll be back again some day. When the script runs across them it expects to timeout, and not get a “helpful” search page. Since it doesn’t timeout it chews on the nonsense from the search page.
Forever.
The database never got updated (nothing there anyway), process upon process went into forever-loops, and eventually killed the whole system.
Anyway, that’s all fixed now. The 10PM run should have a lot of new proxies, and I’m seriously considering running my own damned DNS server.
The last few times the system has hung, I noticed a trend. Each time, without fail, there was a pop-up balloon noting that the wireless network had reconnected (the system is on the wired network, but uses a “secure” ad hoc 802.11b “point-to-point” network to route the wired network to a wireless camera).
This wireless NIC had a Marvell-based chip. I have several of these. I hate them all because they are proprietary and don’t work worth a damn with Linux. Apparently this is yet another reason to despise them.
I pulled it and replaced it with a RaLink RT61 based card. If you want to run Linux wirelessly, RaLink is the only way to fly. It’s been fully supported in the Linux kernel for a few years now and the drivers are in active development. You never need to mess with that god-awful ndiswrapper abortion (don’t get me wrong, ndiswrapper is a very slick hack… it just shouldn’t exist). Unfortunately, RaLink cards are hard to find. I’ve been burned twice by “errors in photography” where the box or the online illustration clearly shows a RaLink chip on the card, but when you open the box the damned thing has a Marvell chip.
It’s been running all week without a hitch. I’ll give it another week and if all goes well I’ll start un-doing my previous attempts at “fixing” the problem, especially that extra gigabyte of RAM I removed a few months back.
In other news, CoDeeN servers continue to disappear. There are now only 34 active servers left in the database.
When I moved the CoDeeN proxies to a standalone text file, there were about 300 total.
Today, there are 50! FIFTY!
I thought perhaps it was something I did, so I ran a resurrection on them all. They’re pretty easy to identify in the database even when they’re down because most of the DNS names have either “planet” or “lab” (or both) in them. Sure enough, they’re showing up as CLOSED, meaning the address is definitely there but nothing’s listening.
It could be they’re cracking down on abusers (recall my problem with my ISP and the Polish CoDeeN operator from a few weeks back).
Whatever the reason, they’re going fast.
Back before the Google Hack became my main modus operandi, I raided the more popular proxy lists. I still raid the best ones every night at 4AM. And they still have mostly crap, but I pick up 10-20 new proxies from them every night.
Today I woke up and the damned system had hung at 2:48AM (this is still driving me nuts). So, I did a manual 4AM run. In the process I discovered one of my scripts wasn’t working anymore.
It was one of those “dicey .ru domains” we all know and love. These clowns use a simple Javascript hack to prevent casual screen-scraping. Turns out they changed it, but not significantly.
So, I re-hacked my hack, did a test run, and picked up a handful of proxies. I have said this many times before and it remains a FACT: Javascript only makes it easier.
There is one obfuscation technique I’ve never been able to hack around, and here, for the first time, it is revealed:
AJAX
Luckily, only one of the Proxy List Boys uses it, and his list is useless. Utter CRAP. But it’s impossible to scrape with a shell script.
At least for me.
If it ever catches on with the listers (and it won’t), it’ll put me out of business.
If you haven’t noticed (I didn’t until earlier this morning), proxies (specifically, tinyproxy.exe) spread by the Koobface virus are taking over!
TCP port 9090, signature port for the tinyproxy.exe, has risen to the number five slot for verified proxies (number ten if you look at all ~1.4M in the database). It will take over port 3128 for the number four spot if Facebook users keep getting pwned at the present rate.
Personally, I don’t use them. The reason for that is they’re all in US, GB, and CA domains, which I normally avoid (US because I live their, the others because of treaties, LEA cooperation, etc). Almost without exception they’re botnet nodes and I’d rather not piss those people off either.
If you’re braver than I, give them a shot because they’re mostly DSL and cable accounts that are almost guaranteed to be fast. Get ‘em while you can because by next Patch Tuesday they’ll be in Microsoft’s “malicious software” gunsights, if they’re not already.
