Hinky’s Proxy Obsession

I’m putting a fork in this sucker here and now.  All four utilities (PoTTY, oSFTP, oSCP, and PLoNK) are available as standalone 32-bit Windows binaries at the PoTTY Download Page at my eponymous main site.

And the source is there if you want to play around with it.  Everything except OpenSSL, which you’ll have to get for yourself here.  You will need VCE 2008 and the Microsoft Platform SDK for Windows Server 2003 R2 to get ‘er done.

What?  You say you have VCE 2010 and the SDK for Server 2008?  Well… I don’t! If you get the code to compile on VCE 2010, let me know.  I’ll upgrade.

You say you want a 64-bit version?  Tough luck.  You’re stuck with 32 bits unless you want to roll your own.  A Mac OS X version?  Good luck with that, too!  A Linux version?  Hmmm… it might work, but I always wondered why anyone would bother with the Linux version of PuTTY.  I have it.  But I seldom use it.  I’m sure there are good reasons, but I haven’t thought of any!

Even though it adds two new options, PoTTY won’t hose your existing PuTTY profiles.  You can still use any version (within reason) of PuTTY with your PoTTY profiles, but of course obfuscation will not be available with PuTTY.

If you like it (don’t forget you need an obfuscated-openssh server as well), tell your friends!

If you don’t like it, tell your enemies.

Back in 2008, when I first published The List, I decided I needed some publicity so I joined a few Yahoo Groups dealing with proxies.  It turned out to be a horrible mistake.  Most proxy groups at Yahoo at the time were either full of junk or SPAM.  Or both.

Not so for proxitown, run by Baron Munchausen (get it?  From “Munchausen Syndrome by Proxy” and the eponymous movie.  How clever!) of Proxy.org fame.  The Baron himself is the lone spammer at proxitown and if you subscribe to this group you get daily updates of available Web (PHP/CGI etc.) proxies.

In theory.

I say that because the Baron stopped sending “daily updates” from proxitown a long time ago.

I stayed with the group because… well, I forgot about it since I wasn’t getting any emails… and also because I have a professional interest in Web proxies (as opposed to “open proxies”, which is my… obsession! ).

I block Web proxies by plugging them into Websense in my role as Chief Network Nazi back at the Salt Mine.

This activity is somewhat antiquated as the newer generation of “Active Web Gateways” are updated in real time – or so the advertising goes.  The version of Websense that I’m stuck with (6.x - the Chief Slave Driver is too cheap to upgrade)  only updates once a day.  Don’t get me wrong – Websense is very good at blocking Web proxies (I’m sure they subscribe to proxitown, too), but there’s always a window of opportunity for any new proxy created after their last database update.

It’s my job to close that window.

In the past, the Baron’s lists were pretty stale and Websense always had the domain names in their database, so there wasn’t much for me to do.  But this time around the Baron has some pretty fresh stuff.  In the last e-mail I checked, three out of the twelve proxies listed were unknown to Websense.  That’s 25%!

Score one for the Baron!

The RDP Back-tunneling issue was nailed by Cygwin’s version of OpenSSH, 5.4p1.  Since obfuscated-openssh is based on 5.2p1, it did not work.  So yesterday I spent hours and hours trying to brute-force Bruce Leidl’s patch into 5.4p1.

After I spent two hours editing the files (I have confessed to knowing nothing about patch/diif files in the past), I wasted the rest of the day testing.  I actually did get it to run, but the client was barfing on perfectly good host keys.

At 1AM this morning, beaten and bleary-eyed, I decided it was a good time to read the docs.  This is what I found in 5.4p1′s ChangeLog:

Due to constraints in Windows Sockets in terms of socket inheritance, reduce the default SO_RCVBUF/SO_SNDBUF buffer size in Cygwin to 65535. Patch from Corinna Vinschen.

WTF?  So I chased this variable down and found it in clientloop.c as SSH_IOBUFSZ in 5.4p1 in the declaration of a buffer inside these two functions:

• client_process_net_input
• client_process_input

That sounded about right.  After all, in the RDP back-tunneling problem the client is having problems processing input from the net, right?  WTF, worth a shot.

So I looked at the same code in obfuscated-openssh.  SSH_IOBUFSZ was not there at all and the buffers were hard-coded at 8192 bytes.

Remember, the ChangeLog said reduce the buffer size to 65535 bytes!

I changed the size, recompiled, and did a quick RDP back-tunneling test.

NAILED IT!

And it’s not a “spotty” fix like the rest of them have been over the years.  It works every fucking time.

And it’s a genuine Cygwin fix!  It’s not an OpenSSH fix at all!  If you Google “Corinna Vinschen” (with or without the quotes, I don’t care) you will find she (he?) works on the Cygwin Project at Red Hat.

There is no cause to rejoice since, being a core WINDOWS PROBLEM, it could get blown away next Patch Tuesday.  Or the Patch Tuesday after that.  Or the one after that, ad nauseum.

Being a core Windows problem (technically known as a “clusterfuck”), there must be something similar in PuTTY that can be jerked around to make its issue with RDP back-tunneling go away. I’m looking but the coding style is so different I haven’t found anything glaringly obvious.

I’m not sure what to do with “obfuscated-openssh-5.4p1″.  At this point I’m sick of it, but it was a decent learning experience, even if the only thing I didn’t learn was to read the docs in the first place.

So it occurred to me… what if I back-tunneled to a secondary NIC on the Windows PC?

That would absolutely get away from the local address<>local address conundrum, or so I thought.

Here at HinkyNet, besides innumerable XP VMs, I have a couple of physical XP boxes to work with and a few Linux boxes to tunnel through.  All I needed was an XP box with two NICs.

I didn’t have that, but I do have a couple of 10/100 USB NICs.  So I dug up my old CAT5 crimping tool and researched how to make an ethernet loopback plug, so I could assign an address to it.

Then, I ssh’d from the XP box with the USB NIC to a Debian box, forwarding port 3389 to the USB NIC.

I logged on to the second XP box and tried to RDP into the first XP box through the ssh tunnel.

Same result: Black Screen Of FAIL

Damn, I was pretty sure that was going to work.

Then I left Wireshark running on the destination XP box, watching the traffic to the USB adapter and tried it again.  After it crashed, I looked at the capture.

NOTHING!

Not a single packet going to port 3389.  Netstat verified terminal services was in fact listening on all adapters (0.0.0.0:3389).

Meanwhile, my RDP back-tunnel over OpenVPN via Linux VM has been up for 12 hours.

There has to be a way!

Inevitably, once you start tunneling ports with ssh you get the Bright Idea to use PuTTY or Cyqwin on a Windows box and forward RDP (usually TCP 3389) through a remote sshd server.  Then, you travel to the remote sshd and try to connect back to that port you forwarded.  If you got it right, you get a Windows login screen, type in your credentials and… you get a Black Screen Of Death.

There have been various solutions to this problem over the years but generally they all FAIL.

At one time Microsoft published a KnowledgeBase article on this very subject.  They even offered a patch (I had it once, but I have no idea where it went), but even that failed.  And, like a lot of KBase articles, that particular one quietly disappeared (they do that a lot).

It should work, but it doesn’t.  Over the years I have tried a number of variations on the RDP back-tunneling problem, all with same results.  OpenVPN?  Nope.  Same problem.  You usually end up using a third box to bounce the port off, which works just fine, if you have a third box.

Today, I stumbled onto a semi-solution that “sort of” works.

Bear with me, this is sort of convoluted.

I run Debian Linux on VMWare GSX server on a Windows XP box.  NAT’d, not bridged (port level securty doesn’t tolerate bridging).  OpenVPN runs on this VM, connecting to a remote Linux box.  Here is the XP desktop with the Linux VM…

Through the VPN, I can hit the XP box’s RDP port.  BUT the same problem shows up.

Most of the time.

Very rarely, it just works.  Sometimes you don’t get the Black Screen Of Death, but the session will freeze and die as soon as the desktop is half-rendered.  But when it works, it works.

Today I stumbled on to one of those sessions.  Solid as a rock.

I fired up a DOS window and ran netstat.  I found I had ~5000 TCP connections in a TIME_WAIT state.  They were all connected to the proxy server.  I knew immediately what this was.  It was another connect-back process I had been working on (long story, so let’s leave it at that).

This other process runs once every ten minutes and makes all those thousands of connections.  After a few experiments I found that I could always connect whenever this other process was running.

For some reason, all those thousands of TIME_WAIT connections were actually helping.  And all of those connections were consecutive ports.  In the middle of them all, there was the local address of the system (not localhost) talking to itself on port 3389, which was very strange to see.  I expected to see the bridged address of  the VM attached to 3389 on the local address.  In fact I have looked at this issue many times before with Wireshark and that’s exactly what you see (the VM address talking to the local address, never the local address talking to itself).

So, some sort of throttling effect was going on.

And what’s the point?  I don’t know.  I’m writing this down to document it for future reference.  But it has me thinking about doing a few experiments to finally get this working.

It Gets Weirder…

That was yesterday.  Today I tried it again and this time I had thousands of connections in CLOSE_WAIT status, from a completely different process.  And there, once again, in the middle of all that, was the local address talking to itself on port 3389.

Not the locahost address (127.0.0.1).  I can’t stress that enough.

This time I fired up WireShark and told it to look at TCP 3389.

Remember, I’m in an RDP session.  There should be packets flying in and out of 3389.

There was nothing.

So I told WireShark to look at the VMWare NAT adapter (remember, I’m connected via an OpenVPN link through a Linux VM on this same box).

NOTHING!

No traffic at all from or to 3389 on any adapter.

It’s like the connection doesn’t even exist.