Hinky’s Proxy Obsession

The other day I was on a support call with a techie from a vendor that will remain nameless (go ahead… guess!).  We were watching some HTTP packets fly by with tcpdump when he suddenly said, “WTF is that?”

“That” was along the lines of this:

http://72.246.30.118/idle/Ga0mdz02wSLOaQ5Q/250
http://72.246.30.118/idle/Gx0mdz02xSLWq-NW/44
http://72.246.30.118/idle/Gx0mdz02xSLWq-NW/121
http://72.246.30.118/idle/Gx0mdz02xSLWq-NW/200
http://72.246.30.118/idle/Gx0mdz02xSLWq-NW/251
http://72.246.30.118/idle/Gx0mdz02xSLWq-NW/310
http://72.246.30.118/idle/Gx0mdz02xSLWq-NW/359
http://72.246.30.118/idle/Gx0mdz02xSLWq-NW/422
http://72.246.30.118/idle/Gx0mdz02xSLWq-NW/481

More or less.  Lots of these suckers.  I see millions of them in my proxy logs every day.

This fellow had never seen these URLs before.  Odd because his company deals with URLs every day.  I just told him to add “not host 72.246.30.118″ to his tcpdump command line and that got rid of it.

Well, the support call eventually ended.  Nothing was resolved, as usual.  But this article is not about him anyway.

That crap is Flash.  The URL never has a hostname.  It’s always an IP address and that address usually belongs to a CDN (Content Distribution Network).  The address above belongs to Akamai.  The second part of the URL is one of open/send/idle.  The third is some sort of content or user or session identifier.  The last part is obviously a sequence number.

If you frequent some sort of Internet Radio station while you’re at work and you play it all day and you leave work with your browser open to that site, you will generate tens of thousands of these URLs.  I’ve seen a single user drop a half a million of these a day.

Now, all you blackhat spooks out there listen up, because this is important.  If you don’t get it already, I’m going to spell it out.

This is a perfect covert channel.

Just faking these URLs offers excellent cover.  The “idle” URLs are all http POSTs.    You can send data out without raising red flags as long as you keep the packet size down.  A single /idle/ URL packs about 215 bytes, but the user will hit a single /idle/ URL 600-700 times, for a total of ~150K.  In the logs, looking for that kind of crap in a multi-user environment boils down to the “needle in a needlestack” problem.

You get the picture.

It gets better if you can do it over a CDN.  This is what I like to call “The Chinese Radio Station Problem”.  It’s a deep pockets hack, because you have to control the server.  The CDN serves to mask the real destination. In my small little mind I see it as something an adversarial government has the resources to do.  Hence, Chinese.

Think about it.  It’s Flash, so there are plenty of known, unknown, and, shall we say,  ”private label” vectors that can be leveraged to add a little some-some to an end-user’s PC.

What that some-some is, is up to you.  You are only limited by your imagination.

If you control the server, the user can still listen to Internet Radio while he sends you his company’s intellectual property.  Logged or not, it still looks like streaming media.

I would conjecture that most companies that block streaming media leave it open for CxOs, which is even better because they get the juiciest details on intellectual property.  You just need to know what kind of media they like.

And for employers who don’t block streaming media, there’s people like Shirley in Accounts Payable, who has all the bank passwords and likes to listen to Christian music all day long.  Double cover.  Anything with “Christian” in it is above suspicion, right?

And what about that Asian dude in Engineering, Tony Lee?  What is he listening to?

Endless opportunity.

Reports of “millions of infected Web sites” in the past few days have been flooding the Intertubes.  These are due to a malware widget displayed on default parked pages at Network Solutions.

According to this article the widget is dropping a Koobface variant primarily on Chinese browsers.

One potentially limiting factor in this attack was that it seemed to target Chinese Web surfers. The malicious widget caused a fake message box to pop up, similar to a message prompt generated by the instant messaging client Tencent QQ. While this chat client is by far the most popular in China, it is probably unknown to most Westerners.

It doesn’t take a genius to connect the dots on that one.

I built 64bit snort (x86-64) on my old friend, Debian 4 (“etch”) and had lots of fun doing it.

It wasn’t quite as hard as I thought it would be, but there were a number of stumbling blocks, so I thought I’d post them here if anyone runs across the same issues.

First, our old pal Libnet1.02a is so fucking ancient it predates the AMD64 processor.  The Makefile simply barfs with an “unknown platform” error.  This was easy enough to fix.  First, locate line #172, which should be a comment starting with  ”Recognize the basic CPU types” (duh).  Right under that line should be a line starting with “vax-*”.  Edit that line to include “| x86_64-* |”.  No quotes, as usual.  Run configure, make, make install and that issue is over and done with.

Libdnet was the next big issue, and it was a problem on my 32bit Deb4 system as well.  Normally I hack shit together in /usr/local/src and libs fall into /usr/local/lib.  Unfortunately, libdnet doesn’t like living there and you can run ldconfig all day long and snort will still bitch about not being able to find it.

So start with “./configure –prefix=/usr” and that problem goes away.

I had to build a few things from scratch because Georgia Tech (gtlib.gatech.edu), my favorite apt repository (they seem to be the fastest around),  apparently stopped mirroring 64bit Deb4 and I was too lazy to look for a working repository.  You shouldn’t have this problem.  I had to build libpcap, flex, and bison from source but there were zero issues with any of those.

It took a while to come up with a decent config for snort but what I finally used was this:

Some of that may be unnecessary.  The dynamic examples are buggy and you’ll end up deleting them anyway (you’ll figure it out – trust me).

Once building snort64 was out of the way, I downloaded the rule set and tried out the different pre-compiled “so” (Shared Object) rules.

None of them worked.

That is of course my problem for using an ancient (2006) version of 64bit Debian.  There are a limited number of pre-compiled preprocessors for x86_64 and none are compatible with Deb4 (they all want ‘GLIBC_2.4′, which Debian 4.0 doesn’t have).

This is not going to be a problem for you folks on the cutting edge of Linux.  If I was running Lenny (I hate Lenny with a passion) I’m sure I could have used one of the 64bit pre-compiled preprocessors for Ubuntu.

But I’m not, so I had to roll my own.  And that’s when it got ugly.

Due to “contractual obligations” (NDAs no doubt), snort/SourceFire is only allowed to distribute certain rules in binary format.  They include the source for the non-NDA rules, but how to actually compile them is a trip into Undocumented Territory (“here be monsters”).

Sure, there’s a README file but all it does is explain their legal obligations and reassure you that one of the the pre-compiled preprocessors should work.  That’s it.  That’s all it says.

Thanks, Marty.

But there is a Makefile, so you’re halfway there.  I had to do the following to build the so preprocessors.

First, copy all the files in the src folder (that came with your VRT rules) into the folder where your snort source lives.  I called the folder “so_rules”.  Then fix the locations in the Makefile, like so (assuming you installed the snort source in /usr/src):

Note the variable $SNORT assumes snort is already built and living in your source folder.

Then, comment out the line that begins with “PATH” because I absolutely guarantee you don’t have an Intel compiler.

Now, find the line that start with “libs :=” and remove the following:

These must be the NDA-disqualified shared objects because there is no source to build them from.

Next, just for fun, find and comment out this line:

Why?  Check out the “deleted.rules” file that came with your VRT rules.  There is a lot of old crap you may never see again, but you never know.

Save the Makefile and type “make” inside the so_rules folder (not the toplevel make).  If all goes well, move the *.so files you just built to a folder under /etc/snort (I used “so_64″) and edit your snort.conf file to point to them (the very last line in the “Step 4″ part of snort.conf).

Finally, test your new config out and see what happens. You should see “166 preprocessor rules” intead of “0 preprocessor rules” now and 13 new “Rules Object” entires (I’m using the May 2010 VRT set, so YMMV).

Again, you shouldn’t have to go through this crap if you have a relatively modern 64bit distro of Linux (newer than 2006 I’d say) since the distributed shared objects should run just fine.

What do Windows users do about the lack of shared object rules?  Suffer, I guess, since there are no pre-compiled binaries, 32 or 64bit.

Hopefully, SourceFire will fix that soon.

Are you listening, Marty?

You may notice the “PoTTY” link above. There’s nothing there yet. That is/was a test for the “Visit Web site” button on the PoTTY “About” box. Right now I’m mulling over where to put the files. I think they’ll end up on www.mrhinkydink.com (not the BlogSpot redirect), if only for “branding” purposes.

Today I obfuscated PLINK and PSFTP (I did PSCP over the weekend), but I haven’t tested them yet, so the entire PoTTY Suite is almost ready to rock’n'roll.  As far as the source code goes, I could zip the entire directory structure up as is, but I need to do something about my idiosyncratic, highly non-standard search paths so you can just import it into VCE 2008 with minimum hassle.  I agree VC6 would be a better format, but I don’t have VC6.

I have had a few hiccups using my own obfuscated-openssh server (Debian 4.0r0).  I have four instances of ObfuscatedPort in sshd_config, but only two of the ports are working.  This is probably something unique to this system (i.e., something I screwed up) since I also have it installed and running on four ports on my CentOS laptop (long story – it’s the only distro that works on a SONY Vaio PCG-K27) with zero problems.

The simple Cyqwin fix continues to shine.  Right now I have Cygwin oossh running in a loop back at my office in the Salt Mine, back-tunneling RDP.  There have been a few connection drops due to proxy issues – it goes out through a very busy proxy – but it reconnects fine.

If you use a proxy with Cygwin oossh (or any ssh client), be sure to set the ServerAliveInterval to something less than a minute.  In my case, the Microsoft ISA proxy back at the office drops all TCP connections after two minutes of inactivity, so you have to keep that pipe flowing if you want to stay connected.  Be it SOCKS or HTTP, it doesn’t matter.  You get two minutes.  Shit or get off the pot.

This is usually not an issue if you can get out through a firewall hole, but YMMV.

I don’t recommend using oossh through a proxy, but sometimes you have no choice.  I have a choice, but I use a proxy because I have this… obsession with proxies.

Did I ever mention that?

Inevitably, once you start tunneling ports with ssh you get the Bright Idea to use PuTTY or Cyqwin on a Windows box and forward RDP (usually TCP 3389) through a remote sshd server.  Then, you travel to the remote sshd and try to connect back to that port you forwarded.  If you got it right, you get a Windows login screen, type in your credentials and… you get a Black Screen Of Death.

There have been various solutions to this problem over the years but generally they all FAIL.

At one time Microsoft published a KnowledgeBase article on this very subject.  They even offered a patch (I had it once, but I have no idea where it went), but even that failed.  And, like a lot of KBase articles, that particular one quietly disappeared (they do that a lot).

It should work, but it doesn’t.  Over the years I have tried a number of variations on the RDP back-tunneling problem, all with same results.  OpenVPN?  Nope.  Same problem.  You usually end up using a third box to bounce the port off, which works just fine, if you have a third box.

Today, I stumbled onto a semi-solution that “sort of” works.

Bear with me, this is sort of convoluted.

I run Debian Linux on VMWare GSX server on a Windows XP box.  NAT’d, not bridged (port level securty doesn’t tolerate bridging).  OpenVPN runs on this VM, connecting to a remote Linux box.  Here is the XP desktop with the Linux VM…

Through the VPN, I can hit the XP box’s RDP port.  BUT the same problem shows up.

Most of the time.

Very rarely, it just works.  Sometimes you don’t get the Black Screen Of Death, but the session will freeze and die as soon as the desktop is half-rendered.  But when it works, it works.

Today I stumbled on to one of those sessions.  Solid as a rock.

I fired up a DOS window and ran netstat.  I found I had ~5000 TCP connections in a TIME_WAIT state.  They were all connected to the proxy server.  I knew immediately what this was.  It was another connect-back process I had been working on (long story, so let’s leave it at that).

This other process runs once every ten minutes and makes all those thousands of connections.  After a few experiments I found that I could always connect whenever this other process was running.

For some reason, all those thousands of TIME_WAIT connections were actually helping.  And all of those connections were consecutive ports.  In the middle of them all, there was the local address of the system (not localhost) talking to itself on port 3389, which was very strange to see.  I expected to see the bridged address of  the VM attached to 3389 on the local address.  In fact I have looked at this issue many times before with Wireshark and that’s exactly what you see (the VM address talking to the local address, never the local address talking to itself).

So, some sort of throttling effect was going on.

And what’s the point?  I don’t know.  I’m writing this down to document it for future reference.  But it has me thinking about doing a few experiments to finally get this working.

It Gets Weirder…

That was yesterday.  Today I tried it again and this time I had thousands of connections in CLOSE_WAIT status, from a completely different process.  And there, once again, in the middle of all that, was the local address talking to itself on port 3389.

Not the locahost address (127.0.0.1).  I can’t stress that enough.

This time I fired up WireShark and told it to look at TCP 3389.

Remember, I’m in an RDP session.  There should be packets flying in and out of 3389.

There was nothing.

So I told WireShark to look at the VMWare NAT adapter (remember, I’m connected via an OpenVPN link through a Linux VM on this same box).

NOTHING!

No traffic at all from or to 3389 on any adapter.

It’s like the connection doesn’t even exist.