Hinky’s Proxy Obsession

I have a lot of security delicacies “on my plate” but SPAM isn’t one of them.  Back at the salt mines, that’s someone else’s problem.  Suffice to say I’ve ignored the subject for years, considering it’s fairly well handled by all the Web mail accounts I have floating around (Yahoo, Hotmail, Gmail, et cetera).

I never even look at the stuff.

However, there is this woman – let’s call her “Helen Dink” – who is no relation to me but by some strange quirk of fate we share the same ISP.  She believes her e-mail address is hdink@myisp.com, but it’s not.  That’s my email address.  I’ve had that address for almost ten years now, but whenever she goes to fill out Web forms online she plugs in my email address!

So I get a lot of her email.

This has been going on for at least five years.  I know more about this woman than I want to (yes, she’s a Facebook user).  Her friends constantly send me email about a variety of crap, from Girl Scout meeting notices to recipes and the like.  It’s incredibly annoying and for the past few years I’ve been writing them back and letting them know, in no uncertain terms that I am not “Helen Dink” and to please remove my email address from their “Contacts” folder.

So today I got an email from “Andrea Wilson”, a normal-sounding American name.  Thinking it’s one of Helen’s buddies I open it up in order to send my standard reply to these things.

But it’s not.  It’s a 419 scam email…

I am Golan Bradley a staff of Natwest Bank ,I am pleased to pass across to you a very urgent and profitable business proposal which I believe will profit the both of us after completion.I will await to receive a positive response from you to enable me give more details Please send your confidential telephone and fax number in your reply to: golan.bradley@removed.com Golan Bradley(Mr.)

The security wonk inside me kicks in and I decide to look at the SMTP headers, thinking I’d be able to track it back to Nigeria or Cameroon (hi, fellas!).

But the headers were extremely legit.  The email went through 2 Exchange servers, a Symantec Brightmail Gateway (an anti-SPAM device), and a ZIX encryption device before ending up at my ISP.

“Andrea Wilson” turns out to be a Real Person™  who works at a hospital in Texas.  And, not surprisingly, she’s an active Facebook user.  Obviously her workstation or laptop or whatever had been summarily pwn3d and was being used to deliver 419 SPAM for person or persons unknown.

Well, that’s her problem.

I briefly toyed with the idea of writing her back and suggesting she have the IT department check out her box, but that’s a notoriously bad idea and generally frowned upon (this comes from the heyday of email viruses, when sending a “HEY ASSHOLE YOUR COMPUTER HAS A VIRUS” email only served to exacerbate the problem).

Maybe I’ve been away from SPAM for too long, but it seems unusual to see legitimate SMTP headers from an obviously corporate environment, considering that lately the vast majority of SPAM comes from Yahoo, Gmail, Hotmail, et cetera.

Everything old is new again.

I got bored with snort and intrusion detection a few years ago.

The last thing I did with snort was try to code a netfilter detector/packet killer for one of Luigi’s UT99 hacks.

It was an epic FAIL.

I even read the docs!

But back in the day I ran snort & ACID (now BASE – how clever) on a number of boxes, both at work and at home.  I got very tired of seeing all those false positives, every day, day after day after day.  And then reading in the snort online docs that a particular rule I was unfamiliar with had “no known” false positives or negatives.

Riiiiight.

And then snort went commercial.  Well, good riddance.

So it’s been at least three years since I touched it.

Suddenly, out of nowhere, my boss the Security Thought Leader decides snort has to be deployed throughout our organization on every-fucking-thing that can run it.  And since I was stupid enough at one time to open my mouth and let the word “snort” fall out of it, it’s my problem.

Lo and behold we’re now up to snort v2.8.6.1.  And it turns out to be a bad time to be implementing snort.

As of June 2010 (Hey!  That’s like… now!) a lot of the old freeware utilities for managing snort are obsolete.  Not that they were any good, but their very existence was helpful in avoiding reinventing the wheel.

And the more things change the more they stay the same.

Like depending on Libnet v1.0.2a.  They still give you a link to a dead Web site in the docs so you can download this ancient piece of code required to build snort.  Luckily (I think it was luck) freebsd.org still had it in their archives.

Also luckily Mr. Security Thought Leader only wants to send snort alerts to a centralized SIM/SIEM syslog server, which simplifies everything (although he doesn’t know that… he would complicate the fuck out of everything if he did know it).  This is the only documented design requirement in the whole project:

Send snort to syslog!

That’s it!  Nothing more.  What he plans to do after that is a mystery.  Right now I have the Proof of Concept running on a few Windows servers and as usual they’re not producing anything of any interest whatsoever.

The Linux servers are another matter entirely.

I hate to say it, but our *ix admins are the biggest newbz I’ve ever met in my long and glorious IT career.  They need to have a vendor hold their hand whenever they do anything and their collective mantra is the age old refrain:

THE VENDOR WON’T SUPPORT IT!!!

To make matters worse, they are going to have to build snort if their vendor doesn’t distribute a 2.8.6 version of snort.  And considering they run Red Hat, they’re going to have a long wait.

So with all that in mind I built snort 2.8.6.1 for my Linux firewall (good old Debian Etch) and configured it for syslog alerts only.

It’s been running for about an hour now and all I’m getting is:

(http_inspect) LONG HEADER [Priority: 3]

… from just about every Web site I visit.  You would think with all this proxy crap running every hour I’d at least get a “Hey! Somebody’s using an external proxy” kind of alert but no.

There’s obviously some tuning that needs to be done (max http header was 750, so I upped it first to 1024, which wasn’t enough, and then to 2048), but with all the doom & gloom about hackers and cybercrime, et cetera, I really expected a shitload more alerts.

I guess my firewall rules are pretty good after all.

FWIW, it took less than ten hours for some idiot named Tedickhead to remove my PoTTY entry at Wikipedia, without any reason given.  Of course, it pissed me off but I refrained from putting it back in for the time being.

I have to admit to being not all that obsessed lately.

The Websense shit took a lot out of me.  In the end it turned out to be an ISA configuration issue, which made me feel a little stoopit, but at least it was an extremely esoteric (“undocumented”) aspect of the configuration.  The fix Websense published seems to leave much to be desired, and is probably exploitable as well.  Time will tell.

I got into a spontaneous SSH shit-storm on Full Disclosure just to do a little promotion for obfuscated-openssh and PoTTY, but apparently nobody cares.

It seems people are just stuck in their ways and no one is going to do anything any different than whatever it is they’re doing right now.

But somehow I did get this guy to mention PoTTY in his blog and give me and obfuscated-openssh a few links.  Spread the word!

Last week I spent an extraordinarily brutal amount of time compiling Google Maps (here and here), which has also added to my general burn-outedness.

On top of that I had some hardware issues in May that caused the List to malfunction in June.  In fact I just noticed that today.

I had a power supply die in my decade-old Windows 2003 server, which was were I was dropping the nightly backups to the database.

I forgot about that.

Subsequently, the mount point where the files should have gone was pointing to itself, which caused the root partition to fill up.  And when that happens things start to choke.

I only noticed it after seeing the List stuck at 666 proxies for about a day.

Normally when power supplies blow it’s a mad scramble to the nearest computer store to get a replacement, but this time I just said FUCK IT and let it slide, ordering a replacement from Tiger Direct.  But a couple of days later I decided to clean up the computer room and discovered I already had a replacement power supply.  And although I didn’t really want to replace it (having already said FUCK IT and being burnt out on technology in general), I replaced it anyway.

To top all that off, rumors are buzzing at work about a new re-org, so I took a five day mini-vacation (Thursday through Monday) to depressurize.

And during all that I missed the two-year anniversary of the List!  I had planned to do… something… but June 7th came and went unceremoniously.

So that’s how June, 2010 is going for me so far.

Oh, and I’m still waiting for the WordPress hammer to fall, too.

That didn’t take long.

What I found was the basic difference between PoTTY & stunnel was that when PoTTY opened a obfuscated-openssh link, the link stayed open.  As long as the link was open, the SWG did not log it.  As soon as the link was closed, it did get logged.

Which still bugs the fuck out of me because connections from PuTTY, PoTTY’s daddy, are logged as soon as they’re made.

A real head-scratcher, but I haven’t looked at PuTTY’s connection at the wire level yet.

Still, the delayed-logging is something of a feature.  You can leave the link open for days and it will not show up in the logs until you drop it.

There is one positive outcome to this: the Secret Sauce can now be revealed.

The time has come!

I haven’t had a lot to say the past month because I’ve been a very busy boy.  As they say, I have a lot on my plate.

Unfortunately it is a plate full of shit.  A variety of turds, but all shit nonetheless.

One of those turds is an evaluation of a “Secure Web Gateway” (SWG) from a manufacturer who will remain nameless.

One of the functions of this SWG is Data Loss Prevention (DLP).  This box wants to know everything about anything that’s coming and going through the corporate firewall.  Web pages, email, ftp sessions, you name it.

It accomplishes this by being very snoopy, to the point of sniffing SSL traffic as it passes in and out of the box.

And you thought SSL was secure, didn’t you?

Not when it goes through an Evil Proxy!

Especially not a proxy that spins faux SSL certificates on the fly, which is exactly what this thing does.

Nothing is safe.  Credit cards numbers?  BAH!  It eats them for breakfast.  Those torrid emails you send to your significant other from your Yahoo account?  BLAM!  Laid bare for all to see.  Nothing evades its steely, unimpassioned gaze.

You are 169% PWN3D if you’re unlucky enough to be stuck behind one of these suckers.

Do you scoff?  Think you can get away with SSH tunnels?  Think again.  SSH is not SSL. Never has been, never will be.  SSL wants certificates.  SSH wants session keys.  Your SSH tunnel will simply choke trying to get through.  I know.  I’ve tried.

So what is a L337 H@><0R to do?

Well, there is a half-solution out there.  It’s called stunnel and it allows you to SSL-ify programs that don’t normally do SSL.  It’s been around for ages.  Since the turn of the century!  Very mature. Very robust.

But there’s bad news.

THERE’S NO FUCKING PROXY SUPPORT!!!!!  TEN FUCKING YEARS THIS GODDAMN THING HAS BEEN AROUND AND THERE’S NO FUCKING PROXY SUPPORT!!!!

Testing this SGW last week, I ran across the same exact problem with OpenSSL’s s_client tool.  S_client allows you to peek into the SSL handshake between a server and a client to see what’s going on.  BUT IT DOESN’T FUCKING SUPPORT PROXIES EITHER!

Jesus FUCKING Christ what is it with these people???  Proxies have been around a damn site longer than SSL.  Where is the love?

I was lucky enough to run across a patch to an EIGHT YEAR OLD version of OpenSSL that adds proxy support.  It was never accepted by the OpenSSL geeks.  WHY IS THAT?  WHAT IS THE PROBLEM? In spite of its advanced age, it worked perfectly during my testing.

And, luckily enough, I found a patch for a five year old version of stunnel.  I haven’t tested it yet, but at least it compiles (although Cygwin barfs all over it at link time).

So what is the point?

Like I said, SSH is not SSL.  The SWG expects a browser-like connection on the inside and a Web server-like connection on the outside.  SSH through a (normal) proxy is a simple HTTP “CONNECT”, a straight-through pipe with no muss or fuss.  You need to make the SWG think you have a browser/server handshake going on before it will make the pipe.  Once the pipe is there it will decrypt everything going through it in order to look for naughty bits.

At the point the pipe starts flowing, you just shove your own encrypted tunnel through it.  It can hack away at that traffic all it wants, it’s not going to find any naughty bits.  And if your encrypted tunnel terminates at an obfuscated-openssh server, you are doubly protected since it will never see the initial SSH key exchange (it may not even be looking for it but… You Never Know).

But… and this is a BIG BUT… if the SWG is built right – and I’m 99% certain it’s not, given the dismal track record of this particular vendor - it should be able to tell that the data going through it is encrypted and kill the session (which seems – to me – like the Right Thing To Do if you’re truly concerned about Data Loss Prevention).  Worst case it will either eat up CPU cycles or outright crash and burn. Best case, it will pass the traffic without making a peep or logging anything at all.

Whatever the outcome, it will be reported here!