Hinky’s Proxy Obsession

I got bored with snort and intrusion detection a few years ago.

The last thing I did with snort was try to code a netfilter detector/packet killer for one of Luigi’s UT99 hacks.

It was an epic FAIL.

I even read the docs!

But back in the day I ran snort & ACID (now BASE – how clever) on a number of boxes, both at work and at home.  I got very tired of seeing all those false positives, every day, day after day after day.  And then reading in the snort online docs that a particular rule I was unfamiliar with had “no known” false positives or negatives.

Riiiiight.

And then snort went commercial.  Well, good riddance.

So it’s been at least three years since I touched it.

Suddenly, out of nowhere, my boss the Security Thought Leader decides snort has to be deployed throughout our organization on every-fucking-thing that can run it.  And since I was stupid enough at one time to open my mouth and let the word “snort” fall out of it, it’s my problem.

Lo and behold we’re now up to snort v2.8.6.1.  And it turns out to be a bad time to be implementing snort.

As of June 2010 (Hey!  That’s like… now!) a lot of the old freeware utilities for managing snort are obsolete.  Not that they were any good, but their very existence was helpful in avoiding reinventing the wheel.

And the more things change the more they stay the same.

Like depending on Libnet v1.0.2a.  They still give you a link to a dead Web site in the docs so you can download this ancient piece of code required to build snort.  Luckily (I think it was luck) freebsd.org still had it in their archives.

Also luckily Mr. Security Thought Leader only wants to send snort alerts to a centralized SIM/SIEM syslog server, which simplifies everything (although he doesn’t know that… he would complicate the fuck out of everything if he did know it).  This is the only documented design requirement in the whole project:

Send snort to syslog!

That’s it!  Nothing more.  What he plans to do after that is a mystery.  Right now I have the Proof of Concept running on a few Windows servers and as usual they’re not producing anything of any interest whatsoever.

The Linux servers are another matter entirely.

I hate to say it, but our *ix admins are the biggest newbz I’ve ever met in my long and glorious IT career.  They need to have a vendor hold their hand whenever they do anything and their collective mantra is the age old refrain:

THE VENDOR WON’T SUPPORT IT!!!

To make matters worse, they are going to have to build snort if their vendor doesn’t distribute a 2.8.6 version of snort.  And considering they run Red Hat, they’re going to have a long wait.

So with all that in mind I built snort 2.8.6.1 for my Linux firewall (good old Debian Etch) and configured it for syslog alerts only.

It’s been running for about an hour now and all I’m getting is:

(http_inspect) LONG HEADER [Priority: 3]

… from just about every Web site I visit.  You would think with all this proxy crap running every hour I’d at least get a “Hey! Somebody’s using an external proxy” kind of alert but no.

There’s obviously some tuning that needs to be done (max http header was 750, so I upped it first to 1024, which wasn’t enough, and then to 2048), but with all the doom & gloom about hackers and cybercrime, et cetera, I really expected a shitload more alerts.

I guess my firewall rules are pretty good after all.

FWIW, it took less than ten hours for some idiot named Tedickhead to remove my PoTTY entry at Wikipedia, without any reason given.  Of course, it pissed me off but I refrained from putting it back in for the time being.

I have to admit to being not all that obsessed lately.

The Websense shit took a lot out of me.  In the end it turned out to be an ISA configuration issue, which made me feel a little stoopit, but at least it was an extremely esoteric (“undocumented”) aspect of the configuration.  The fix Websense published seems to leave much to be desired, and is probably exploitable as well.  Time will tell.

I got into a spontaneous SSH shit-storm on Full Disclosure just to do a little promotion for obfuscated-openssh and PoTTY, but apparently nobody cares.

It seems people are just stuck in their ways and no one is going to do anything any different than whatever it is they’re doing right now.

But somehow I did get this guy to mention PoTTY in his blog and give me and obfuscated-openssh a few links.  Spread the word!

Last week I spent an extraordinarily brutal amount of time compiling Google Maps (here and here), which has also added to my general burn-outedness.

On top of that I had some hardware issues in May that caused the List to malfunction in June.  In fact I just noticed that today.

I had a power supply die in my decade-old Windows 2003 server, which was were I was dropping the nightly backups to the database.

I forgot about that.

Subsequently, the mount point where the files should have gone was pointing to itself, which caused the root partition to fill up.  And when that happens things start to choke.

I only noticed it after seeing the List stuck at 666 proxies for about a day.

Normally when power supplies blow it’s a mad scramble to the nearest computer store to get a replacement, but this time I just said FUCK IT and let it slide, ordering a replacement from Tiger Direct.  But a couple of days later I decided to clean up the computer room and discovered I already had a replacement power supply.  And although I didn’t really want to replace it (having already said FUCK IT and being burnt out on technology in general), I replaced it anyway.

To top all that off, rumors are buzzing at work about a new re-org, so I took a five day mini-vacation (Thursday through Monday) to depressurize.

And during all that I missed the two-year anniversary of the List!  I had planned to do… something… but June 7th came and went unceremoniously.

So that’s how June, 2010 is going for me so far.

Oh, and I’m still waiting for the WordPress hammer to fall, too.

That didn’t take long.

What I found was the basic difference between PoTTY & stunnel was that when PoTTY opened a obfuscated-openssh link, the link stayed open.  As long as the link was open, the SWG did not log it.  As soon as the link was closed, it did get logged.

Which still bugs the fuck out of me because connections from PuTTY, PoTTY’s daddy, are logged as soon as they’re made.

A real head-scratcher, but I haven’t looked at PuTTY’s connection at the wire level yet.

Still, the delayed-logging is something of a feature.  You can leave the link open for days and it will not show up in the logs until you drop it.

There is one positive outcome to this: the Secret Sauce can now be revealed.

The time has come!

I haven’t had a lot to say the past month because I’ve been a very busy boy.  As they say, I have a lot on my plate.

Unfortunately it is a plate full of shit.  A variety of turds, but all shit nonetheless.

One of those turds is an evaluation of a “Secure Web Gateway” (SWG) from a manufacturer who will remain nameless.

One of the functions of this SWG is Data Loss Prevention (DLP).  This box wants to know everything about anything that’s coming and going through the corporate firewall.  Web pages, email, ftp sessions, you name it.

It accomplishes this by being very snoopy, to the point of sniffing SSL traffic as it passes in and out of the box.

And you thought SSL was secure, didn’t you?

Not when it goes through an Evil Proxy!

Especially not a proxy that spins faux SSL certificates on the fly, which is exactly what this thing does.

Nothing is safe.  Credit cards numbers?  BAH!  It eats them for breakfast.  Those torrid emails you send to your significant other from your Yahoo account?  BLAM!  Laid bare for all to see.  Nothing evades its steely, unimpassioned gaze.

You are 169% PWN3D if you’re unlucky enough to be stuck behind one of these suckers.

Do you scoff?  Think you can get away with SSH tunnels?  Think again.  SSH is not SSL. Never has been, never will be.  SSL wants certificates.  SSH wants session keys.  Your SSH tunnel will simply choke trying to get through.  I know.  I’ve tried.

So what is a L337 H@><0R to do?

Well, there is a half-solution out there.  It’s called stunnel and it allows you to SSL-ify programs that don’t normally do SSL.  It’s been around for ages.  Since the turn of the century!  Very mature. Very robust.

But there’s bad news.

THERE’S NO FUCKING PROXY SUPPORT!!!!!  TEN FUCKING YEARS THIS GODDAMN THING HAS BEEN AROUND AND THERE’S NO FUCKING PROXY SUPPORT!!!!

Testing this SGW last week, I ran across the same exact problem with OpenSSL’s s_client tool.  S_client allows you to peek into the SSL handshake between a server and a client to see what’s going on.  BUT IT DOESN’T FUCKING SUPPORT PROXIES EITHER!

Jesus FUCKING Christ what is it with these people???  Proxies have been around a damn site longer than SSL.  Where is the love?

I was lucky enough to run across a patch to an EIGHT YEAR OLD version of OpenSSL that adds proxy support.  It was never accepted by the OpenSSL geeks.  WHY IS THAT?  WHAT IS THE PROBLEM? In spite of its advanced age, it worked perfectly during my testing.

And, luckily enough, I found a patch for a five year old version of stunnel.  I haven’t tested it yet, but at least it compiles (although Cygwin barfs all over it at link time).

So what is the point?

Like I said, SSH is not SSL.  The SWG expects a browser-like connection on the inside and a Web server-like connection on the outside.  SSH through a (normal) proxy is a simple HTTP “CONNECT”, a straight-through pipe with no muss or fuss.  You need to make the SWG think you have a browser/server handshake going on before it will make the pipe.  Once the pipe is there it will decrypt everything going through it in order to look for naughty bits.

At the point the pipe starts flowing, you just shove your own encrypted tunnel through it.  It can hack away at that traffic all it wants, it’s not going to find any naughty bits.  And if your encrypted tunnel terminates at an obfuscated-openssh server, you are doubly protected since it will never see the initial SSH key exchange (it may not even be looking for it but… You Never Know).

But… and this is a BIG BUT… if the SWG is built right – and I’m 99% certain it’s not, given the dismal track record of this particular vendor - it should be able to tell that the data going through it is encrypted and kill the session (which seems – to me – like the Right Thing To Do if you’re truly concerned about Data Loss Prevention).  Worst case it will either eat up CPU cycles or outright crash and burn. Best case, it will pass the traffic without making a peep or logging anything at all.

Whatever the outcome, it will be reported here!

Four days of proxies on the front page!

We are living through hard times, boys and girls!  So hard that even the Dinkster himself had to rely on a PHP proxy to do his forum haunting (thank you Baron Munchausen!).

It’s been a rough week on other fronts as well.  I got caught up in McAfee’s mess back at the Salt Mine.  Not only am I the local Network Nazi, but I also manage McAfee’s crappy AV for the entire enterprise.  Luckily that day (Wednesday) I was telecommuting via RDP back-tunneling (over obfuscated-openssh on Cygwin) so I was not in the thick of things.

I was “in the cloud”, as it were.

I was also wise enough never to have installed Service Pack 3 on my Salt Mine PC, so I was one of the lucky ones.  For a variety of reasons, I never trusted it.  I was almost ready to apply it once IE 7.0 came out, but then I heard there was no roll-back to IE 6 on machines with SP3, so I passed.  I have it on all the XP machines here on DinkNet, but I use different AV.

And that was an odd thing.

I have Microsoft Security Essentials (MSE) on my main box and that morning it died.  Very mysteriously.  The little green system tray icon was just plain gone and when I went to restart it from Control Panel, Services the system told me it could not be found.

This was before the news came out that the whole thing was due to a turd dropped on the world by McAfee, so I was quietly sweating bullets.  Had some bug followed me home?  Or crawled through my other covert tunnel, OpenVPN?  I switched boxes while I re-installed MSE on that system.  Then I rebooted it and performed a full scan.  Nothing.

And “nothing” doesn’t mean shit these days, with fast-mutating bugz like Zeus floating around the Interwebs.  The virus definitions you get today are for crap that has been around for months.

While all this is going on I get a call from my sprog, Inky Dink, and it turns out he’s having AV problems too!  And I know damn well he doesn’t run McAfee because I personally installed MSE on his system!

What the motherfucking fuck was going on here?

But it turned out Inky had been victimized by one of those scareware AV programs.  I pointed him to malwarebytes.org and he took care of it himself later that evening.

Again, all this time we had no idea it was a McAfee problem.  What was I to think?  AV software was dying everywhere as far as I could tell from my small corner of the Universe.  Was it cyberwar?  Was the the “Digital Pearl Harbor” the trade press has been crying about for the last four months?  Was Google’s January hack the warning shot?

No.  It was ludicrous.  It had to be a series of coincidences, so I kept my mouth shut during the Salt Mine phone conference.

Other people were not so cautious.  They started spreading all sorts of FUD.  All it takes is one jerk to read one unsubstantiated claim on one Internet forum and as soon as that happens he’s sending e-mail out to everyone and his brother and the next thing you know you’re in full chickens-with-their-heads-cut-off mode.

Luckily even though that particular jerk (our very own local security wannabee) made an idiot of himself that day and cooler heads prevailed.  The only thing he damaged was his own credibility.

By about 10:30AM that morning the news finally came out and we went into Full Damage Control Mode.  When the dust cleared, about 25% of our systems were down.

McAfee later stated it only affected one half of one percent of their customers.  Do tell.  Maybe they based that number on the phone calls they got that day (“All lines are busy, please hold!”).  Maybe they thought it was just rubberneckers that took their site offline.

And WTF happened?

This event was curious in that the update that caused this mess arrived early that day.  Normally, and I admit I haven’t checked in some time, we get that update between 11:30AM and 2:30PM EST.  The timestamp on the files said they came in at 4:37AM.  Why?  Did their QA department in Bangalore (or Shanghai or whatever) take off early that day?

If McAfee’s Legal Department gets their way – and there is no doubt in my mind it will get its way – we may never know what happened.