Penetration Testing: Dead in 2009
Bill Brenner, paraphrasing Brian Chess, CSO Online, 8 Dec 08

Brian Chess stirred things up last month by declaring pen testing dead in 2009.  Ivan Acre wrote a response, also for CSO Online: “Twelve Reasons Pen Testing Won’t Die.”   Matasano declared the statement irrelevant: “[pen testing] is synonymous with security assessments…most of my customers use these words interchangeably, and they are some of the most sophisticated purchasers of security.”  Eduardo dicho la misma.   

The whole affair is silly.  Everyone is using the same words, but not talking about the same thing.  It highlights the immaturity of our industry:  we don’t have consistent terminology, much less a common  framework to clarify issues.

Inconsistent terminology makes drama

Each of these posts used “penetration test” differently:

• network vulnerability assessment – a comprehensive network scan to identify vulnerabilities in all network-facing applications
• application penetration test - a comprehensive application review to identify vulnerabilities in one network-facing application
• network penetration test - an attempt to exploit vulnerabilities to gain illegitimate access

As the folks over at SecurityCurve note, the Payment Card Industry Data Security Standard’s requirement 11.3 dictates both a network pen test and an application pen test each year.   Requirement 11.2 dictates a network vulnerability assessment.  SecurityCurve uses the standards to support their position pen testing isn’t going anywhere soon; I’ll use it to reinforce the terminology distinction.

An application penetration test is part of software development.  The release schedule should provide ample time for review after code freeze.  The process should be an orderly progression of iteratively fewer changes.  Network vulnerability assessments and network penetration tests are part of network operations.  The only place the pieces come together is on the live network.  Smart admins and good configuration control make the process as orderly as possible, but network disorder increases with the number of network clients.

Brenner’s article was sloppy.  He characterized Chess’s definition of penetration testing as “the art of probing company networks in search of exploitable security holes that can then be fixed.”    This clearly refers to network penetration testing, but the rest of the article mixed quotes about network penetration tests (Jabbusch) and application penetration tests (Caceres, Riggins).  It’s pretty easy to fabricate drama when you’re asking your sources two different questions.  Did I mention our community needs an common framework?

Chess intended to discuss application penetration testing, obvious since Chess runs a company specializing in source code analysis.  For software development, Microsoft’s SDL is arguably the industry’s exemplar — including application pen testing immediately prior to release.  There’s no real contender for the industry’s network operations equivalent.  I can’t fill that role, but I’ll continue this pen testing conversation.

Network penetration testing:  please die in 2009

I’ll be as clear as I can be:  network penetration testing needs to die in 2009.

Four days before Chess’s “Pen testing will die,” Jack at Uncommon Sense Security wrote:

Come on, there are really only two possible outcomes to a penetration test:
ONE: You confirm something you already know, that you are vulnerable to a sufficiently skilled and determined attacker.
TWO: The Penetration Tester you hired isn’t good enough.
Uncommon Sense Security, The Fallacy of Penetration Testing, 4 Dec 08

A network pen test typically asks one question:  can someone get in?  I’ll save you the cost of those consultants and answer: yes.   If you’re looking for a binary answer and pay anything to get it, it is too much.   Here’s one crucial point:  in the real world, the bad guy comes after your network at the time and place of his choosing. It’s an easy equation:  (1) he documents your network-facing services, (2) waits until a matching vulnerability is published, (3) exploits it before you patch it.  As soon as you limit pen testers to an artificial time window, you remove a crucial attacker advantage.

Attacker advantages don’t stop at time.  Even if you minimize risk to patch cycle exploitation, there’s still inadvertent user actions on your network: through email, web, returning laptops or even USB drives in the bathroom.  Even if your user training is top-notch, there’s still physical access: through wireless, network endpoints in a retail store, etc.   As the value of your data increases, the likelihood of a trusted user being coerced to assist also increases.  The attack vectors are too numerous; networks do not have nicely fenced perimeters with a single exit gate at the firewall.   As long as your threat models assume so, you’re no better off than the French and their ill-fated Maginot Line.  The constancy of ankle-biter malware on your network should be evidence enough.  Arbitrary code execution is inevitable.

You do not need a pen test to test the effectiveness of your network protection measures, you need a pen test to exercise your detection and response measures.

Network security is equal parts protection, detection and response.  Protection is the first line of defense, but when protections are insufficient, you must be able to detect the attacker and initiate appropriate response actions.   Protection is critical, but you can not stop a patient and resourceful attacker. When he lands inside your network, what’s your level of confidence your systems will detect his presence?  Sure, if he uses a cleartext command shell, Snort will trigger.  Sure, if the same code/servers/domains are used in 10 bajillion other places, your enterprise antivirus will quarantine the binaries.    But if his network traffic is SSL and his binaries use a custom packer to evade AV signature?

I’ll save you cost of the consultants again:  you won’t detect him.

Pen testing: a detection and response exercise

Admittedly, the realities are more complicated.  Your tools are more diverse and attackers are sloppy, leaving ample traces to detect.  Here’s the $64,000 question:  when your security guy sits at his console Monday morning — cup of coffee in hand, slightly hungover from watching rugby at the pub the night before — will he recognize the signs?  Or will he just scowl, grumble and move on?   Pen testing should be a detection and response exercise. Not only do we have real technical problems in our detection tools, but your admins have to be constantly alert for any evidence of unusual activity.  It’s hard.  (That’s why the US military uses FPCONs – and recently added INFOCONs)

Ask this of a pen testing consultant and he will pander to you. (“yep, we’ll do that!”) Then most will turn and run nessus or some other automated vulnerability scanner.   They’re cheating you and your admins of a real threat emulation.  You’re left with no idea how your organization will detect or respond in a realistic scenario.   And it pisses your admins off, because when the pen testers can’t find some low-hanging-fruit-stupid-mistake on your perimeter, they “assume access” and proceed to pillage your internal network.  The whole affair becomes counterproductive.

We can’t just retrofit typical pen tests, as the ninnys are sure to suggest.   It’s evident in the terminology itself.  The connotations are entirely misplaced:  a pass/fail attempt (test) to gain unauthorized access (penetration).  The term is adversarial, and not in a good way.   Calling it something more like an intrusion detection exercise will more closely align connotations with the need.   An exercise is still adversarial, but in a better way.

Continuing to test our protections and hope for 100% is futile.  By doing so we sacrifice resources we should be devoting to detection and response.   Don’t try so hard to stop attackers, but expect and plan for  a certain amount to succeed.  Use the resources you save to make sure you know they’re there.