An Information Security Community
Let’s say you are using Intrusion Detection Systems in a hosting environment and you really care about your customers because you are the internal hosting provider for a large corporation and all its subsidiary companies and even some related corporations.
You are hosting webspace on multihoming servers, dedicated servers, email, custom applications and all kind of stuff you cannot control. Some of it is managed by the customers themselves, some is managed by other external firms on behalf of your customers.
You would have to deal with all kinds of attacks and events each day. If you see that one of your customers is using a vulnerable Content Management System or a vulnerable component for a popular CMS, you would want to give your customer a notice, maybe because it’s corporate policy or because you just want to do a good job and add some more security to the world.
What you need in such an environment is a really good IDS, good analysis tools and good correlation. You are low on staff and need something with very little management overhead. You do not want to waste your time with reading release notes, figuring out where your IDS will break if you apply this or that update first, troubleshoot connectivity to remote sensors or learn a new workflow with every product change.
A workflow should adapt to your needs and not to the product you are using.
Let’s say sou do not have the funding and personnel for installing an maintaining a SIEM (Security Information and Event Management).
You need results! You need to know what’s going on. What you need is something that works and gives you high quality intelligence data at a glance. This is where most commercial IDSs and IDS vendors fail today.
And this is also the fault of the IDS customers because most do not really analyze events and thus do not realize that the products they are using suck in many ways. This way the vendor never learns how to do it right - because he doesn’t need to. For most IDS customers, the IDS is nothing more than a checkbox on the corporate compliance checklist.
As a matter of fact, most vendors I have talked to were genuinely baffled that I am actually analyzing events and what results I am getting because I do research about what kinds of attacks we are seeing. If there is a new wave against some popular webapp or CMS, I find out what google dorks the attackers probably have used and so I determine how and why they decided to attack us (of course most of this stuff on the attackers side is fully automated). I check if we - i.e. if our customers - have installations that are possibly vulnerable to that new kind of attack and so on.
Most IDS vendors today promise to automate much of this with a feature called correlation.
What is correlation?
Correlation describes how two or more events are related to each other. The relation between two or more events does not necessarily have to be causal, although for intrusion detection you would probably prefer to know if a relation is causal.
You can correlate all kinds of stuff. Some examples:
- which events came from the same attack source
- which events from different sensors are related? (cross sensor correlation)
- which events from different sources could be related?
- which events originate from attack sources that are already known as such?
(e.g. public blacklists, known botnet C&C servers, known compromised hosts)
- which events come from geographic locations that statistically generate more malicious traffic than others?
- which events are related with regards to behavior?
(correlation of signature based intrusion detection events with statistical data and behavioral anomaly detection)
- which events target hosts or software with known vulnerabilities
- how did the target react or reply to the attack?
Almost all implementations do much less correlation than you’d wish for.
Even if they tell you they do all of the above, they probably do it in a very limited way.
Why? Because it’s very difficult to do! And because most people do not really analyze events.
Correlation with Sourcefire
Sourcefire delivers Intrusion Detection and Prevention with its 3D sensor appliances.
Snort is the IDS/IPS engine, and Realtime Network Awareness (RNA) does passive OS and service fingerprinting. Sourcefire also delivers a vulnerability database.
One of the examples for correlation is the following (vendor quote):
For example, Sourcefire IPS sees a packet in the network that exploits a vulnerability in Windows XP SP2. Because this packet is being sent to a host that has been classified as a Linux host, the 3D System rates the impact of the event as “Not Vulnerable.”
A similar popular example is an attack against MS IIS when the target system is actually running Apache/Linux.
However my observation is somewhat disappointing. When I got an alert indicating an attack against IIS, Sourcefire flagged the event “orange” (impact level 2), meaning the target was “Potentially vulnerable” although the RNA database had 95 percent confidence in that the host was Linux and 100 percent confidence in that the Service was Apache(UNIX). Not exactly what you would have expected, right?
But are other vendors better with correlation? I haven’t seen any better implementation of correlation so far and at least Sourcefire’s implementation has the potential for good correlation. Maybe they are only careful not to be overly confident in their passive fingerprinting features and correlation, so they rather have such an event flagged as “potentially vulnerable”.
Correlation with other vendors
There are several IDS and SIEM products out there, providing some kind of correlation or other.
Tenable Security does something called passive vulnerability scanning (I guess this is somewhat similar to Sourcefire RNA) and correlates this with logs from servers and other sources. IBM ISS has something called Security Fusion, McAfee has some correlation capabilities and other vendors might have as well, even if I did not have the opportunity to look into all of them.
Most vendors use a combination of data that comes as a byproduct of intrusion detection and netflow statistics, as well as data from port- and security scanners like Nmap, Nessus, Qualys or other.
The bottom line is that good correlation is extremely difficult and complex, so most vendors concentrate on things they are confident with - which is understandable. On the other hand, correlation today is not as granular and intelligent as it could be. But hey - you will always need some human brain to make sense of all this.
And the last is a fact most IDS customers grossly underestimate.
Intrusion Event Analysis
Most products really suck when it comes to event analysis.
As far as I know, all vendors today fail to provide the event analyst with useful data right in the alerts he or she is reviewing. Most vendors abstract correlation using some or other kind of impact rank. If you want to know what the system knows about the attack and the targeted system, you have to drill down deep - for each event.
Let me say this very clear, since I have to analyze hundreds and thousands of events each day, especially web-based attacks: I hate having to drill down deep in order to see what’s actually going on!
Sometimes I think for web-attack detection I could skip the IDS front-end and concentrate on web-server access logs, because these are much easier for me to parse than IDS events in the analysis front-end of some commercial IDS product. It’s a joke how bad many IDSs are with regards to the analysis front-end.
Most systems are still optimized for analysis of relatively few attacks against flaws in well known operating systems and applications, not for the now prevalent web-based attacks.
In a hosting environment you will see a ton of web-based attacks (e.g. SQLi) each day.
You cannot filter sql injection in such an environment where you have all kinds of different applications from all kinds of customers, so a WAF (web application firewall) does not really help. And I’d rather have web developers learn how to write secure code than having to rely on a WAF. Most customers are actually very grateful when I tell them what happened and why, and what they can do in order to write better applications.
But back to the example: You have all of these sql-injection attacks, maybe around 500 for the last day. Would you want to have to drill down through two or more layers of details for each sql-injection alert until you come to the actual request that has triggered the alert? Most probably not!
A typical sql-injection would look like this:
What you need now is a view like this:
server, URI, parameter, value, response
some.domain.com, /index.php, id, 4711%20union%20select 200
Displaying multiple rows at a time, you can analyze many events at a time. With some experience you would see at a glance which events are real attacks and which are not. If you see something interesting and new, you would do a quick google search for the attacked component, e.g. “option=com_joltcard&Itemid=XX&task=view&cardID=X+AND+1=2+UNION+”.
You would then find that it’s an attack against a vulnerable Joomla! component and which version of the component is vulnerable. You can then check if some of the websites you are hosting has this vulnerable component installed. If the attacked script or component is unknown, you could use sqlmap, a browser or other tools in order to determine, if there is an sql-injection vulnerability. In order to determine if the attack/request was probably successful or not, you would want to have the server response code in the alert as shown above.
IBM ISS at least provides this kind of view in SiteProtector, but they suck in other aspects.
Sourcefire does not provide such a detailed event table view at all and they do not even provide the response code in the packet data view, because snort works very different from ISS. You would have to add tag directives to each rule in order to record the next packets in the session. But even with that, you would get too much data. Remember, you only want to display the http response code, not the data of the whole response packet. ISS Proventia does protocol analysis, i.e. it parses the protocol, dissects it into all the different fields and it has advanced heuristics for sql-injection detection, which snort has not. So ISS is superior regarding this point.
A snort signature for SQLi would trigger for "State of the Union: Obama selects new head of Information Security"
ISS would not trigger on that because it applies lexical and semantic analysis for this kind of attack.
Both union and select are valid lexically for SQL but in this case even the word selects is not.
And then it does not fit semantically since in SQL the word select follows directly after the word union for a union select, so it would be wrong here to trigger for a union select statement. I am generally disappointed with Sourcefire’s capabilities for detection one of the most prevalent and most dangerous attack types today, which SQLi still is.
The VRT ruleset has very few rules for detecting sql-injections and they are not very good. They do not detect many types of sql-injections like blind injections that are generated by an sqlmap-scan, which is in fact very noisy.
Instead, you have to rely on rules that detect usage of the sqlmap user agent.
I must correct myself for I have written bullshit. The above example is nice for showing the general problem of simple pattern match rules, but the example is not true for VRT provided rules and not even for the Emergingthreats ruleset. The current sid:13990; rev:8 of the VRT ruleset uses a more advanced pcre regular expression for detection of "union select" or "union all select" injections. What is still true though is that there is no good heuristic for detecting all kinds of SQLi. If you know of good snort rules that reliably detect blind and other sql injections and have a very low false positive ratio, please let me know.
I had some good mixed results with rules from www.emergingthreats.net, which are - of course - not supported by Sourcefire. I’d expect more from a commercial vendor who has its own vulnerability research team. I like Snort, but the results have not improved much over the last five years or so.
I am on and off using Open Snort for a while now and I still love it, but it’s time to move on and add protocol analysis and dispose of some old concepts. A snort alert detail in Sourcefire’s DC packet view looks very much like it looked in ACID/BASE for years now.
And the above was mostly about sql-injection so far, not mentioning all the other kinds of attacks like RFI, LFI, command injection, XSS and other.
These kinds of attacks come in the hundreds or thousands each day. So you have to deal with them extremely effectively and efficiently.
As said before, most analysis front-ends suck, regardless if it’s a SIEM or only a front-end for a specific intrusion detection system. McAfee is wasting screen estate with bulky window frames. This front-end is so ugly and inflexible it makes you sick in the stomach. ISS has some good views for bulk analysis. One thing I hate with ISS is its inconsistency with when a table cell can be copied to the clipboard an when not, just to mention one of many examples where ISS SiteProtector sucks (this might have changed in newer versions). One tradeoff with ISSs detailed event view where you really get good insight to what’s going on is the fact that have to scroll horizontally forever since the table is very large. Some more customization options for arrangement of the most important columns would be desirable here.
Today, in an environment where you have all kinds of internet-services and attacks you would need different views for high level data and high volume alerts. For high volume alerts like all kinds of web-based attacks you need specialized views that enable you to skim through the actual data of importance very fast.
Ever skimmed through a webserver access log? You don't even have to read each line - injections, RFI, LFI and directory traversals can be spotted even at very high scrolling-speed. The human eye can easily detect changes from the everyday "normal" attacks. Those new variants are the kind of attacks that I'm after.
The “event details” view of IBM ISS SiteProtector to my surprise is actually the best I have seen so far.
But as said before, SiteProtector sucks in other aspects (like no good facility for creation of custom rules).
For everyday analysis you'd want a lightning fast front-end (time is crucial and nobody likes to work with a sluggish front-end).
In addition to an abstract impact flag or priority you would want helpful information right in the alert:
Did the target reply at all? Was an “attack response-signature triggered? With what http response codes did the target reply if http was used? What OS is the target system is really running (if known), service and version of the attacked port, (if known). Is the source blacklisted, greylisted or is the source a known botnet C&C or is it a compromised server? What's the geolocation of the attacker-ip etc. pp.
I could ramble on like this forever. However I know it’s no easy job to create an IDS and analysis front-end which is really good. But I am hoping that there is hope.