Like Falling Off a Log

William Shatner had the right idea. For 80 episodes of the original Star Trek in the late 60s, Captain Kirk kept a log, complete with star date, so that he could look back at any time and remember what happened. Sadly, over 40 years later, modern IT departments often fail to follow his advice.

Hundreds of thousands of small businesses leave Microsoft Windows Small Business Server humming quietly away in the corner, gathering information on computing events, both expected and unexpected. And yet, because many of these companies don’t even have a dedicated IT professional, these logs rarely, if ever, get looked at.

There’s really no excuse, given that computing logs have been around for as long as the seminal sci-fi show. “Logs were there in the IBM System 360, 40 years ago, pretty much for the same reason that they’re there in VMware version 4.0 – to tell administrators what’s going on”, says A.N. Ananth, CEO of log management and security event and incident management company Prism Microsystems.

Compliance should be driving us to be more diligent about keeping track of what is happening to our systems, and reviewing the results. Even the UK government, which seems sometimes to move at a glacial pace, is beginning to take notice. Mandatory requirement 37 within its Security Policy Framework calls for “A forensic readiness policy that will maximise the ability to preserve and analyse data generated by an ICT system that may be required for legal and management purposes”.

In the private sector, regulations such as PCI mandate effective log management. “People spend a lot of money on PCI going through the check boxes”, says Reed Henry, senior vice president of marketing at ArcSight, which sells log management solutions. “Requirement 10 of that standard says that you must collect logs on what has happened and review them daily.”

One reason why many companies may neglect to manage their logs properly is that the term implies no clear benefits. Ananth argues that local management is a feature – a means to an end – rather than a benefit. The meaningful application of log management lies in security event and incident management (SEIM), which takes log information and uses it both to highlight immediate threats, and to help analyze breaches that have already occurred.

SEIM provides a meaningful filter that enables administrators to better understand their logs. Ananth says that the application can look at system logs and identify three classes of problems.

“There is the class of problem that you understand very well”, he explains. Twenty logins from the same IP address is an issue that you can predefine rule templates for in an SEIM system.

“The second class of problem is the one that you’ll identify when you see it”, he says. If a database administrator grants access to payroll information at 2 AM on a Sunday morning in an organization that operates from 9 to 5, five days per week, it might raise alarm bells. However, it might not be the kind of incident you can predict.

Finally, an SEIM system might derive information about a hitherto unknown zero-day attack from system logs, he warns.

Allocating the human resource to analyze that data will be the initial challenge for many companies, although Ananth says that even 10 minutes a day scrutinizing the logs, or an SEIM system that collates those logs, can be enough to highlight the majority of security issues.

“Look at the summary. If something bothers you, then drill down into detail”, he advises. “For example, we have customers that look at failed logins, or sometimes successful logins, but only after hours.”

Division of Labor

Many companies won’t yet have a procedure to analyze the information. Which begs the questions: Who should do it? Who should they confer with? How should issues be escalated?

Andy Morris, product marketing director at log management tools company LogLogic, says that many of his more successful customers create three roles to support successful log management and analysis: a log analyst, a security management lead, and an investigator.

The log analyst produces a regular summary report, pulling out trends that might be appropriate for further investigation. Anything out of the ordinary, such as a spike in unsuccessful logins from a particular IP address, might make it into the summary.

“Every week or two they get these teams together”, he says, explaining what happens when the log analyst reviews the report with the security management lead. “The security lead will take that information and say ‘you’re seeing an odd trend – that could be indicative of, say, a denial of service attack’”, Morris says. They can then correlate that with other events that they’re seeing in the industry, and create rules to deal with it in the future.

The investigators will then work through the logs to find a source for the specific event that triggered the security measure. They might use event information to identify, for example, whether the perpetrator was external or came from within the organization.

“This is all driven by the log guy, because the log guy is generally finding the anomalies”, Morris says, adding that in his experience, such meetings generally take one or two hours. Occasionally, however, the process may be driven by the security management lead, who has identified a current security trend in the industry for the log analyst to begin looking for. An example could be repeated attempts to contact internal machines via port 443, which was one of the initial infection vectors for the Conficker network worm.

A Lack of Precise Standards

One of the biggest challenges facing log management and SEIM firms, and their customers, is data collection. One might think that after 40 years, the industry would have arrived at common standards for both collecting log data from heterogeneous systems, and exchanging them between different log management products. After all, as organizations grow in size, they are likely to use different log management in different divisions.

Unfortunately, data collection and log management formats are decidedly eclectic. “It’s one of the biggest challenges that we have in the industry. There is no standard around logging”, Morris acknowledges. He describes two layers of analysis: normalization and categorization.

Normalization involves translating log event information from particular devices into a standard schema developed by the log management company. Morris has device experts in-house that interrogate devices, making them produce as many log lines as possible, and then translate those into a normalized schema that the company’s product can use.

Second, the company lumps multiple types of log events together that may describe a similar incident. Even if a device describes an event in three different ways, from an administrator’s perspective, the issue may be the same. So LogLogic had to develop a taxonomy that makes its logs easier for log analysts and administrators to understand.

While both vendors and their customers tackle these challenges, the industry is pushing further still. Ross Brewer, vice president and managing director, EMEA, for log management firm LogRhythm, wants to introduce more business intelligence-style functions into log management systems that will make it easier for systems administrators to find answers to the kind of questions that compliance managers might ask.

“The auditors will come in and say ‘show me all of the configuration changes carried out by administrators in the past 60 days’”, he explains. He argues that in many cases, current log management systems are not equipped to answer such questions, because a configuration change might take on multiple forms within a particular system’s log formats, making it difficult to categorize under a single search.

Prism’s Ananth wants to see local management and its applications pushed even deeper into the organization, becoming relevant not just to IT staff, but also to the line of business managers that they ultimately support.

“The custodians of the data and its owners are very different”, he says. “Right now, logs target the custodians. But if there’s something screwy going on with the data, then the custodian would never know it.”

Part of that journey involves presenting information to business managers in the format that they can understand. In many cases, they will bring their own business knowledge to the table, and will use the log data presented to them in ways that IT staff may not.

Roles play a key part in that process, according to Todd Chambers, chief marketing officer at log management company Courion. Allocating specific roles to employees within an organization, along with an understanding of appropriate systems activities for them, will help to translate anomalous IT events into something that business people can understand. “The ability to look at your entitlements and use those basic roles to identify when people are operating outside policy is becoming more sophisticated”, he says.

“Scotty, we need more power!”

We are beginning to see applications drawing on log events that will bring us closer to this kind of knowledge. For example, at the recent RSA Conference, Symantec introduced a data indexing technology called Data Insight, which correlates unstructured files such as Word documents with the people who own map data in the business. This information, combined with log data, could help to raise the alarm when, for example, an inappropriate person accesses a file that is no longer owned by someone within an organization, because the original creator has left.

We still have a long way to go before we can achieve quite the same level of sophistication with our logs as Gene Rodenberry originally envisaged for Captain Kirk. But then, Captain Kirk was the equivalent of the CEO in a modern organization. IT logs are the purview of the modern-day Scotties – the boys in the engine room. As any IT professional will tell you, keeping the corporate warp drive ticking along, regardless of any threats that may emerge, is a more difficult job than many would care to admit.