Each applications, OS, network and security devices have their own way to log events, and so far, there is no widely adopted standard that allow to easily integrate all logs into SIEM solution. Here’s the main standard and their key points:
The IDMEF standard, mainly focused on IDS, is now almost dead.
CEF (Common Event Format):
The Common Event Format was introduced by ArcSight, and is mainly targeted and adopted by a few security vendors (which is ArcSight main business). CEF only defines a record format, which is simple and extensible. Device vendors can implement it easily over the protocol of their choice.
CEE (Common Event Expression):
The CEE language defines:
- Taxonomy: a data dictionary and object-action-status taxonomy (ACTION, OBJECT, and STATUS to indicate what happened, to whom did it occur, and what was the result)
- Log Syntax (enconding in JSON or XML)
- Log transport (transport protocol characteristics to comply with the standard are specified, defining a level of compliance)
- Event log recommendations
The standard is still in version beta, and we will have to wait to see if it is widely adopted by the IT industry. But the fact that it is led by Mitre and that it is publicly available are points praying in favour of its adoption. But will it be enough to catch up with CEF advance ?
RFC 5424 (syslog):
This new RFC updates the venerable syslog protocol and, while keeping backwards compatibility, corrects several aspect of the protocol (time zone, character encoding). But it also introduces optional structured data between the header and the message, and defines data identifiers. CEE and this new protocol might have shared several things, but in practical, CEE is not using the structured message opportunity offered by the new syslog (which avoids to add this syslog version as a prerequisite, that would have slow CEE adoption).
What about support by log management/SIEM products ?
Of course, ArcSight supports is own protocol (CEF). Splunk can interpret natively all data in the name-value pairs form, so CEF, CEE and RFC 5425 can be understood by Splunk with not much effort. Moreover, Splunk has is own standard, the Common Information Model, mainly intended for Splunk apps developers (also defining tagging for Splunk apps internal usage).
Event interoperability is the first step to normalisation and categorisation needed for an efficient analysis of IT logs, but so far, no standard is clearly emerging, and connecting different sources of events will still require manual efforts for a while.