The main log management solutions available on the market have different features, and different way of handling the data. This article focus on how ArcSight Logger, Loglogic and Splunk are handling archives, and what are their integrity functionalities.
How the different log management solutions are handling the data archiving ?
ArcSight allocates data by one gigabyte block on the Logger. In case of very low rate of events in a specific data group, the archive will still have a one gigabyte size, even if the original events are less than on gigabyte. If archive are scheduled, this can only by done day per day, for the previous day. However, archiving data does not means that the data is removed from the Logger: instead, the data is marked as ready for being overwritten, based on the retention settings.
Once archived (and overwritten), the data can still be included in the searches if “reloaded” (it is just accessed from the storage location, not copied back locally to the Logger and index is not available anymore).
With Loglogic, for high volume of data, archiving logs requires a dedicated Loglogic appliance (ST) ! Even if it has SAN interfaces and specific management for EMC Celerra WORM (Write Once, Read Many), ST appliance is just managing archives for an LX one.
If your volume of logs is not too high, you can use an appliance of the MX series, than can do both indexing and archiving.
Splunk offers the most advanced and flexible data rotation and retention functions: data rolls from one stage to another depending on various conditions, like age and size. Each step can be customized.
Available stages are hot (new data is still added), warm, cold, and frozen (for archive purposes, but deleted by default).
Data integrity embedded functions
Data integrity is disabled by default in ArcSight and Splunk, while activated on Loglogic.
In ArcSight, various algorithms (from MD5 to SHA-512) are available. However, integrity is only done on the raw event, not on the normalized event (so raw event must be also preserved).
There is no validation function (Update: Edge7 published a tool to check integrity). But if the data is stored on the appliances, since there is no access to the system, it can be reasonably considered as safe.
On Loglogic, MD5 integrity is done by default on raw data, and SHA-256 is also available. Integrity verification is manual and has to be done file by file (and there is one file per hour).
Splunk can sign each event, or each bucket of data (a unit of indexed data) when it is archived. On both cases, data integrity is verified on retrieval, and can be done also manually (or scripted) for the archives.
Logs integrity and PCI compliance ?
The PCI DSS requirement 10.5.5 is “Use file-integrity monitoring or change-detection software on logs to ensure that existing log data cannot be changed without generating alerts”. So does most of the solutions available on the market are not compliant to PCI standards out of the box (even if branded as by marketing departments) ? Or does it depends on the QSA interpretation and capacity to check all the requirements (the PCI member’s favorite answer to detailed technical question seems to be “it’s up to the QSA”).
It’s quite unfair to compare those products since they have some quite different goals: Loglogic focuses on compliance, while ArcSight is more security generic, and offers events normalisation and aggregation through their SmartConnectors. And ArcSight Logger is a good log storage solution for their flagship correlation product: ESM.
On the other hand, Splunk is radically different: the Splunk company has no other solution for correlation, so it offers more advanced functionalities, and may compete ArcSight’s correlation functions (and Splunk can even be interfaced with ArcSight ESM). It is also far more flexible, but depending on what you want to do, it might be more complex to configure.