The book Implementing Splunk: Big Data Reporting and Development for Operational Intelligence written by Vincent Bumgarner and for which I was a reviewer (yes, it’s a unashamed advertisement) is now available.

February 1, 2013, 2:42 pm lock

Each applications, OS, network and security devices have their own way to log events, and so far, there is no widely adopted standard that allow to easily integrate all logs into SIEM solution. Here’s the main standard and their key points:

The IDMEF standard, mainly focused on IDS, is now almost dead.

CEF (Common Event Format):
The Common Event Format was introduced by ArcSight, and is mainly targeted and adopted by a few security vendors (which is ArcSight main business). CEF only defines a record format, which is simple and extensible. Device vendors can implement it easily over the protocol of their choice.

CEE (Common Event Expression):
The CEE language defines:

  • Taxonomy: a data dictionary and object-action-status taxonomy (ACTION, OBJECT, and STATUS to indicate what happened, to whom did it occur, and what was the result) 
  • Log Syntax (enconding in JSON or XML)
  • Log transport (transport protocol characteristics to comply with the standard are specified, defining a level of compliance)
  • Event log recommendations
CEE goes far beyond the scope of CEF, by specifying protocols details and providing details about what should be logged.
The standard is still in version beta, and we will have to wait to see if it is widely adopted by the IT industry. But the fact that it is led by Mitre and that it is publicly available are points praying in favour of its adoption. But will it be enough to catch up with CEF advance ?

RFC 5424 (syslog):
This new RFC updates the venerable syslog protocol and, while keeping backwards compatibility, corrects several aspect of the protocol (time zone, character encoding). But it also introduces optional structured data between the header and the message, and defines data identifiers. CEE and this new protocol might have shared several things, but in practical, CEE is not using the structured message opportunity offered by the new syslog (which avoids to add this syslog version as a prerequisite, that would have slow CEE adoption). 

What about support by log management/SIEM products ?
Of course, ArcSight supports is own protocol (CEF). Splunk can interpret natively all data in the name-value pairs form, so CEF, CEE and RFC 5425 can be understood by Splunk with not much effort. Moreover, Splunk has is own standard, the Common Information Model, mainly intended for Splunk apps developers (also defining tagging for Splunk apps internal usage).

Event interoperability is the first step to normalisation and categorisation needed for an efficient analysis of IT logs, but so far, no standard is clearly emerging, and connecting different sources of events will still require manual efforts for a while.

December 30, 2012, 1:13 am lock

The latest version Splunk (5.0) is now out, with some nice improvements:

  • The most visible missing feature for users (customers ?) is PDF report generation: Splunk is now able to generate natively PDF reports (including for report scheduling). You can forget the crappy PDF report app .

  • Report acceleration (similar to ArcSight trends) that allows fast reports generation even on huge amount of data is now accessible with one click action. It differs from summary indexing on a few points: it is done on the indexer, it does not requires a summary index, but all searches does not qualify for it. For the latest one, continue to use summary indexing.

  • Index replication: for high-availability deployments, you now have an option to replicates indexes to avoid losing data.
  • There are other improvements about the API and a focus on big data. Read more here.

October 31, 2012, 4:00 am lock

In a non-dsitributed architecture (your indexer is also the host receiving the events), you might want to keep Splunk running as a non-privilegied user but still be still receive syslog from remote hosts. You have (mainly) two solutions:

  • Setup your favorite syslog daemon (syslog-ng or rsyslogd) to listen to port 514, and then configure Splunk to read the syslog files (this will have the advantage to avoid missing any syslog message when you restart Splunk.

  • Use iptables to redirect incoming traffic from port 514 to an non-privilegied port (like 1514) and setup Splunk to listen to the same ports:

    iptables -A PREROUTING -t nat -p udp –dport 514 -j REDIRECT –to-port 1514
    iptables -A PREROUTING -t nat -p tcp –dport 514 -j REDIRECT –to-port 1514

  • March 13, 2012, 3:10 am lock

    Splunk 4.3 is out for a few days, and this new release contains some nice improvements:

    • Sparklines (like in BlueCoat): * | chart sparkline count by host gives the following result:
    • Splunk Sparklines

    • Flash is replaced by HTML5 (for recent browsers; flash is still used for old browsers), but the behaviour of flashtimeline or reports is kept unchanged. Allows mobile device usage for accessing Splunk dashboards.
    • Real-time backfill for Real-time Views by default
    • Dashboards can be edited by users using drag and drop, without having to use XML.
    • IPV6 support for searches, web interfaces and distributed deployments
    • Bloom filters to enhance performance
    • Structured data field extraction for JSON and XML
    • Data preview is now available when importing data from files

    • And some more new stuff

    January 13, 2012, 1:00 am lock

    Arcsight recently presented their new version of the Logger. Some of the new features are:

    • Distributed reports over multiple Logger
    • User configurable dashboards
    • Event summary (overview)
    • Live event viewer
    • LDAP and AD directory integration
    • dedup and transaction search commands
    • SNMP polling support

    Read the complete article »

    December 20, 2011, 3:01 am lock

    The main log management solutions available on the market have different features, and different way of handling the data. This article focus on how ArcSight Logger, Loglogic and Splunk are handling archives, and what are their integrity functionalities.

    ArcSight / Loglogic / Splunk

    How the different log management solutions are handling the data archiving ?
    ArcSight allocates data by one gigabyte block on the Logger. In case of very low rate of events in a specific data group, the archive will still have a one gigabyte size, even if the original events are less than on gigabyte. If archive are scheduled, this can only by done day per day, for the previous day. However, archiving data does not means that the data is removed from the Logger: instead, the data is marked as ready for being overwritten, based on the retention settings.
    Once archived (and overwritten), the data can still be included in the searches if “reloaded” (it is just accessed from the storage location, not copied back locally to the Logger and index is not available anymore).
    Read the complete article »

    September 5, 2011, 12:42 am lock

    The Juniper VPN SSL solution (Secure Access) is undoubtedly the most advanced of the market today, and I’ve always been satisfied with it. However, a few days ago, one of my customers show me his VPN SSL, for which he enabled the “virtual keyboard“.
    I’ve never been really convinced about the security level added by virtual keyboards. Even if it prevents key loggers from capturing sensitive credentials, it is more easy for someone to see the code clicked with a mouse than with a keyboard. And more important, if a malicious software is able to intercept keystrokes, he can take screen-shot when the user is clicking, or sniff the password inside the browser before it is sent to the server (for the worst virtual keyboards ones).

    In this case, the Javascript based virtual keyboard just enters the values in the password input form, which is read-only.

    So, I’ve wrote a simple Greasemonkey script that modifies the Juniper web pages and allows to enter directly the password in the form instead of using this annoying keyboard: JuniperVpnSslRemoveVirtualKeyboard.
    Read the complete article »

    August 15, 2010, 1:18 am lock

    As reported by Kaspersky, most browsers (and proxies ?) supports URL with IP addresses in format others than decimal, which can be a good way to bypass network security:

  • http://0×42.0×66.0×0d.0×63/
  • http://0×42660d63/
  • http://1113984355/
  • http://00000102.00000146.00000015.00000143/

    The previous URL are working with both Firefox and Chrome.

  • March 24, 2010, 2:15 pm lock

    You will find below a patch for WAFW00F (a tool used to fingerprint Web Application Firewall) that allows to identify Imperva SecureSphere WAF.
    On characteristic of Imperva is to respond with an HTTP/1.0 message, even if the request is made in HTTP/1.1. The other WAF I’ve worked with do not have the same behaviour (but there may be a few false positive).

    This was tested with Imperva 6.2 and 7.0 in transparent bridge mode.

    The whole modified can be found here (or an the project issue list) and the diff is below.
    Read the complete article »

    October 29, 2009, 10:14 am lock