At its core, Splunk satisfies the offloading and centralized logging requirements dictated by the Defense Information Systems Agency’s (DISA) Security Technical Implementation Guidelines (STIGs).
While collecting logs that DISA requires may check a few of the boxes toward achieving compliance, so much more can be done with the logs files than just holding onto them until they’re needed. Rapidly searching log files for quickly identifying root causes, correlating events across multiple log files from multiple systems simultaneously, setting up alerts based on designated error codes, and providing dashboards/reporting capabilities represent a cross-section of the additional capabilities available through this powerful tool.
Adding security apps to Splunk such as Splunk Enterprise Security (ES) can be immensely helpful if you have the need and resources to manage them. Splunk recommends that ES should be managed by a team of six people with an increase in hardware resources of 20% over a normal Splunk installation. Other security apps may need some enhancement for them to work more effectively in your environment.
Securing Splunk
A large part of running Splunk in a secure environment is securing Splunk itself. Splunk Hardening Standards are a great starting point to enhance the product’s security.
- Securing Splunk will involve having to turn parts of the base install of Splunk off depending on the use of the Splunk server. On a search head/indexer, the Splunk KV store can be turned off. (In general, it is a best practice to disable features of applications that are unused to prevent any exploits.)
- Splunk also provides methods to allow for single sign-on setups for larger environments.
- In addition, Splunk can use common multifactor authentication methods such as RSA and can be configured to work with LDAP.
Securing Communications
Securing Splunk communication is also very important, whether it’s securing user’s access to the Splunk web interface or securing server to server communications.
To secure both, you’ll need to replace the self-generated certificates and replace them with third-party certificates. Each of the search head/indexers and heavy forwarders will need a set of certificates as well. The certificates for server to server communication will need to contain the full certificate chain arranged from top to bottom in the following order, server cert, key, intermediate CA cert, and root CA cert, all in PEM format. Certificates for web authentication will only contain the web cert in PEM format.
Server certs are utilized in the server.conf, the inputs.conf and the outputs.conf. The web certificate should be referenced in the web.conf to secure the web interface.
server.conf:
inputs.conf
outputs.conf
web.conf
Reviewing Security Logs
A security best practice recommended by DISA involves reviewing server access logs on a daily basis. Even on a relatively small environment of less than a hundred servers, this could be an insurmountable task. There are a couple of issues with trying to meet this requirement manually. First, how many person hours are organizations willing to dedicate to this? They’d need a team reviewing hundreds if not thousands of lines per server per day looking for entries that don’t belong. Secondly, how much havoc could a single malicious person do in an environment if they were not detected for 24 hours?
The solution to these challenges is to leverage the logs that are already being collected and indexed by Splunk, building a look up that reviews the logs based on a pre-determined amount of time that fires an email alert when certain specified events occur. An added security benefit is using Splunk to monitor accounts that are shared, that should only be accessed during certain times, or software only accounts that shouldn’t be logged into by anyone. By using Splunk to check for these various scenarios, you are greatly enhancing the overall security posture of the operating environment to identify potential issues in near real-time, all with no additional licensing costs.
Malware Detection
Splunk can be configured to enhance malware detection, improving compliance with DISA guidelines. Certain system files require monitoring for any changes which include passwd, shadow, and inetd.conf. Splunk offers a universal forwarder called filemonitor which provides an alert if any changes are detected. As part of these guidelines, home directories for all users should also be monitored for any changes. If a directory monitor is set up to watch these directories it will capture all changes, to include the bash history.
Data Separation
Data separation is a key security component that Splunk greatly enhances. Different logs should be set to be sent to different indexes within Splunk. After the logs are separated into different indexes, roles are created that control access to the indexes, and finally users are assigned roles, which controls access based on need. For example, Developers may need access to application logs, but shouldn’t retrieve system or database logs, the separate indexes and roles support this level of separation. Restrictions can even be set to allow users only read access to logs on the Splunk server without allowing export to their local machines.
Additional Apps and Add-ons
There are many apps and add-ons available in the Splunk-base to assist in satisfying many STIG requirements while improving the overall security posture. However, some special consideration may be needed when using apps and add-ons. For example, using the Splunk DB Connect app provides great benefit in collecting logs and metrics from Oracle databases, but your Splunk servers in the environment may need to be configured in specific ways to ensure app usage is done in a STIG compliant manner. STIGs require the kvstore on a Splunk search head/indexer be disabled, this is a best practice that was mentioned earlier, but in this instance is required. The kvstore is one of the elements of a Splunk install that the DB Connect app relies on. To maximize the available functionality and to allow for Splunk DB connect app usage, a second Splunk server should be seat up as a heavy forwarder that’s leveraged into doing double duty as a data collection node.
An added benefit of using the Splunk DB Connect app is utilizing it to enhance another tool commonly used to manage Oracle related components, Oracle Enterprise Manager (OEM). OEM is a great tool managing WebLogic and Oracle databases, but it has some limitations particularly in the form of reporting that can be overcome through Splunk. My co-worker has already written an excellent post on this which provides step by step instructions on the setup: Using Splunk with OEM Metrics.
Additional apps might be useful for enhancing security but were designed and built to be run manually, such as the SQL Injection Search. This app isn’t particularly useful in the form that it was originally built. The app requires the user to manually enter the sourcetype of the logs they would like to search and which of the queires that are built into the app they would like to run against the logs. The app does not provide a drop-down list for either the sourcetype or the query, both must be typed into text boxes. However, when the app is deconstructed it contains some comprehensive regular expressions (REGEXs) to detect SQL Injection attempts. These REGEXs can then be taken out, put into a search, and scheduled to run as you’d like against many different sets of logs, firing an email alert if an event is found.
The possibilities are seemingly endless when it comes to Splunk security features, enabling a robust solution for any enterprise. Did I miss anything important? Let me know your thoughts or questions by sending me a note through AEM's website.