Something that most organizations do not know is that if you are using Splunk, the syslogs should never be used directly in Splunk. It may be one of the easiest ways to get started, yet it will turn out to become a data nightmare. This is an issue that many folks have when first using Splunk. The need for a log management system like Splunk becomes evident when auditors start asking for “logs” while performing duties. In many cases, syslogs are used and collected from a variety of devices and applications. The initial resting place for these logs is on a server with a Linux syslog server running rsyslog or syslog-ng. Less common, Windows holds syslogs in applications like Kiwisyslog.

Without a professional partner like cStor, companies look to the Internet for examples of making network inputs for syslogs. It is an easy and simple way to start seeing results right away. Yet, as previously stated, it is not a good practice. Sending syslogs straight to Splunk provides plenty of headaches. This is not Splunk’s fault; it is just the nature of the issue with most log collection products.

Why not send logs directly into Splunk? Hopefully it is clearly stated—just don’t! And now for the reasons.

There will be a disruption of the data collection. If the Splunk indexer is restarted that data is sent to, the system will lose syslog data. There are several reasons to restart the indexer: applying updates, rolling restarts for index clustering. Restarting Splunk happens more often than it would on a syslog service on a dedicated server. There is a speed advantage to restarting the syslog service, it is faster than restarting Splunk. There is also a loss to the ability to load balance incoming data across multiple indexers (e.g. Index Clustering).

More reasons include that if different types of devices syslog’s stream to the same network input on Splunk, then setting sourcetype and destination index becomes tedious. There is a great deal more flexibility in data handling, routing and filtering with rsyslog or syslog-ng than with a Splunk network port. Think about dropping noisy events before they hit the Indexers. In most cases there are already network ACLs in place and syslog configuration being done on source devices. No changes would be made to that. Looking at a business continuity/disaster recovery plan perspective, when using something like Puppet then re-deploying a failed syslog server with its fairly static configuration is easier and good. As well as having an extra backup of the original log data by archiving it to compressed files automatically.

From a pure security standpoint, If Splunk listens on port 514 it will need elevated privileges for the whole Splunkd process and its child processes. This gives a much smaller attack surface on a dedicated syslog service.

How to prepare for success—here is the secret sauce. Something that a large number of IT groups do not implement is the PTR DNS record for reverse DNS.

Splunk will try and get the host field on network inputs by default using reverse DNS. Syslog-ng and rsylog do this as well. So make sure DNS records are configured. One other item that needs consideration is DNS caching servers. DNS performance and the volume of lookups could potentially be an issue. Please take the time to read more on the topic – Splunk DNS Caching and dnsmasq.

A bonus note. If the Splunk SNMP Modular Input is used, there is now an option to perform the reverse DNS lookup to get the host field information. A FQDN is way better than an IP.

Before jumping into anything with Splunk, prepare the syslog sources by doing three things.

1. Decide on a good device naming scheme—perhaps asa8.phx2.myorg.com represents a firewall in a Phoenix second office location.
2. Implement BOTH A and PTR records. The reverse record is going to be as important as the naming convention.
3. Make sure the syslog sources are using NTP and preferably GMT for timezone.

This is going to give you three key benefits.

1. Wildcards in Splunk forwarder pickup configuration can be used. So if the network team adds a new switch, it will be known. As long as it is named BOTH A and PTR records and sent to the official syslog server, then logs will magically just flow into Splunk.
2. No changes required by the Splunk Admin. It just WORKS for already configured types of devices. The sourcetype can be easily controlled and index device types go into Splunk. The host field is then a useful human-readable name.
3. Easily add automatic metadata in Splunk based on the device naming convention—such as geolocation information.

Make sure there is a good implementation plan for Splunk. Like other data-heavy applications, it can become a mess quickly. Remember, cStor is here to assist you in creating the solutions to best serve your unique challenges. Catch our What the Splunk? Webinar for more insights.