6 Ways All-Flash Is Transforming Your IT Infrastructure Terra Verde and cStor form Strategic Partnership to Deliver Turnkey Cybersecurity, Compliance and Managed Security Solutions

EXPERT TIP: Syslogs Are Key to Any Splunk Environment

logo_splunk_2color_KSomething that most organizations do not know is that if you are using Splunk, the syslogs should never be used directly in Splunk. It may be one of the easiest ways to get started, yet it will turn out becoming a data nightmare. This is an issue that many folks have when first using Splunk. The need for a log management system like Splunk becomes evident when auditors start asking for “logs” while performing duties. In many cases, syslogs are used and collected from a variety of devices and applications. The initial resting place for these logs are on a server with a Linux syslog server running rsyslog or syslog-ng. Less common, Windows holds syslogs in applications like Kiwisyslog.

Without a professional partner like cStor, companies look to the Internet for examples on making network inputs for syslogs. It is the easy and simple way to start seeing results right away. Yet, as previously stated, it is not a good practice. Sending syslogs straight to Splunk provides a great deal of headaches. This is not Splunk’s fault; it is just the nature of the issue with most log collection products.

Why not to send logs directly into Splunk? Hopefully it is clearly stated—just don’t! And now for the reasons.

LOGS.wood-828754_960_720There will be a disruption of the data collection. If the Splunk indexer is restarted that data is sent to, the system will lose syslog data. There are several reasons to restart the indexer: applying updates, rolling restarts for index clustering. Restarting Splunk happens more often than it would on a syslog service on a dedicated server. There is a speed advantage to restarting the syslog service, it is faster than restarting Splunk. There is also a loss to the ability to load balance incoming data across multiple indexers (e.g. Index Clustering).

More reasons include that if different types of devices syslog’s stream to the same network input on Splunk, then setting sourcetype and destination index becomes tedious. There is a great deal more flexibility in data handling, routing and filtering with rsyslog or syslog-ng than with a Splunk network port. Think about dropping noisy events before they hit the Indexers. In most cases there are already network ACLs in place and syslog configuration being done on source devices. No changes would be made to that. Looking at business continuity/disaster recovery plan perspective, when using something like Puppet then re-deploying a failed syslog server with its fairly static configuration is easier and good. As well as having an extra backup of the original log data by archiving it to compressed files automatically.

From a purely security standpoint, If Splunk listens on port 514 it will need elevated privileges for the whole Splunkd process and its child processes. This gives a much smaller attack surface on a dedicated syslog service.

How to prepare for success—here is the secret sauce. Something that a large number of IT groups do not implement is the PTR DNS record for reverse DNS.

Splunk will try and get the host field on network inputs by default using reverse DNS. Syslog-ng and rsylog does this as well. So make sure DNS records are configured. One other item that needs to considered is DNS caching servers. DNS performance and the volume of lookups could potentially be an issue. Please take the time to read more on the topic – Splunk DNS Caching and dnsmasq.

A bonus note. If the Splunk SNMP Modular Input is used, there is now an option to perform the reverse DNS lookup to get the host field information. A FQDN is way better than an IP.

Before jumping into anything with Splunk, prepare the syslog sources by doing three things.

  1. Decide on a good device naming scheme—perhaps asa8.phx2.myorg.com represents a firewall in a Phoenix second office location.
  2. Implement BOTH A and PTR records. The reverse record is going to be as important as the naming convention.
  3. Make sure the syslog sources are using NTP and preferably GMT for timezone.

This is going to give you three key benefits.

  1. Wildcards in Splunk forwarder pickup configuration can be used. So if the network team adds a new switch, it will be known. As long as it is named BOTH A and PTR records and sent to the official syslog server, then logs will magically just flow into Splunk.
  2. No changes required by the Splunk Admin. It just WORKS for already configured types of devices. The sourcetype can be easily controlled and index device types go into Splunk. The host field is then a useful human readable name.
  3. Easily add automatic metadata in Splunk based on the device naming convention—such as geolocation information.

Make sure there is a good implementation plan for Splunk. Like other data heavy applications, it can become a mess quickly. Remember, cStor is here to assist you in creating the solutions to best serve your unique challenges. Catch our What the Splunk? Webinar for more insights.

Gregory Kiker
About Gregory Kiker
Gregory is the Cybersecurity Practice Leader at cStor.  The vision of cStor is to provide the means to protect our customers through best of breed products, services, and consulting.  Greg drives this vision with over 20 years of IT experience.  His IT knowledge spans a wide range of disciples from Infrastructure Management, Network Management, Storage, Information Risk Management, Application Development, Database Management, and Cyber Security.  Greg’s executive experience over the years gives him a customer focused perspective and understanding of the special situations that many companies face.  He attended The New Orleans Baptist Theological Seminary  studying Theology and Regis University in Denver studying Business Management.  He is now pursuing a degree in Archaeology in hopes of retiring and mimicking Indiana Jones. 

Comments are closed.