Sysmon to Splunk

While Windows Event Forwarding (WEF) is great for collecting all your events, it’s not as easy to use as software built for indexing and searching all this data; plus, with Splunk, you would only have to learn one main search syntax. The below instructions, work on getting your new data to Splunk along with some technical add-ons and quick searches so you can start utilizing this data. I am running the free Linux version of Splunk since this is my test lab. Due to this being a test environment, you will see things that are not best practice for a large environment as indexes and other features that are not included in the free version. A few important things, you will want to change the index to your Windows event indexer on all the search queries I present unless you want to wait for it to check unrelated logs in other indices.

  1. On your Splunk instance, install the Windows TA.
    2. root@Splunk:/opt/splunk/bin# ./splunk install app /tmp/add-on-for-microsoft-sysmon_604.tgz
  2. Also, let’s add the VirusTotal checker to our Splunk installation. We can use this to compare hashes against VirusTotal without leaving Splunk.
    1. Installation: root@SIEM:/opt/splunk/bin# ./splunk install app /tmp/virustotal-checker_13.tgz
      App ‘/tmp/virustotal-checker_13.tgz’ installed
    2. Note: You need to restart the Splunk Server (splunkd) for your changes to take effect.
    3. Continue to Step 6 for use of this query.
  3. The installation of the Splunk Universal Forwarder on the WEF server is straightforward.
    1. Select the license agreement and for on-premise deployment. To continue, select Customize Options.
    2. Click Next past the SSL information as we haven’t set up a certificate authority with this domain.
    3. On the following screen, select Local System as all the events will be forwarded to this device.
    4. On the Events and logs to collect, I select everything as we are just testing this product out. Below we will use the Performance Monitor to create some graphs on system usage.
    5. You can select a deployment server but this is optional.
    6. Required: This is the screen where we will enter the IP address of the Splunk instance along with the port. After this, just wait for the forwarder to install.
  4. Monitor Server Performance (Perfmon): Once data is being collected and indexed by Splunk, one quick search can reveal CPU and memory usage over the past X amount of time in graph form for the WEF server.
    1. Search: index=main host=<ComputerName> sourcetype=”Perfmon:CPU Load” | timechart span=5m avg(Value) as cpu_usage by host
    2. Example:
  5. Another quick one we can do is with available memory for a host to show you the baseline of what the server is using over the course of a day.
    1. Search: index=main sourcetype=”perfmon:available memory” host=<ComputerName> | eval memory=Value/1000000 | timechart span=15m avg(memory) as “Available Memory” by host
    2. Note: The value on the Y-axis is in Megabytes (MB) and can be changed with the eval transform.
    3. Example: index=main sourcetype=”Perfmon:Network Interface” host=”nathan” counter=”Bytes Received/sec” | eval RValue=Value*0.000008 | timechart span=10s avg(RValue) as “Received Traffic” by host 
    4. At the end, we can combine all to a dashboard to monitor our event collecting server.
  6. Virus Total Lookup (Sysmon): You can also run the following queries to report back the processes that have started along with the SHA256 hash.
    1. Tip: You may want to extract the SHA256 and MD5 hashes to skip the REX command to pull out the hash value. The below examples will use REX for each search. See 7D for extracting these fields.
    2. Search: index=main host=”<Forwarder Name>” “<Computer Name>” EventCode=1 | rex field=Hashes “SHA256=(?<SHA256>.*)” | stats count by ParentImage, SHA256
    3. This has a regular expression to pull out SHA256 of the file that executes. Going forward, I will pick SHA256 for comparison to VirusTotal.
  7. Run the following query and it will check the SHA256 against VirusTotal. Note: This will limit you to 5 hashes and I usually setup an Event action for it. This will allow you to send the hash to VirusTotal and open a new browser window for you.
    1. Search: host=<ComputerName> sourcetype=”WinEventLog:Microsoft-Windows-Sysmon/Operational” EventCode=1 | dedup Hashes | rex field=Hashes “SHA256=(?<sha256sum>\w+)” | rename sha256sum as “sha-256sum” | vt field=sha256sum | table ParentImage, sha256sum, vt_link, vt_detection_count, vt_ratio
    2. As can be seen below, I ran a check against Splunkd from the universal deployment.
  8. We can also set up a Workflow action in Splunk that will allow us to click on a particular event and have it search VirusTotal for that hash. This will open up a new browser window with the results.
    1. First, we will want to extract the SHA256 hash from the Hashes field.
    2. Extraction RegEx: SHA256=(?<SHA256>\w*)
    3. Below is how we will configure the lookup with a Workflow Action. This will basically take a field and pass it over to a website, in our case VirusTotal.
    4. Once this is done, we can easily search VirusTotal from just a basic search on EventCode 1 if there is anything that needs further analysis.
  9. Network Connections: The following search will show you initiated connections and includes the program that initiated each one. I would probably set this to specific sources, destinations, or programs; otherwise, it will provide too much data to look through.
    1. Search: EventCode=3 Protocol=tcp Initiated=true | eval src=if(isnotnull(SourceHostname), SourceHostname, SourceIp+”:”+SourcePort) | eval dest=if(isnotnull(DestinationHostname), DestinationHostname+”:”+DestinationPort, DestinationIp+”:”+DestinationPort) | eval src_dest=src + ” => ” + dest | stats values(src_dest) as Connection by Image ComputerName
  10. The next search we will do is based around which countries the destination IP address is located in. This will search all initiated connections from your endpoints and plot them on a map of Earth.
    1. Search: EventCode=3 Protocol=tcp | iplocation DestinationIp | stats count by Country | geom geo_countries featureIdField=Country
    2. I didn’t have many connections to countries outside of the United States, so I modified it to show the states.
    3. Search: EventCode=3 Protocol=tcp | iplocation DestinationIp | stats count by Region | geom geo_us_states featureIdField=Region

Final Thoughts:

There have been numerous times throughout my career where I wish I would have just a tenth of the amount of this data to help resolve and investigate various issues. Hopefully, this gives you a starting point to manipulate the data you collect. Future searches that I find or create will likely just be small blog posts with descriptions and uses.

You can add a local computer if it is not joined to a domain by installing the Splunk Universal Forwarder for Windows and editing the inputs.conf file with the following entry. This is helpful in lab situations or if come across computers that are not in the domain but you still want to monitor.


Location: C:\Program Files\SplunkUniversalForwarder\etc\system\local

Add to Inputs.conf:
disabled = 0