One of our avid twitter followers asked how to reliably install the Splunk Universal Forwarder on a Windows host with PowerShell last week. I’ve posted about all the intricacies involved before but improvements in open-source tools for PowerShell have made it a whole lot easier. You can take a look at the original article, but follow along here instead. We’re going to walk through what’s involved.

Installing as a Local SYSTEM user is easy. Here is the recipe:

Invoke-Command –ComputerName S1,S2,S3 –ScriptBlock { `
New-PSDrive S –Root \\SPLUNK\Files -PSProvider FileSystem; `
Start-Process S:\splunkforwarder-6.1.1-207789-x64-release.msi `
    –Wait -Verbose –ArgumentList (`
        “AGREETOLICENSE=`”Yes`””, `
        “DEPLOYMENT_SERVER=`”SPLUNKDEPLOY:8089`”” `
        “/Liwem!”, “C:\splunkinstall.log” ) `
}

Let’s recap what you need to do to install a Splunk Universal Forwarder on a Windows host as a domain user:

1. Add a new service account for the domain user
2. Prepare the host to run Splunk
3. Install the Splunk Universal Forwarder with the MSI

Step 1 can be a one-time activity if you run all your Splunk Universal Forwarders as the same user, so let’s do that first. In PowerShell 3, there are Active Directory domain utilities. I like to place my service accounts in an organizational unit called “OU=Service Accounts” off the top-level domain structure. You can use the following command:

New-ADUser –Name svc_splunk –SamAccountName svc_splunk `
    -Description “Service:Splunk UF” `
    –DisplayName “Service:Splunk UF” `
    -Path “OU=Service Accounts,DC=splk,DC=com” `
    -AccountPassword (Read-Host –AsSecureString “Account Password”) `
    -CannotChangePassword:$true `
    –ChangePasswordAtLogon:$false `
    -PasswordNeverExpires:$true `
    –PasswordNotRequired:$false `
    -SmartcardLogonRequired:$false `
    –Enabled:$true

This is fairly basic stuff for the modern domain admin – the important thing to note is you are prompted for a password – you will need it later. Aside from that, you may need to open up some firewall holes if you have the Windows Firewall enabled – port 8089 and 9997 (or whatever your receiving port is) on the outbound side. If I need to do it, I do it everywhere via a group policy.

Now comes the complicated part. Preparing the host to run Splunk means giving the svc_splunk user a lot of privileges. Specifically, you need to add the user to the Administrators group and give the user specific OS-level rights. This used to be really complicated, but we are going to simplify it by utilizing a set of open-source utilities called Carbon (you can download Carbon at http://get-carbon.org). Download the Carbon package and install them as directed on the website. I placed mine in my WindowsPowerShell\Modules directory so that they are always available. They are that useful.

Let’s take a look at adjusting those permissions the new way. First up is adding my service account to the Administrators group. This was already fairly easy, but with Carbon it’s even easier:

Add-GroupMember –Name “Administrators” –Member DOMAIN\svc_splunk

Adjusting the local security policy used to keep me up at night. It was fraught with peril. With Carbon, this is now easy:

Grant-Privilege –Identity DOMAIN\svc_splunk –Privilege `
    ( SeTcbPrivilege, SeChangeNotifyPrivilege, SeBatchLogonRight, `
    SeServiceLogonRight, SeAssignPrimaryTokenPrivilege )

The only gotcha here is that the privileges are case-sensitive, so be careful. Once we have done this, we have completed the host preparation. Now all we need to do is to install the binaries and install the service. I do this via the MSI installer with:

New-PSDrive –Name S –Root \\SPLUNK\Files -PSProvider FileSystem
Start-Process S:\splunkforwarder-6.1.1-207789-x64-release.msi `
    –Wait -Verbose –ArgumentList (`
        “AGREETOLICENSE=`”Yes`””, `
        “LOGON_USERNAME=`”DOMAIN\svc_splunk`””, `
        “LOGON_PASSWORD=`”MyPasswordHere`””, `
        “DEPLOYMENT_SERVER=`”SPLUNKDEPLOY:8089`”” `
        “/Liwem!”, “C:\splunkinstall.log” )

Note that I put the complete fully-qualified path to the MSI here – it’s important. The msiexec seems to break without it. I also set up a CNAME in DNS for the deployment server – it allows me to point it anywhere I want. This is the same command I install as a local system user, but with the additional parameters to specify the domain user.
Script it? Why, certainly. I just put the commands (including an “Import-Module Carbon”) into a ps1 script and put it on \\SPLUNK\Files. Now I can do this:

Invoke-Command –ComputerName S1,S2,S3 –ScriptBlock { `
    New-PSDrive –Name S –Root \\SPLUNK\Files -PSProvider FileSystem; `
    S:\InstallUniversalForwarder.ps1 }

There is a future here. Desired State Config is a newer feature available in PowerShell v4. We can check permissions, file locations, and service settings and both install and upgrade within a single entity. That, however, is a topic for another blog post.

It is always gratifying when one of my readers comes to me with a problem. I love challenges. This one had to do with one of my old posts surrounding Local Administrators remotely. Of course, the way to do this is via WMI. However, it doesn’t quite work the same way locally. This is because the WMI call to Win32_Group.GetRelated() returns other stuff as well. So the question posed was “how do I get the list of Local Administrators locally.” More specifically, I want to monitor the local Administrators group.

I look at this two ways. Firstly, I want to get a regular list of names in the Administrators group and secondly, I want to monitor for changes to the Administrators group during the day. Let’s start with the first one. My favorite tool for this is, of course, PowerShell. After a bit of back and forth, I came up with a script that works on both Windows Server 2012 and Windows Server 2008R2 – there is a little bit of syntax change in the Where-Object between PowerShell 2 (used on Windows Server 2008R2) and PowerShell 3 (used on Windows Server 2012 and above).

(Get-WMIObject Win32_Group | Where-Object { $_.Name –eq ‘Administrators’ }).GetRelated() | Where-Object { $_.__CLASS –eq “Win32_UserAccount” –or $_.__CLASS –eq “Win32_Group” } | Select-Object __CLASS,Caption,SID

The script-block form of Where-Object has always been available and hence is suitable here. My original script has just Where-Object Name -eq ‘Administrators’ which doesn’t work in PowerShell 2. After we have the GetRelated() information, we can search for the information we are really after – the users (which has a __CLASS of Win32_UserAccount) and the groups (which has a __CLASS of Win32_Group). Finally, we select the information we are interested in. As you are probably aware if you are a constant reader of the blog, we use SA-ModularInput-PowerShell to run our PowerShell for us. This module requires us to Select-Object the fields prior to finishing. We can now put this script into our inputs.conf file:

[powershell://LocalAdmins]
script = (Get-WMIObject Win32_Group | Where-Object { $_.Name –eq ‘Administrators’ }).GetRelated() | Where-Object { $_.__CLASS –eq “Win32_UserAccount” –or $_.__CLASS –eq “Win32_Group” } | Select-Object __CLASS,Caption,SID
schedule = 0 30 2 ? * *
sourcetype = PowerShell:LocalAdmins
source = PowerShell
disabled = false

[powershell2://LocalAdmins]
script = (Get-WMIObject Win32_Group | Where-Object { $_.Name –eq ‘Administrators’ }).GetRelated() | Where-Object { $_.__CLASS –eq “Win32_UserAccount” –or $_.__CLASS –eq “Win32_Group” } | Select-Object __CLASS,Caption,SID
schedule = 0 30 2 ? * *
sourcetype = PowerShell:LocalAdmins
source = PowerShell
disabled = false

Notice that I have two versions – one is PowerShell 2 and the other is PowerShell 3. On Windows Server 2012 and above, only PowerShell 3 is available – you have to install PowerShell 2 separately. Thus the second one doesn’t get run. On Windows Server 2008 R2 the reverse is true – you don’t get PowerShell 3 – only PowerShell 2 is installed by default. If you have standardized on PowerShell 3 already (and you really should have done so by now), only enter the first stanza. There isn’t a Common Information Model for group membership inventory, but I tend to assume there will be in the future. In keeping with the CIM, let’s define our field extractions properly and ensure there are tags. This would fall under the Inventory Data Model and be a “Group” type. Our membership can be either a Group or a UserAccount and will have a domain and a username and a security ID. Start with props.conf:

[PowerShell:LocalAdmins]
REPORT-cim = localadmins-type, localadmins-userdom, localadmins-sid

Then define the extractions in transforms.conf:

[localadmins-type]
REGEX=(?ms)__CLASS=Win32_(?<member_type>.*?)\n

[localadmins-userdmon]
REGEX=(?ms)Caption=(?<member_domain>[^\\]+)\\(?<member_name>.*?)\n

[localadmins-sid]
REGEX=(?ms)SID=(?<member_sid>.*?)\n

Finally, our tags.conf looks like this:

[sourcetype=PowerShell:LocalAdmins]
inventory = enabled
group = enabled

Now that we have our list of local administrators, how do we get the changes? The changes are recorded in the Security Windows Event Log. This is kept in the Splunk_TA_windows and I highly recommend that this is distributed with common inputs enabled – one of which is the WinEventLog://Security input. Don’t want everything? You can limit the event codes you need with a whitelist and blacklist. The relevant event codes are:

• NT5 (Windows Server 2003 and before): 635-639
• NT6 (Windows Server 2008 and beyond): 4731-4735

The Splunk App for Windows Infrastructure contains specific field extractions for these events, allowing you to easily incorporate them into your own dashboards. Now you can truly monitor the local Administrators group on all your servers.

If you are a Windows admin and use Splunk then you’ve likely deployed Splunk_TA_windows on your endpoints. It’s a central method for handling Windows data and has all the extractions you need to handle Windows event logs. We’ve just released version 4.7.0. So what’s new and should you upgrade?

The first thing we did was we organized the data. The well considered best practice is to not put data in the default index. Yet here we were putting data in the default index. That has now changed. By default, we create three indices for you:

• perfmon is used for performance data
• wineventlog is used for event logs
• windows is used for everything else

This change will not affect you if you’ve been using a local inputs.conf file (as you should). However, new installations beware – this change requires that the indices be available on your indexing tier or you will get messages about indices missing.

The second thing we did was we turned off all the inputs. That’s right – Splunk_TA_windows no longer gathers data by default. There are many reasons for this, but the primary reason is that we license by data volume so we didn’t want you to be blowing out your license because we turned something on by default. Make the conscious decision on what to gather based on what you want to show off.

The third thing we did was make the data more CIM compliant. CIM compliance has always been a part of the Splunk_TA_windows for use with Enterprise Security. If you’ve looked at the Common Information Model recently you will have noticed that the CIM now encompasses more than just security – it adds many data models for IT/Operations usage. We’ve embraced this and as a result, the Windows data now will automagically appear in Enterprise Security and the data models that are implemented in the Common Information Model app. You can now use the Windows data in standardized data models like this:

Finally, we incorporated a lot of customer feedback. We can’t test on every Windows variant out there (although we do try to get as many common variants as we can), so sometimes we miss something and you tell us about it.

There is one thing to take note of going forward. Microsoft is marking Windows Server 2003, Vista and before as end of life on July 14th, 2015. As a result, we will also be de-supporting the field extractions for 3-digit windows security event logs, and we are deprecating them with this release. What does this mean? We will be separating the 3-digit field extractions into another app – we’ll call it TA-legacywindows or something like that. If you install the TA-legacywindows then everything happens as before – the extractions are just in a different app. We will remove the field extractions affected from Splunk_TA_windows some time after the end-of-life notice and after TA-legacywindows has been released. We will also bump that second digit again so you know something major has happened.

Needless to say, future versions of the Windows Infrastructure app, Exchange app and other apps for Microsoft technologies will rely on the updated TA.

Got requests, comments, or bugs with the Splunk_TA_windows – drop us an email at microsoft@splunk.com.

On Tuesday, I introduced one of the first presentations at .conf2014 – a major update to the SA-ldapsearch app. This new app has now launched and you can download it at http://apps.splunk.com/app/1151/. The app consists of four specific commands: ldapsearch, ldapfetch, ldapfilter and ldapgroup.

Improvements include:

• We dropped the requirement for Java on your search head
• We added support for Search Head Pooling
• We added a GUI configuration page and connection testing
• We provided full UTF-8 support

The ldapsearch command is a generating command and is used in a similar way to other generating commands like inputlookup. You run it like this:

| ldapsearch domain=SPL search="(objectClass=user)" attrs="sAMAccountName,cn"

We have added some new features in this release. Firstly, the output is in JSON, so it’s a better format when you are looking at the raw events that come back. Secondly, you can search a subset of the data using the basedn and scope parameters. Let’s say all your users are in the ou=People container and it’s flat. You can optimize the above search like this:

| ldapsearch domain=SPL basedn="ou=People,dc=spl,dc=com" scope="sub" search="(objectClass=user)"

Of course, we kept the decoding of the attributes like the objectSID and other features you know and love.

If you want to augment the events you already have then you can use ldapfetch and ldapfilter. ldapfetch takes a distinguished name and fetches the object from Active Directory. ldapfilter uses fields to do a search. For example, let’s say you have an eventtype that has fields src_user and nt_domain (standard fields from the Common Information Model) and you want to add the persons real name and telephone number:

eventtype=my-event 
| stats count by src_user,nt_domain
| ldapfilter domain=$nt_domain$ search="(sAMAccountName=$src_user$)" attrs="cn,telephoneNumber"
| table src_user, nt_domain, cn, telephoneNumber

If that eventtype has a field called dn in it and the field contains the distinguished name of the object, we can do the same thing with ldapfetch:

eventtype=my-event 
| stats count by dn
| ldapfetch dn=dn attrs="sAMAccountName,cn,telephoneNumber"
| table sAMAccountName, cn, telephoneNumber

The final one is ldapgroup. This takes a distinguished name of a group and expands that group to the membership, taking into account nested groups and cyclical groups. It adds five multi-value fields to the event to provide information about the membership, including if they are listed in the group directly or indirectly via another group. You use it like this:

| ldapsearch domain=SPL search="(objectClass=group)" attrs="cn,distinguishedName"
| ldapgroup
| table cn,member_name,member_type

Hopefully, this new upgrade to our popular integration with Active Directory will help you with your reports. If you like this app, then perhaps you should also check out the Splunk App for Windows Infrastructure and monitor your Active Directory environment as well.

We’ve been (rightly) criticized for a couple of things in recent years. Firstly, when you configure a Windows Event Log, it’s too big. This is because we combine the event log object with the message from the locale-specific DLL and that includes a bunch of common explanatory text. I don’t really need to know what a login really means (to the tune of 1K of data ingest) every time someone logs in, especially when these events are happening hundreds of times a minute. Secondly, our event log extractions are for US/English only. Got German Windows? Sorry – our extractions don’t work for that. Finally, we discard the additional data that is provided in the event log object. A primary example of this is the Logon Performance Report, which Jason Conger has written about. In addition, we occasionally lost contact with the locale-specific message DLL and that caused us to not write the event at all, losing even more data.

The locale-specific message DLL is the culprit here. It changes depending on the locale of Windows you have installed, causes the message bloat and throws away data that we are interested in. What if we didn’t use the locale-specific message DLL? What if we stored the event logs in XML?

With Splunk 6.2, that’s exactly what you can do. The effort does not come for free, however. We’ve already done the hard work of updating the Splunk_TA_Windows to include CIM compliant extractions for the Security Event Log. If you have done your own extractions (or you are using an alternate TA that contains extractions), then these need to be updated to support the XML format. However, converting to the XML format solves all the problems I mentioned.

So, how does it work? First of all, you need to update your inputs.conf to render the XML. We’ve designed it so that nothing changes on upgrade (backwards compatibility FTW!) You need to add a line to your WinEventLog definitions like this:

[WinEventLog://Security]
index=security
current_only=1
evt_resolve_ad_obj=0
renderXML=1
disabled=0

Once you have restarted your universal forwarder, you will see XML coming in with a different source type (XmlWinEventLog:Security, for example).

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>
 <System>
  <Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54849625-5478-4994-A5BA-3E3B0328C30D}'/>
  <EventID>4672</EventID>
  <Version>0</Version>
  <Level>0</Level>
  <Task>12548</Task>
  <Opcode>0</Opcode>
  <Keywords>0x8020000000000000</Keywords>
  <TimeCreated SystemTime='2014-04-29T22:15:03.280843700Z'/>
  <EventRecordID>2756</EventRecordID>
  <Correlation/>
  <Execution ProcessID='540' ThreadID='372'/>
  <Channel>Security</Channel>
  <Computer>sacreblue</Computer>
  <Security/>
 </System>
 <EventData>
  <Data Name='SubjectUserSid'>AUTORITE NT\Système</Data>
  <Data Name='SubjectUserName'>Système</Data>
  <Data Name='SubjectDomainName'>AUTORITE NT</Data>
  <Data Name='SubjectLogonId'>0x3e7</Data>
  <Data Name='PrivilegeList'>SeAssignPrimaryTokenPrivilege
                             SeTcbPrivilege
                             SeSecurityPrivilege
                             SeTakeOwnershipPrivilege
                             SeLoadDriverPrivilege
                             SeBackupPrivilege
                             SeRestorePrivilege
                             SeDebugPrivilege
                             SeAuditPrivilege
                             SeSystemEnvironmentPrivilege
                             SeImpersonatePrivilege</Data>
 </EventData>
</Event>

Note that the keys are always rendered in English, irrespective of the system locale of the machine that is generating the events. This event was generated from a machine running with French as the system locale – note the field names.

The effect of this on your data ingest rate is rather dramatic. Firstly, the size of the Windows Security Event Log is reduced by almost 70%. That means that a typical Active Directory Monitoring solution will occupy just over 1Gb of data per 1,000 users (instead of the 4Gb of data that it would take up in the old format). In addition, you don’t have to write custom code to get at those extra fields – the Logon timing report discussed above is now possible with just a simple dashboard.

For those of you running Enterprise Security, you will be happy to hear that, as of Splunk_TA_Windows 4.7.0, the XmlWinEventLog format is recognized within the TA and all the same field extractions are performed on the new format. That means you can take advantage of the data ingest savings straight away.

You might be wondering if there is any reason to not use the XmlWinEventLog format as your default now? Unfortunately, yes, there is. When we wrote the Windows Infrastructure and Exchange apps, we used the old format. The field extractions for these apps will be updated in the next release. Some dashboards will not render correctly if you change your data format right now.

Last week, I covered the XML Event Logs – an awesome feature that will reduce your data ingest, increase the fidelity of the data that is stored and allow us to work with localized data. Today, I want to discuss another localization feature – or at least a delocalization feature – perfmon.

Prior to Splunk 6.2, Windows perfmon was always collected localized. If you wanted the % Processor Time counter, you had to specify the localized version of this. If you were running on a french version of Windows, you would have to specify object=Processeur and counter=”% Temps Processeur” in both your inputs.conf and searches. Given that there are over 30 different localized versions of Windows, this really meant that apps only worked in US/English unless you took extraordinary measures to adjust them for the different locales. I even wrote a blog post about how to alleviate this condition.

In Splunk 6.2 we introduced a flag to the perfmon stanzas in inputs.conf called useEnglishOnly. Instead of specifying the counters and objects in your installed language, you can specify them in US/English. For instance:

[perfmon://CPU]
counters = % Processor Time; % User Time; % Privileged Time; Interrupts/sec
disabled = 0
instances = *
interval = 10
object = Processor
useEnglishOnly = true
index = perfmon

This will record the counter and object names as US/English in the index as well. The end result is that you can – with a simple change to the inputs.conf stanza – record, store and search for everything in the US/English version, effectively ensuring that your app works with all locales that Microsoft produces. We’ve already made this change to the disabled perfmon inputs on Splunk_TA_Windows v4.7.3.

There is one caveat to this. The underlying API that we use (PdhAddEnglishCounter, for the curious) does not allow us to add perfmon counters with a wildcard. This means that the counters parameter must explicitly list each counter you are interested in indexing.

Our hope, of course, is that you will write and distribute excellent apps that are locale agnostic. The pairing of XML Event Logs and English-Only Perfmon should assist in accomplishing this. We are certainly working on releasing the premium and Splunk supported apps with this in mind.

For about the last year, I’ve been working on an update to the Splunk App for SharePoint. But it isn’t the one you would expect. I’ve been working to open source the app. At the end of the day the best person to write an IT Operations app for Splunk is the person who is intimately involved in the running of the workload. Today, we are flicking the switch and opening up the project. We are allowing you to directly file bugs and feature requests; we are allowing you to submit code; and we are encouraging you to get involved in the project.

So, how can you do this. Firstly, you will want to have some sort of test environment. Most SharePoint admins I know have a small test SharePoint environment that gets used for new functionality. That is an ideal environment for this. You will want to download the Add-on for Microsoft SharePoint (it supports SharePoint 2010 and SharePoint 2013) and get that implemented so that you have data flowing in. Don’t forget the Add-on for Microsoft Windows and other add-ons that may be appropriate to your environment (like SQL Server).

Now that you have data flowing in, you can download the Splunk App for SharePoint. We will be doing regular releases of the app and uploading them to Splunkbase. In fact, version 0.2.0 (that matches the TA-Microsoft_SharePoint v0.2.0 that was just released) will be uploaded today. However, that isn’t the bleeding edge. To get the bleeding edge, you will need to go to the source – in our case, that is https://github.com/splunk/splunk-app-sharepoint.

As is normal with open source projects, you will need to do a little more work in order to get this going. If you are just following the project, then you can download it as a ZIP file, unpack it and then build it. You will need a build system that has Java and ant installed. To build the project, just change into the same directory as the build.xml and run

ant dist

This will create the app distribution in the build/dist directory.

You can use our Issue Tracker if you want to file bugs or feature requests. We’ve also set up a Google Groups mailing list for discussion of the project.

What about making changes? Your first step is to fork the repository. Once you have forked the repository you can use the standard git tools to sync the repository to your local development machine and make changes.

What about contributing? First of all, we need to get the basics on contributing to a Splunk app out of the way. You can check out our Open Source page for information on contributing. Once that is out of the way, make changes, then issue a Pull Request back to us. We will review your changes and may contact you about them (it’s a good idea to discuss your merge request on the mailing list). Then, assuming all is in order, we will merge your changes into the master branch and they will be incorporated into the next release.

If you manage SharePoint and you have ever dreamed of what it would be like to have the perfect tool – now is your chance. Get involved and let’s make something awesome.

Every IT guy has a set of tools that they use every day. One of mine is sysinternals. It’s a set of Windows utilities made available by Microsoft that do a whole slew of things. You can install them with chocolatey (another in my toolset) or downloaded and unpacked from their website. If you use Windows and this toolset isn’t in your arsenal, maybe it’s time.

Back in August, I got a request from one of our engineers asking me if we had any plans to support the collection of Sysmon data. Sysmon is a Windows system service (yes, another agent) that logs system activity to the Windows Event Log. However, it places all the important stuff in the XML data block – that bit of the Windows Event Log that we did not expose until 6.2.0. Now that we have the renderXml parameter on WinEventLog, we can do something about it. I was reminded of this utility last week when one of our security researchers asked about network connections.

Installing it is relatively simple. Download the package, unzip it, then run:

sysmon -i -n -accepteula

The -i installs the service and the -n instructs it to monitor network connections. Once that is done, the service will start dropping events in the stream Application and Services Logs/Microsoft/Windows/Sysmon/Operational. You can receive these logs in Splunk by using the following inputs.conf entry:

[WinEventLog://Microsoft-Windows-Sysmon/Operational]
disabled = false
renderXml = true

Now that you have events in Splunk, there is a wealth of information available to you. The basic search is:

sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational"

The normal EventCode field is available and there are two basic event codes available to you. There are several event codes generated, but the two you will want to be concerned with are EventCode=”1″, which is a process creation and EventCode=”3″, which is a network connection.

Which brings us back to our problem. Last week, one of our security researchers was asking about monitoring network connections. Most specifically, he wanted outbound connections, the process ID of the process that created the outbound connection and the parent process ID of that process. As he explained it, if you see a connection to a known bad IP or domain and you suspect malware, then the process ID of the process opening the connection is the malware and the parent process is likely the malware dropper. We have all this information, so let’s take a look.

First of all, we have our outbound connection. The events look like this:

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Sysmon' Guid='{5770385F-C22A-43E0-BF4C-06F5698FFBD9}'/><EventID>3</EventID><Version>3</Version><Level>4</Level><Task>1</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime='2014-11-24T18:54:04.495825000Z'/><EventRecordID>61155</EventRecordID><Correlation/><Execution ProcessID='3360' ThreadID='2796'/><Channel>Microsoft-Windows-Sysmon/Operational</Channel><Computer>SPLUNK.ahall.local</Computer><Security UserID='S-1-5-18'/></System><EventData><Data Name='UtcTime'>11/24/2014 6:54 PM</Data><Data Name='ProcessGuid'>{00000451-4DA3-5459-0000-0010E8F30000}</Data><Data Name='ProcessId'>884</Data><Data Name='Image'>C:\Windows\system32\svchost.exe</Data><Data Name='User'>NT AUTHORITY\NETWORK SERVICE</Data><Data Name='Protocol'>udp</Data><Data Name='Initiated'>false</Data><Data Name='SourceIsIpv6'>true</Data><Data Name='SourceIp'>ff02:0:0:0:0:0:1:3</Data><Data Name='SourceHostname'></Data><Data Name='SourcePort'>5355</Data><Data Name='SourcePortName'>llmnr</Data><Data Name='DestinationIsIpv6'>true</Data><Data Name='DestinationIp'>fe80:0:0:0:354c:3220:8159:7591</Data><Data Name='DestinationHostname'>SPAPP10</Data><Data Name='DestinationPort'>50739</Data><Data Name='DestinationPortName'></Data></EventData></Event>

In order to get this properly extracted, we need to do some work with props and transforms. Fortunately, Splunk provides a KV_MODE of xml that extracts some of the data. However the Data elements need to be extracted separately and some of the automated extractions didn’t work, so I rolled my own. Here is my props.conf:

[XmlWinEventLog:Microsoft-Windows-Sysmon/Operational]
REPORT-sysmon = sysmon-eventid,sysmon-version,sysmon-level,sysmon-task,sysmon-opcode,sysmon-keywords,sysmon-created,sysmon-record,sysmon-correlation,sysmon-channel,sysmon-computer,sysmon-sid,sysmon-data

And here is my transforms.conf:

[sysmon-eventid]
REGEX = <EventID>(\d+)</EventID>
FORMAT = EventCode::$1

[sysmon-version]
REGEX = <Version>(\d+)</Version>
FORMAT = Version::$1

[sysmon-level]
REGEX = <Level>(\d+)</Level>
FORMAT = Level::$1

[sysmon-task]
REGEX = <Task>(\d+)</Task>
FORMAT = Task::$1

[sysmon-opcode]
REGEX = <Opcode>(\d+)</Opcode>
FORMAT = Opcode::$1

[sysmon-keywords]
REGEX = <Keywords>(0x[0-9a-fA-F]+)</Keywords>
FORMAT = Keywords::$1

[sysmon-created]
REGEX = <TimeCreated SystemTime='(.*?)'/>
FORMAT = TimeCreated::$1

[sysmon-record]
REGEX = <EventRecordID>(\d+)</EventRecordID>
FORMAT = RecordID::$1

[sysmon-correlation]
REGEX = <Correlation>(.*?)</Correlation>
FORMAT = Correlation::$1

[sysmon-channel]
REGEX = <Channel>(.*?)</Channel>
FORMAT = EventChannel::$1

[sysmon-computer]
REGEX = <Computer>(.*?)</Computer>
FORMAT = Computer::$1

[sysmon-sid]
REGEX = <Security UserID='(S-[0-9a-fA-f-]+)'/>
FORMAT = SecurityID::$1

[sysmon-data]
REGEX = <Data Name='(.*?)'>(.*?)</Data>
FORMAT = $1::$2

With full extraction, we can now do some interesting searches. Here is a search that shows all the outbound connections that a particular process uses (together with the user and computer that is running it):

sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode=3 Protocol=tcp Initiated=true | eval src=if(isnotnull(SourceHostname), SourceHostname+":"+SourcePort, SourceIp+":"+SourcePort)  | eval dest=if(isnotnull(DestinationHostname), DestinationHostname+":"+DestinationPort, DestinationIp+":"+DestinationPort)  | eval src_dest=src + " => " + dest | stats values(src_dest) as Connection by ProcessGuid ProcessId User Computer Image

Similarly, you can lookup processes as they are created:

sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode=1 NOT User="NT AUTHORITY\\SYSTEM" | stats values(User) as User,values(CommandLine) as CommandLine,values(ProcessId) as ProcessId,values(ParentProcessId) as ParentProcessId values(ParentCommandLine) as ParentCommandLine by LogonGuid

Note that we are exempting NT AUTHORITY\SYSTEM (and don’t forget the double-backslash) because we literally get thousands of them.

How about getting the command line that was run for each outbound non-local network connection?

sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational" EventCode=3 Protocol=tcp Initiated=true | where DestinationIp!="127.0.0.1" AND DestinationHostname!=SourceHostname | table _time User Computer ProcessId ProcessGuid DestinationHostname DestinationPort | join type=inner [search sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational" EventCode=1 | table _time ProcessGuid ProcessId CommandLine]

I’m currently working on releasing a CIM-compliant TA-microsoft_sysmon to handle these use cases. You can download the latest version from my GitHub repository, or watch for it arriving on apps.splunk.com soon.