Thursday, August 18, 2022

Remotely managing Sysmon configuration through Graylog Sidecar


Sysmon is a tool from Microsoft that can help with collecting better logs (compared to default Windows logs) regarding the system. The logs can be very helpful for detection of malicious behavior. Sysmon get's installed as a service and a driver usually along with whatever configuration file you provided. 

It can be kinda annoying to update sysmon configuration as sysmon doesn't come with a remote management system. If you have group of PC's that need different configuration, it can be a bit annoying to go push that out. 

There are some options for updating sysmon configs. 

  • You could setup a scheduled task that runs a script to look for a new config and does update. 
  • You can also use EDR tools that you may already have in place to run scripts or commands to update sysmon. 
  • You can utilize one of the Windows remote management features such as remote powershell. 
  • Ansible would work too. It would let you update sysmon config based on groups as Ansible lets you put machines into groups.
Some of the issues you may have with update methods above is 
  • problem with managing different group of PC's 
  • updates not being pushed out as soon as possible 
  • having to open up ports/services on Windows that you'd rather not.

I decided to go with Graylog sidecar for managing configuration. Graylog sidecar is usually used to manage configuration for log shippers (Beats for example) but it can be adapted to manage sysmon configuration as well!! Sidecar runs as an agent that will connect to Graylog server to get sysmon config updates. 


Graylog & sysmon, obviously.

You need graylog server installed. You can technically use graylog sidecar without using graylog to store your windows or sysmon logs.

I'm using an admin account for the POC but use whatever the appropriate account is for your requirements.

Setting up sidecar:

We'll manually set this up first but you can deploy sidecar agent and sysmon, and install both at the same time by creating a package or an initial installation script.

First, we'll need to create a sidecar token.

Go to the sidecars page, create a new token.

Next, on Windows host, download and install sidecar agent

By default, Sidecar files are located at: C:\Program Files\Graylog\sidecar

After installation, run the following commands (as admin) to install graylog as a service and start the service:

"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service install

"C:\Program Files\graylog\sidecar\graylog-sidecar.exe" -service start

(you can do all of this automatically if you build a package for your organization)

The host should show up on Sidecar overview page

Setting up sysmon support on Windows host:

Now that sidecar connection works fine, we need to setup sysmon. I'm placing sysmon executable at C:\Program Files\Graylog\sidecar\sysmon64.exe (make sure its lower case or at least the case matches the sidecar.yml config file)

Next, we need to edit sidecar.yml file to allow the use of sysmon64.exe (again, you can build a package and include sidecar.yml file that already supports this)

Here's what I have in the config file:

Restart the graylog sidecar service to ensure that it starts up again and sidecar.yml config file doesnt have any errors:

Next, we'll need to install sysmon w/ initial configuration (do this during initial graylog sidecar agent installation)

Now we can setup sysmon configuration in sidecar to do updates.

Setting up sysmon sidecar configuration in Graylog:

Go to collection configuration page in Graylog and click Create Log Collector button.

Create something that looks like this:

We're using foreground execution since we just need to execute the command to update sysmon config and exit.

The update command is sysmon -c CONFIG_FILENAME so we're using -c "%s", %s is the filepath of our config, when it's written to the disk.

Default template can be used to use a default config but I'm leaving it empty here.

Next on Collection configuration page, click Create Configuration

Name your configuration and add the xml config content.

You can create and add more configurations for different systems you may have.

Pushing configuration to a host:

Go to the sidecar Administration page

Check Sysmon and select the right sysmon configuration and apply the configuration.

Graylog webui may say the update failed but you can click the host you updated and click Show details to see more:

Above you can see that the configuration updated without an issue. 

You can also confirm the update by looking at event id 16 from sysmon in your SIEM or Event Viewer like below

Building initial installation package recommendations:

Your initial sidecar agent package should do the following:

  • Drop sysmon executable in the sidecar folder
  • Sidecar.yml file needs to contain path for sysmon executable in the allowed files
  • Install sysmon with whatever initial configuration you'd like to use
If you build an installation package yourself, you don't have to do all the stuff manually above...

Once you deploy sidecar agent + sysmon initially, you can remotely manage the sysmon configuration through Graylog sidecar UI.

Links/Resources: - I came across this after finishing this write up...