vRLI & Logstash: Splitting ESXi logs.

With this next post I would like to share my first experiment with Logstash and Docker, while nothing here is rocket science they were both relatively new to me and so I thought to write a little post about it.

The the intent / requirement here was to split the logs between vRLI and another logging system, the goal was to remove load from vRLI as the cluster could not longer be expanded horizontally or vertically due to physical host constraints. Unfortunately after tuning / tweaking vRLI the cluster was still suffering heavily.

By sticking Logstash in the middle we could split the logs based on filters, allowing us to store all important ESXi logs we need for the platform within vRLI and the network guys could still have the DFW in another dedicated logging solution.

Let’s get started, here is my repo with the Dockerfile / config files for Logstash, make sure to update logstash.conf to match your desired destinations.

input {
  tcp {
    port => 5044
    type => syslog
  }
  udp {
    port => 5044
    type => syslog
  }
}

filter {
    if [message] =~ "dfwpktlogs" or [message] =~ "FIREWALL_PKTLOG" {
        mutate {
            add_field => { "LOGKIND" => "DFWJUNK" }
        }
    }
}

output
{
  if [LOGKIND] == "DFWJUNK" {
    tcp {
        host => "192.168.16.123"
        port => "514"
        codec => line {format => "%{message}"}
        }
  }
  else
  {
    tcp {
        host => "192.168.16.122"
        port => "514"
        codec => line {format => "%{message}"}
        }
  }
  stdout { codec => rubydebug }
}

To build the container with my config just run the command below and once the container is built run it,

docker build -t esxi2logstash2vrli c:\PathToFolder

Now configure the ESXi host to forward to the Logstash instance / port as desired, when it starts ingesting logs you should see something like this in the console.

As you can see it is picking up DFW related logs and tagging it as “DFWJUNK“, this will split it to a separate syslog destination.

So let me show you a little example from vCenter just to compare logs….

In the VCSA VAMI I configured 2 syslog destinations, one to vRLI directly and the other the logstash container… which would then filter and forward to vRLI again…

If we were to use SYSLOG instead of TCP or UDP as the OUTPUT in Logstash we would end up with a double syslog header which we don’t want. By using just TCP or UDP, we only see a small difference on the timestamp when vRLI ingested the log but the log is still identical and all the expected vRLI fields still function.

Just note that the “SOURCE” is now logstash, so you will need to depend on the “hostname” field instead for the original source.

Hope this was helpful.

vMan