** this has been fixed starting from build 1.2.0-75**
When I first saw the post on Cisco ASA log collection for OMS Log Analytics I rushed to configure it to complement the security sources we’re collecting. Alas it didn’t work the device name was wrong, basically it was the time of the syslog entry.
After a while I’ve been able to return to the subject, fix it and learn a lot on how the oms agent for Linux works.
How to fix ASA log collection
If you’re interested in collecting ASA logs all what you need to know is summarized in this paragraph and you can skip any further reading.
First of all, assuming you want your device name/ip logged, you must properly configure the syslog service on the ASA, the command is logging device-id. In my case I just chose to log the hostname of the ASA device so I just typed: logging device-id hostname.
If you’re using the GUI ASDM you find the same settings under Configuration|Device Management|Logging|Syslog Setup|Advanced
Then you must configure the ASA device to forward the syslog entries to the Linux omsagent, I’m not going to rewrite the ASA configuration manual here, there are however a couple of setttings you must be aware of:
- first, do not use the EMBLEM format
- second, the official documentation makes you configure the syslog server as the syslog (r-syslog in my case) daemon on the Linux box and then from there makes it forward the events to the oms agent fluentd syslog filter (more on this later). You can by pass this double hop and sendthe logs directly to the oms agent. This is up to you, but a little mod needs to be applied to the configuration file provide by Microsoft.
To properly configure the ASA log collection just modify the security_events.conf file accordingly to the following:
<br />type syslog port 25225 bind 0.0.0.0 #bind 127.0.0.1 this binds to the loopback adapter and thus cannot be reached from the outside protocol_type udp tag oms.security format /^(?<time>[^ ]*\s*[^ ]* [^ ]* [^ ]*) (?[^ ]*) : (?[a-zA-Z0-9_%\/\.\-]*)(?:\[(?[0-9]+)\])?(?:[^\:]*\:)? *(?.*)$/ #format /^(?<time>[^ ]*\s*[^ ]* [^ ]*) (?[^ ]*) (?[a-zA-Z0-9_%\/\.\-]*)(?:\[(?[0-9]+)\])?(?:[^\:]*\:)? *(?.*)$/ </time></time> type filter_syslog_security
As you can see I commented out the original settings and replaced them with the ones that work in my environment.
Lastly remember to sudo service omsagent restart, because the filter configuration files such as security_events.conf are loaded just at agent start time.
After that my log are eventually the way they’re supposed to be
Lessons learned and resources
Foreword, probably many of you already know about this, but for me it has been insightful having always took the oms agent for Linux as a black box that I just had to install. This is a notes from the field paragraph.
To troubleshoot what was going on I had to dig inside the omsagent code on GitHub unfortunately the project doesn’t allow yet pull requests.
The omsagent afaik collects data form various sources:
- omi, that is the source I was aware of since it has been the second interface used by the SCOM Linux agent after OpenPegasus. It’s basically a winrm for Linux
- collectd, is used or can be used (afaik it is not on by default) to collect any performance metric from the Linux system and forward it to OMS (https://blogs.technet.microsoft.com/msoms/2016/07/14/bring-your-custom-collectd-metrics-into-the-oms-log-analytics-platform/)
- fluentd(https://github.com/fluent/fluentd), is used at the core of the agent collection capabilities, it collects, transforms and forwards I would say any log based on an extensible model of providers and filters. All the other sources are processed through the fluentd module before being forwarded to the cloud workspace.
The Cisco ASA collection is based on a fluentd filter (http://docs.fluentd.org/articles/filter-plugin-overview) that parses the syslog entries based on a specific match (the %ASA token). The key information was the format configuration tag, that is a regexp applied to the incoming entries. The first thing to check is to have a sample entry as it is passed to the module, I found a couple of ways:
- configure the rsyslog deamon to dump ASA events
- mess with the format regexp string so that it doesn’t match the incoming entry, at that point the entry is logged in /var/opt/Microsoft/omsagent/log/omsagent.log
One you get the entry string is just a matter of trying the correct regexp string, I use regex101.com
I would be interesting to check what happens if a new filter ruby file is added, obviously it must prepare the data based on a builtin Type (OMS Type), if it is dynamically loaded at gent startup it can be a great way to add data to native Types without using the REST ingestion interface and thus creating a new custom Type.