Collecting Custom Events

Preliminary actions

Golang does not support all POSIX features (golang regex synax).

To check correctness a developed regex, you can use an regex tools like regex 101 online tool (select golang in menu).

Examples of regular experssions for dhound you can find in rule files of predefined security events that are located by path: /etc/dhound-agent/rules.d/.

Dhound-agent uses rules files based in YAML format. Don't forget to validate yaml files before loading into dhound-agent. For example, you can use Yaml Online Converter for this.

Register on site

Login into your dhound security area

Go to Settings->Custom Events. Register custom security event (reserved sids: 100000-200000) and assign to particular custom security group (reserved gids: 100000-200000). This step will allow to minimize configuration in dhound-agent and use only sids and gids.

Custom rule file

Login on the server where dhound-agent installed

Navigate to the folder with dhound rules

Linux based systemcd /etc/dhound-agent/rules.d/

Windows based systemcd c:\Program Files\dhound.io\dhound-agent\config\rules.d

Make a copy of a custom security event rule file (use separate rule file for each log file types and make friendly file name for your custom security events)

Linux based systemsudo cp custom.yml mycustom.yml

Windows based systemcopy custom.yml mycustom.yml

Configure rules for collecting custom event. The template contains comment how to fill the fields:

# for more information about collecting custom event see here - https://knowledge.dhound.io/how-to-use-dhound/custom-security-events/configure-collecting-custom-events
# example of collecting custom events
#define list of files to parse, use asterisk to include files with dynamic file names
paths: 
- /var/log/app/applog*
- /var/log/app/*accesslog
- 'c:\inetpub\logs\LogFiles\FTPSVC*\*.log'
# exclude files that should not be parsed, dhound does not support parsing archives
excludefilesregex: ((.gz)|(.zip))$
# format of time in log files, milliseconds are not supported
eventtimeformat: YYYY MMM DD hh:mm:ss
#define the max age in hours of files to parse
deadtime: 360h 
# define list of events that can be extracted from source files
events:
  # (required) custom security id should be in rang 100000-200000
- sid: 100001 
  # (required) Posix regex with named variables, <eventime> and <ip> are required to be presented in event, 
  # all other named variables will be extracted from regex match and include into the event information
  regex: ^(?P<s1>.+?) (\S+?) (\S+?) \[(?P<eventTime>.+?)\] "POST /wp-login\.php (.*?)" 302 \d+ "((http(s|)://(?P<field1>.*?)/.*?)|-)"
  # (optional) override default message associated with this security event, parsed fields can be injected into a message  by using # prefix before field
  # message: #field1. access admin part of the 
  # (optional) override default security group that this security event will belong to
  # gid: 100000
  # (optional)additional static fields that should be included into events
  # if ip is not parsable from regex, it can be included as statis field, for example, ip: local
  # fields:
    # field1: <static value>
  # (optional) per each field can be added regex expression, if regex match is success, this event will be ignored and will not be sent on server
  # exclude:
    # ip: ((0\.)|(127\.)|(192\.168\.)|(10\.)|(172\.(1[6-9]|2[0-9]|3[0-1])\.)|(fc00:)|(fe80:))

Required fields:

• paths: list of log files to parse, use asterisk to include files with non-static names
• excludefilesregex: posix regex to exclude particular files or file types
• eventtimeformat: format of event time in parsing log files
  Formats:
  • M - month (1)
  • MM - month (01)
  • MMM - month (Jan)
  • MMMM - month (January)
  • D - day (2)
  • DD - day (02)
  • DDD - day (Mon)
  • DDDD - day (Monday)
  • YY - year (06)
  • YYYY - year (2006)
  • hh - hours (15)
  • mm - minutes (04)
  • ss - seconds (05)
  • AM/PM hours: 'h' followed by optional 'mm' and 'ss' followed by 'pm', e.g.
  • hpm - hours (03PM)
  • h:mmpm - hours:minutes (03:04PM)
  • h:mm:sspm - hours:minutes:seconds (03:04:05PM)
  • Time zones: a time format followed by 'ZZZZ', 'ZZZ' or 'ZZ', e.g.
  • hh:mm:ss ZZZZ (16:05:06 +0100)
  • hh:mm:ss ZZZ (16:05:06 CET)
  • hh:mm:ss ZZ (16:05:06 +01:00)
• sid: custom security id that you registered on the site
• regex: regular expression in posix format for extract security event data from log file

Check validness of yaml rule file in a yaml tool (for example, Yaml Online Converter)

See Also

• Troubleshooting dhound-agent on Linux (on this resource)
• Security Groups and Events (on this resource)
• List of predefined security groups and events (on this resource)
• Configure collecting custom security events (on this resource)