Collecting Custom Events

Dhound allows to collect custom security events on your servers. Below is the guidence how to collect custom security events.

Preliminary actions

Collecting custom security events on Linux is based on parsing log files by regular expressions. You should be familiar with this language and check created expressions in a correspondend tool that can check Regular Expressions in Posix format.

Golang does not support all POSIX features (golang regex synax).

To check correctness a developed regex, you can use an regex tools like regex 101 online tool (select golang in menu).

Examples of regular experssions for dhound you can find in rule files of predefined security events that are located by path: /etc/dhound-agent/rules.d/.

Dhound-agent uses rules files based in YAML format. Don't forget to validate yaml files before loading into dhound-agent. For example, you can use Yaml Online Converter for this.

Register on site

Login into your dhound security area

Go to Settings->Custom Events. Register custom security event (reserved sids: 100000-200000) and assign to particular custom security group (reserved gids: 100000-200000). This step will allow to minimize configuration in dhound-agent and use only sids and gids.

Custom rule file

Login on the server where dhound-agent installed

Navigate to the folder with dhound rules

Linux based systemcd /etc/dhound-agent/rules.d/
Windows based systemcd c:\Program Files\\dhound-agent\config\rules.d

Make a copy of a custom security event rule file (use separate rule file for each log file types and make friendly file name for your custom security events)

Linux based systemsudo cp custom.yml mycustom.yml
Windows based systemcopy custom.yml mycustom.yml

Configure rules for collecting custom event. The template contains comment how to fill the fields:

# for more information about collecting custom event see here -
# example of collecting custom events
#define list of files to parse, use asterisk to include files with dynamic file names
- /var/log/app/applog*
- /var/log/app/*accesslog
- 'c:\inetpub\logs\LogFiles\FTPSVC*\*.log'
# exclude files that should not be parsed, dhound does not support parsing archives
excludefilesregex: ((.gz)|(.zip))$
# format of time in log files, milliseconds are not supported
eventtimeformat: YYYY MMM DD hh:mm:ss
#define the max age in hours of files to parse
deadtime: 360h 
# define list of events that can be extracted from source files
  # (required) custom security id should be in rang 100000-200000
- sid: 100001 
  # (required) Posix regex with named variables, <eventime> and <ip> are required to be presented in event, 
  # all other named variables will be extracted from regex match and include into the event information
  regex: ^(?P<s1>.+?) (\S+?) (\S+?) \[(?P<eventTime>.+?)\] "POST /wp-login\.php (.*?)" 302 \d+ "((http(s|)://(?P<field1>.*?)/.*?)|-)"
  # (optional) override default message associated with this security event, parsed fields can be injected into a message  by using # prefix before field
  # message: #field1. access admin part of the 
  # (optional) override default security group that this security event will belong to
  # gid: 100000
  # (optional)additional static fields that should be included into events
  # if ip is not parsable from regex, it can be included as statis field, for example, ip: local
  # fields:
    # field1: <static value>
  # (optional) per each field can be added regex expression, if regex match is success, this event will be ignored and will not be sent on server
  # exclude:
    # ip: ((0\.)|(127\.)|(192\.168\.)|(10\.)|(172\.(1[6-9]|2[0-9]|3[0-1])\.)|(fc00:)|(fe80:))

Required fields:

  • paths: list of log files to parse, use asterisk to include files with non-static names
  • excludefilesregex: posix regex to exclude particular files or file types
  • eventtimeformat: format of event time in parsing log files
    • M - month (1)
    • MM - month (01)
    • MMM - month (Jan)
    • MMMM - month (January)
    • D - day (2)
    • DD - day (02)
    • DDD - day (Mon)
    • DDDD - day (Monday)
    • YY - year (06)
    • YYYY - year (2006)
    • hh - hours (15)
    • mm - minutes (04)
    • ss - seconds (05)
    • AM/PM hours: 'h' followed by optional 'mm' and 'ss' followed by 'pm', e.g.
    • hpm - hours (03PM)
    • h:mmpm - hours:minutes (03:04PM)
    • h:mm:sspm - hours:minutes:seconds (03:04:05PM)
    • Time zones: a time format followed by 'ZZZZ', 'ZZZ' or 'ZZ', e.g.
    • hh:mm:ss ZZZZ (16:05:06 +0100)
    • hh:mm:ss ZZZ (16:05:06 CET)
    • hh:mm:ss ZZ (16:05:06 +01:00)
  • sid: custom security id that you registered on the site
  • regex: regular expression in posix format for extract security event data from log file

Check validness of yaml rule file in a yaml tool (for example, Yaml Online Converter)

Load into dhound-agent

Open dhound-agent main config file: /etc/dhound-agent/config.yml(Linux) or c:\Program Files\\dhound-agent\config\config.yml(Windows)
# for more information about dhound-agent configuration see here -
  # (required) client access token
  # (required) server identifier
  serverkey: MMMDDDFFFBLK
  # (optional) proxy settings
  # proxy: http://localhost:8080
  # enable all rules specified in rules.d folder: true/false
  allrules: true
  # if allrules is false, specify name of rule files without extension (for example, sshd, tcp-out, etc)
  rules: [sshd, fail2ban, pure-ftpd, tcp-out, apache-wordpress]
# network interface (optional), used for track investigation, only for Linux systems networkinterface: "eth0" # this is useful functionality for output traffic incidents investigation, not available on arm devices trackdnstraffic: true If allrules set to false, add your created file (without extension) to the rules array.

Reload dhound-agent

Linux based systemsudo /etc/init.d/dhound-agent restart
Windows based system# reload manually 'Dhound Agent' windows service

Check log files for errors produced by dhound-agent

Linux based systemtail -f -n 100 /var/log/dhound-agent/dhound.log
Windows based systemc:\Program Files\\dhound-agent\log\dhound-agent.log

See Also