## SEC - Simple Event Correlation Tool ##


SEC's a really nice, lightweight tool that runs through using perl regex primarily,
parsing any given logs that are generated within the /var/log/ directory. It's a nice
middle road between org/commercial software and the base logging [and highly flexible]
ability inherent to nix os's. This tool is easily one of the most powerful opensource tools
available for such things.

The sec tool can single out a particular message, or it can group together sets of related
log messages [i.e: when a particular user has logged into say, ssh]. You can
set threshold limits for sec that once these are reached - then a particular
action [or actions] can be taken.

The given action to be taken can be one of any number of potential avenues, such as
feeding any of the results into a bash script, or chained together through a multitude of
bash built-ins or non-built-ins - all depending on what the person wants as an end
result for their particular sec message/log results.

SEC does have an incredibly complex depth to its capabilities. Everything won't be covered here, aside from installation, config, running it at a simplified level [host/home network], how to direct your logs to sec, and doing some basic pattern-matching through it's perl/regex capabilities.



** Granted this tutorial is only for particular folk with an interest in such things. Home network/host-based log detection/correlation is what this tutorial attempts to show, though this can obviously be scaled out quite alot in order to reach it's intended use/goal. Though this can be a nice learning exercise when learning to deal with rsyslog/linux logging && event correlation. If this tutorial in some form sparks interest, then it's done it's job.

** Apologies beforehand, I used mousepad on linux to draft this, so some paragraphs might look a bit goofy.


Requirements:

-Linux
-Simple Event Correlation Tool
-Knowledge of basic linux commands
-Knowledge of syslog/rsyslog, configuration files & the linux logging system
-Knowledge of regular expressions


Download through git clone:

git clone https://github.com/simple-evcorr/sec.git
Cloning into 'sec'...
remote: Enumerating objects: 184, done.
remote: Counting objects: 100% (24/24), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 184 (delta 12), reused 20 (delta 12), pack-reused 160
Receiving objects: 100% (184/184), 392.18 KiB | 498.00 KiB/s, done.
Resolving deltas: 100% (107/107), done.[/i]

## Change directory into the sec folder, copy the 'sec' binary into /usr/local/bin
in order for it to be ran anywhere within the linux directory structure. Then copy
the man page 'sec.man' to /usr/local/share/man/man1/sec.1 in order for it to be
accessed/ran anywhere within the linux directory structure.

cd sec/
sudo cp sec /usr/local/bin
sudo cp sec.man /usr/local/share/man/man1/sec.1


## Man page:

man sec


## To start:

sudo sec

## Output:

SEC (Simple Event Correlator) 2.9.0
No --bufsize command line option or --bufsize=0, setting --bufsize to 1
Interactive process, SIGINT can't be used for changing the logging level..

**Not much going on here, so we need to start sec with a few given options.**



## The first option being 'input=', then specifying the full path to a given log/set of logs. This is one of many potential sources of input that can be funneled to sec:

sudo sec -input=/var/log/syslog

**BUT.. this is only one scenario for an input value. Later in this tutorial I will go over
a more common, and slightly better value instead of specifying an actual log path as
a source.**

## The next option to be run with the sec command is '-conf='. SEC doesn't actually come with it's own configuration file, you need to make it yourself. It's fairly simple though:

- Create a directory in /usr/local/ called 'sec', and also set strict permissions
to the sec directory with chmod 700, all in one shot:

mkdir /usr/local/sec && sudo chmod 700 /usr/local/sec

** Don't use sudo here with 'mkdir', otherwise you won't be able to cd into the new directory, because 'sudo' in this case only works with shell non-builtins 'mostly', though there are several shell builtin's that work with sudo. The 'cd' command is what's called a 'shell builtin'. So exclude using 'sudo' with mkdir. There's way's around this, though it's not inherently a good practice to do those things ime.

** If you're not familiar with shell builtins vs. shell non-builtins, please reference
the internet for the complete lists.

** A nice command to check whether a command is a shell builtin/non-builtin is 'type':

ex:

'type cd'
'type echo'


## Now use touch to create a base config file in the sec directory, and also
set permissions with chmod to 700 [strict perms, only the primary user can r/w/x].
You only want YOU getting into this directory and noone else:

sudo touch /usr/local/sec/sec.conf && sudo chmod 700 /usr/local/sec/sec.conf

** You can create and list multiple config files to the '-conf=' option if
need be. Creating and specifying multiple config files can allow you to define
multiple rulesets for different situations.

** For now we'll just use this single base .conf file, and now our sec
command looks like this thus far:

sudo sec -input=/var/log/syslog -conf=/usr/local/sec/sec.conf



## Now you need to create a separate log file in /var/log/ to store any matches generated
from the sec tool. You also need to set the user:group for the log file to root:adm.

** 'adm' means administrative and is used as the primary group for all things logging. Good
practice to set the group to this.

So in one line:

sudo touch /var/log/sec.log && sudo chown root:adm /var/log/sec.log && sudo chmod 660 /var/log/sec.log



## Now we use the '-log=' in the sec command option to specify the path to our sec.log
which we'd just created, so now the command curently looks like:

sudo sec -input=/var/log/syslog -conf=/usr/local/sec/sec.conf -log=/var/log/sec.log



## Now we use the debugging level option '-debug=' which sets the verbosity level for sec. The default value is 6 - 'log all given activites for the specified input file'. For this example we'll just set it to 6 anyhow so that we're aware. Now the command looks like:

sudo sec -input=/var/log/syslog -conf=/usr/local/sec/sec.conf -log=/var/log/sec.log \
-debug=6

** The '\' allows line continuation for your command if it begins to get too long. Using
the backslash prettifies extra long command line commands. User preference though.



## Now the last option we'll run is '-detach', which allows sec to run as a background daemon:

sudo sec -input=/var/log/syslog -conf=/usr/local/sec/sec.conf -log=/var/log/sec.log \
-debug=6 -detach

** Unless you want the entire output display of sec to go to stdout [standard output/terminal], then you can do so, otherwise just stick it to the background.



## Now to check that the sec daemon is running, if stdout on the terminal doesn't fully
display the run completion, you can just check the /var/log/sec.log, as it should post the full stdout display/run completion:

sudo cat /var/log/sec.log

Thu Oct 21 19:58:52 2021: SEC (Simple Event Correlator) 2.9.0
Thu Oct 21 19:58:52 2021: Changing working directory to /
Thu Oct 21 19:58:52 2021: Reading configuration from /usr/local/sec/sec.conf
Thu Oct 21 19:58:52 2021: No valid rules found in configuration file /usr/local/sec/sec.conf
Thu Oct 21 19:58:52 2021: No --bufsize command line option or --bufsize=0, setting --bufsize to 1
Thu Oct 21 19:58:52 2021: Opening input file /var/log/syslog
Thu Oct 21 19:58:52 2021: Daemonization complete


** To restart the sec daemon the easiest way imo is the killall command, with
the -HUP signal. The HUP signal is a nice way to restart this daemon. The HUP signal
when used with a running daemon allows the daemon to reload it's current configuration:

sudo killall -HUP sec

## Easiest way to collect log data and analyze it through sec is just to use a named pipe to funnel data from the specified log paths in /var/log/ using the command 'mkfifo'. We'll just call this pipe 'psec':

sudo mkfifo /var/log/psec

^^ This will replace the earlier stated 'input=/var/log/syslog' i'd mentioned, as named pipes for this tool are more efficient [ime] This named pipe will pick up all the specified logging data from the respective logging daemon/s.

** If you're not familiar with named pipes and their usage, please refer to the man page or
similar reading online.

## Now for the above to work you need to edit/specify the new named pipe and which facilities/priorities you want to collect in '/etc/rsyslog.conf'. For now I'm just going to collect all kernel logs with a priority level of 'warning' or higher, and i'm going to specify the named pipe 'psec' that was made earlier:

kern.warning |/var/log/psec

^^ When specifying the path to a named pipe in the rsyslog.conf file, the standard
pipe command '|' needs pre-appended to the specified named pipe path in the rsyslog.conf file.


## Now the updated command looks like:

sudo sec -input=/var/log/sec -conf=/usr/local/sec/sec.conf -log=/var/log/sec.log -debug=6 -detach

^^ Although we've specified our input/source, we still will need to write some rules in our newly made sec.conf file in /usr/local/sec/sec.conf in order for sec to be able to comb logs.


## Writing bundles of rule statements in sec.conf is fairly straightforward. Each rule statement consists of 'key=value' pairs. Each line in each rule statement consists of this 'key=value' pair.

## The sec tool recognizes blank lines and lines with '#' to be the end of the the entire rule statement, so be careful using either of those.

## An example of an extremely oversimplified rule statement from the /usr/local/sec/sec.conf file that searches the given log line/s for an 'ssh' substring, then taking the given line from the log and writing to /var/log/sec.log:

type=Single
continue=TakeNext
ptype=SubStr
pattern=ssh
desc=matches 'ssh' substring from log line
action=shellcmd /bin/echo '$0' >> /var/log/sec.log

## Below is the next rule statement in the /usr/local/sec/sec.conf file. This is a continuation from the first rule statement, with a little more complexity. This rule statement looks for the substring 'root' in the log entries picked up by the previous rule statement, and if any of those lines contain 'root' as a substring, then this rule parses through the lines in the sec.log file, grep'ing out all the source ip's pertaining to root ssh login attempts, putting them in a separate file for even further processing/action if need be:

type=Single
continue=DontCont
ptype=SubStr
pattern=root
desc=continues matching log lines for root attempts
action=shellcmd touch ~/rtiplogin.txt && cat /var/log/sec.log | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" >> /path/to/iplist.txt


## Explanation of above key=value pairs in the sec.conf file:

- The 'type' key is set to the 'single' value, which combs lines in succession, taking an action based on the pattern specified. This is the default value. There's no need to change this right now, though later I'll show a tiny tiny bit more of secs capabilities in this area.

- The 'continue' line specifies either 'TakeNext' or 'DontCont'.

-'TakeNext' tells the given rule statement to keep on searching through the log entries for other rules that may apply [per the sec.conf file].

-'DontCont' tells sec that if the given rule statement matches the log entry then to perform
the given action for that rule statement and then stop at that particular statement and not to continue matching against any other rule statements. So sec will continue matching and performing actions until it meets the 'DontCont' value.


- Next the 'ptype' line specifies how the following line 'pattern' is to be interpreted through choice of sec's pattern matching capability. The pattern matching types are:

-RegExp[num]: a perl regular expression
-SubStr[num]: a sub string
-NRegExp[num]: negated regular expression
-NSubStr[num]: negated sub string

** For this tutorial we won't be using the 'N' prefixed pattern types.

** '[num]' defaults to '1' [i.e: the last entry in the log file] though you can specify a number that will compare the pattern given for the last [num] amount of log entries.

- The 'pattern' line is the specified pattern per the previous line 'ptype'. To be super simplistic I just used the ptype of 'SubStr'. Though you can write as thorough of a regular expression as needed if you use 'RegExp'.

** If you're unfamiliar with regular expressions/perl regex there are useful tools/generators online that will take your line of characters/strings and convert them to a regexp of your choice.

- The 'desc' line is nothing but a short description of whichever event's being matched per the rule statement.

!! I won't be going into much more detail on the pattern matching types and how to write them. I'll leave that up to the reader to figure out. The sec man page [as with any man page] is your absolute best friend. Always remember.. 'man pages > tutorials' in the long run.

- The last line is 'action', one of the most important lines. This is where this tool shines, and with linux's near infinite capability/malleability. Scrolling back up to my example sec.conf file with the 2 rule statements, the second/last rule statement has the action:

shellcmd touch ~/rtiplogin.txt && cat /var/log/sec.log | grep -oE "\b([0-9]{1,3}\. {3}[0-9]{1,3})\b" >> /path/to/iplist.txt

** This action is pretty simple, and for the sake of brevity and time I decided not to write anything too verbose. In this example file's created called 'rtiplogin.txt' in the whateverusername's directory, then cat's ran to send out the full sec.log files contents through a pipe to grep with the 2 option '-oE', which specifies to print only the matched parts of the line using 'E' extended regular expressions.

!! If you don't know the difference between regexp and e-regexp there are plenty of resources out there. <3

** The 'action' line you can also specify 'positional variables' based on your 'pattern' line. Closed parenthesis '( )' when using regexp here denotes a positional variable. So the first set of parenthesis would denote $1, the second set of parenthesis $2, so on n' so forth.

!! Using these positional variables I will show in a later example. Positional variables such as these have alot of power when you have the need to correlate log entries that are directly related.



- The above as I said is overly simplified, and the capabilities of this tool extend much much greater than above. We'll look at some other potential avenues with a bit greater depth. We'll look at another rule type called 'SingleWithThreshold'.

- The 'SingleWithThreshold' rule type matches the pattern/s accordingly and keeps count of this within a specific time window.

- For 'SingleWithThreshold' you need to also specify the 'window' line in your sec.conf file rule statement. Following the 'window' line you also need to specify the time value in seconds, so for this example we'll just do 'window=40'. The sec tool starts counting as soon as the first match is made per the rule statements.

- Also the other line to use in this instance is 'thresh'. This is the threshold limit. This is followed by the total number of matches, so for example: 'thresh=8' means that per the given time window of 40 seconds if there's 8 matches to the rule then the proceeding action will be taken, in this case if the first rule statement in this conf file succeeds, the next rule begins [notice the window and thresh keys added to the second rule statement], then if root login is picked up at a rate of 8 root ssh login attempts in 40 seconds, then the following action is started - which as i stated before - sec runs the touch command to create a root ip login list 'rtiplogin.txt', then cat's ran to send out the given log entries, piping this to the grep command with the options '-oE' to grab the ip's for those specific login attempts, which are then wrote to 'rtlogin.txt' for further processing/action [though in this sec.conf example] there's no further processing due to the rule line 'continue=DontCont'.

## Below is the updated rule statements, including what I'd mentioned above:


type=Single
continue=TakeNext
ptype=SubStr
pattern=ssh
desc=matches 'ssh' substring from log entry
action=shellcmd /bin/echo '$0' >> /var/log/sec.log



type=SingleWithThreshold
continue=DontCont
ptype=SubStr
pattern=root
desc=continues matching log entry/s for root attempts
window=40
thresh=8
action=shellcmd touch ~/rtiplogin.txt && cat /var/log/sec.log | grep -oE "\b([0-9]{1,3}\.) {3}[0-9]{1,3}\b" >> /path/to/iplist.txt

**'$0' is a positional variable, which in this case it's associated with the log entry that's being picked up given this rule statement. The action echo's this picked up log entry to the sec.log file.

**SEC has a vast collection of rule types aside from 'Single' and 'SingleWithThreshold', obviously I will not go over ALL of them. The extensibility of this tool in this area is truly something. The man page for this is massive. A whole book could easily be written on this tool's capabilities

** I'll show one more rule type in hopes to highlight this tools capabilities:



## The 'Pair' rule type

-The 'Pair' rule type uses a pair of matches in one rule statement. It matches the first event and executes the specified action. This entire rule statement will disregard any subsequent rule statements/matches in the config file until the pair of matched events in this rule statement succeeds. This one is pretty neat ime.

-Below is an example of this:

-Sample log entry for 'su':

Jun 12 09:42:55 host1 su(pam_unix)[11214]: session opened for user root by john(uid=500)
Jun 12 10:20:13 host1 su(pam_unix)[11214]: session closed for user root

[i]type=Pair
ptype=regexp
pattern=(\w+\s+\d+\s+\d\d:\d\d:\d\d)\s+(\w+)\s+su\(pam_unix\) /
(\[\d+\])\:\s+session opened for user root by (\w+)\(\w+\=\d+\)
desc=User $4 was successful in su to root at $1 on system $2. Accept username: $4?
action=shellcmd /bin/echo '%s' | /bin/mail -s "Superuser Attempt: Alert" admn@yourdomain.com
ptype2=regexp
pattern2=(\w+\s+\d+\s+\d\d:\d\d:\d\d)\s+$2\s+su\(pam_unix\) $3\:\s+session closed for user root
desc2=Potential mal username: %4 closed their su session at %1 on system %2
action2=shellcmd /bin/echo '%s' | /bin/mail -s "SU Session Close Warning" admn@yourdomain.com

^^ Here you are getting matches on 2 separate, but related, log lines/entries. Looking at
the last half of this rule statement notice how there's 'ptype2', 'pattern2', 'desc2',
'action2'. These appended '2's are specifying the second half of this paired rule statement.
The other reason is that you are matching two separate but related log entries.

** '%s' specifies the 'desc' line string

## For the first half of this rule statement I have used the regexp pattern type.
Inside the pattern I have also defined a number of positional variables/elements of the
log entry I'm to match as variables: the hostname on which the su login session took place,
the user who issued the command, the time that this login session opened and closed,
and the process id [pid] that issued the su command. You can then see that I have used some
of these positional variables in the 'desc' and 'action' lines.

## The second pattern type on the last half of this rule statement will also be using
regexp. With this pattern the first question is - how do you know if the log entry indicating the end of the su session is related to the inital log entry opening the su login session?

## Well, sec can use these positional variables from the first pattern line 'pattern' and these variables can form part of the regular expression being matched in the second pattern line, 'pattern2'. In the first pattern line 'pattern' I defined the hostname of the system the su login session was active on as '$2' and the process [pid] of the session as $3. If you refer to those variables in the pattern2 line, then the sec tool knows you are referring to variables defined in the initial/first pattern line. You then use the host name and process id [pid] to match the incoming log entry against the first log entry.

## This also brings up another question. How does the sec tool tell the difference between the positional variables defined in the two pattern lines when I use them in the desc2 line?

## Well, the positional variables for the first pattern line, if I want to use them again in the 'desc2' or 'action2' lines, are prefixed by '%'. Then the positional variables from the second/bottom half pattern line are prefixed with $. You can see I have used the '$4' variable defined in the first pattern line in the desc2 line by calling it '%4'.


## Ok, this tutorial is getting a bit long and quickly leaving the scope of things. What's been shown is an incredibly small fraction of what can be set up and done with this tool.