Syslog server/client

Syslog is elementary part of base system. It works like client and server at once. 
Every daemon, service, kernel, ... generates messages about its activities. These messages are stored in logs in "/var/log".
Messages can be basically divided according to daemon/service/host which generates these messages or importance (log level) of messages.
Syslog is running in background (ps aux | grep syslog or pgrep syslogd).
In client mode, syslog sends messages to host where's the syslog server running.
In server mode, syslog receives messages from hosts and stores messages in "/var/log". Note that meaning of "hosts" include also our computer (localhost), which is mainly only host from syslog receives messages.


You don't have to have "syslogd_enable="YES"" in "rc.conf" because it's running as default local service. What's more importnant is syslogd flags.
By default syslogd is running with flags "-ss -cc".
-ss - doesn't open socket(IP+port) at all (for more:man syslogd)
-cc - disables compression of repeated occurrences


So, with "-ss" or "-s" flag, syslog won't receive messages from remote hosts, only from localhost. Hence, if you want to receive those messages, change syslogd flags. Flags which I use:

syslogd_flags="-4 -b [IP] -C -cc"

-4 - runs on IPv4
-b [IP] - bind to specific IP otherwise *.*:514 (UDP)
-C - creates file-name of log if doesn't exist. (Security risk)
-cc - disable compression


You can add "-v" or "-v -v". This can be sometimes useful. Difference between both above: (from router in my LAN)

-without:

Feb 12 22:25:17 192.168.1.254 Set Device Time to:...

-v:

Feb 12 16:47:33 <1.5> 192.168.1.254 Set Device Time to:...

-v -v:

Feb 12 17:16:38 <user.notice> 192.168.1.254 Set Device Time to:...


Also, using "-d" for debug running of syslogd is very handy in troubleshooting etc. as you will see later. With this flag and "/etc/rc.d/syslogd restart", syslogd runs on foreground and it's possible to cancel it by Ctrl+C.


Config for syslogd is stored in "/etc/syslog.conf". Contains client and server options.
This is the original "syslog.conf": (commented lines removed)

*.err;kern.warning;auth.notice;mail.crit        OQ/dev/console

*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err   /var/log/messages

security.*                                      /var/log/security

auth.info;authpriv.info                /var/log/auth.log

mail.info                                      /var/log/maillog

lpr.info                                         /var/log/lpd-errs

ftp.info                                        /var/log/xferlog

cron.*                                          /var/log/cron

*.=debug                                    /var/log/debug.log

*.emerg                                      *

!ppp

*.*                                               /var/log/ppp.log

!*

*.err - every process which generates erros and higher
kern.warning - warnings and higher, just for kernel messages
* (path) - messages are forwarded to all logged in users
*.=debug - only debug, not higher
authpriv.none - authpriv is not logged
You can adjust logging to your needs with explanation above. As you can see, there are files in path (auth.log etc.). If you want to log to file, create file first (chmod 600) and then restart syslogd.


The path can be associated with host. For instance, we would like to send "mail.info" to hostname/IP logger.server.com and another example to send mentioned to user named "logger".

mail.info                                     @logger.server.com

mail.info                                     logger

Use TAB instead of spaces to separate columns.


If you don't want to receive console messages, remove or adjust corresponding line with "console" as path (first line).


One problem occurred when I wanted to receive messages from another device (router). Messages router generated were writing to several files (messages and server.log-my own). I didn't want to log it twice, that's out of main idea. I wanted to log it strictly to one file. I started syslogd with "-d" option.
I added this to the beginning of the config (before console line):

+192.168.1.254                                 

*.*                                             /var/log/server.log

It's a host block contained from two lines. In first, you add host, in second you add file to log to.

This is what I saw:

[root@MCBSD|/var/log]>/etc/rc.d/syslogd restart

syslogd not running? (check /var/run/syslog.pid).

Starting syslogd.

listening on inet and/or inet6 socket

sending on inet and/or inet6 socket

off & running....

init

cfline("*.*           /var/log/server.log", f, "*", "+192.168.1.254")

cfline("*.err;kern.warning;auth.notice;mail.crit        OQ/dev/console", f, "*", "+192.168.1.254")

cfline("*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err   /var/log/messages", f, "*", "+192.168.1.254")

cfline("security.*       /var/log/security", f, "*", "+192.168.1.254")

...

...

As you can see, at the end of each log option, +IP is added. So what that means, those messages matching the log option will be written including those from +IP. This leads to spreading messages from IP (192.168.1.254) to each logfile. This is not the purpose.


Check another case. This time we are not gonna put +IP at the beginning of the config file, but to the end of file. What will be the result?Result will be the same! Check:

[root@MCBSD|/var/log]>/etc/rc.d/syslogd restart

syslogd not running? (check /var/run/syslog.pid).

Starting syslogd.

listening on inet and/or inet6 socket

sending on inet and/or inet6 socket

off & running....

init

cfline("*.*                                /var/log/server.log", f, "*", "*")

cfline("*.err;kern.warning;auth.notice;mail.crit        OQ/dev/console", f, "*", "*")

cfline("*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err   /var/log/messages", f, "*", "*")

cfline("security.*                             /var/log/security", f, "*", "*")

...

...

Yep, you can see asterisk instead of +IP (last asterisk in each line). In system, asterisk generally means "any". This is even worse. Imagine you have more devices which are sending logs to the same server. There will be in each log file on the server tangle of messages from different IPs. 

Only workaround I've found so far is, turn the above negative to positive but before one more hint. I wrote about this:

+192.168.1.254                                 

*.*                                             /var/log/server.log

This notation is not complete. In syslog we have program and host blocks. Each have to be closed. That one above is still open. This is the correct notation:

+192.168.1.254                                 

*.* 

+*

For program block use following:

![daemon]

*.*                                             /var/log/daemon_example.log

!*

So now, with all knowledge. There is the final config with respective debug output. Config:

+localhost                                     
#
*.err;kern.warning;auth.notice;mail.crit        OQ/dev/console
*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err   /var/log/messages
security.*                                      /var/log/security
auth.info;authpriv.info                         /var/log/auth.log
mail.info                                       /var/log/maillog
lpr.info                                        /var/log/lpd-errs
ftp.info                                        /var/log/xferlog
cron.*                                          /var/log/cron
*.=debug                                        /var/log/debug.log
*.emerg                                         *
#
!ppp
*.*                                             /var/log/ppp.log
!*
+*
#
+192.168.1.254
*.*                                             /var/log/server.log
+*

Debug:

...

off & running....
init
cfline("*.err;kern.warning;auth.notice;mail.crit        OQ/dev/console", f, "*", "+localhost")
cfline("*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err   /var/log/messages", f, "*", "+localhost")
cfline("security.*             /var/log/security", f, "*", "+localhost")
cfline("auth.info;authpriv.info                      /var/log/auth.log", f, "*", "+localhost")
cfline("mail.info               /var/log/maillog", f, "*", "+localhost")
cfline("lpr.info                  /var/log/lpd-errs", f, "*", "+localhost")
cfline("ftp.info                   /var/log/xferlog", f, "*", "+localhost")
cfline("cron.*                      /var/log/cron", f, "*", "+localhost")
cfline("*.=debug         /var/log/debug.log", f, "*", "+localhost")
cfline("*.emerg                                  *", f, "*", "+localhost")
cfline("*.*                                      /var/log/ppp.log", f, "ppp", "+localhost")
cfline("*.*             /var/log/server.log", f, "*", "+192.168.1.254")

...

...

Instead of "localhost" you can use "@" as alias for hostname but I prefer more clear expression. Finally, you can see that all logs come to right file.

I noticed using "localhost" may cause syslog stop working. I've noticed that local machine sends logs as hostname. For example hostname of my computer is "MCBSD" so if I use "localhost" there are no messages from "localhost" at all, even we think "localhost" as local machine or name for IP of local machine. So be aware and rather use hostname of your computer.


In order to test syslog, check utility "logger" and man page.


Some final hints: 
-don't test syslog server to receive logs from itself (+localhost)
This can causes circular behavior.
-don't bind syslogd to IP obtained by DHCP during boot without SYNCDHCP in rc.conf for particular interface specially you have debug flag for syslog in rc.conf. Can overload system.
-make sure the UDP port 514 is permitted by firewall