Ensure vital service logging sends to salt master AND locally

[renoirb]

Improve logging from all vital services.

Policy: send to syslog local1 facility AND locally. So that we can still get all logs on salt master, BUT not lose anything in case of lost UDP packets.

Fact; except salt, every nodes uses rsyslog. salt master uses syslog-ng instead.

Services

• db*, for MariaDB; maybe activate free MariaDB Audit plugin
• mail*, for Postfix, OpenDKIM, etc.
• app*, project*, for mod_php
• piwik*, for php5-fpm
• notes*, for hypothesis (through gunicorn?)
• account*, for various FXA services (through intel?)
• ...

Ref

• https://mariadb.com/kb/en/mariadb/general-query-log/
• http://serge.frezefond.com/2013/09/mariadb-audit-plugin-logging-to-syslog/
• http://www.reddit.com/r/PHP/comments/20qgjj/logging_php_errors_to_syslog/
• http://www.rsyslog.com/doc/rsyslog_conf_filter.html
• http://gunicorn-docs.readthedocs.org/en/latest/settings.html?highlight=log#logging

[DanyC97]

Hi,

first of all great project 👍
May i ask you why you decided to send all the logs to master instead of pushing from each minion/ VM directly to ELK ?

Thanks and keep doing this great work :)

[renoirb]

Hi @DanyC97 because one of the oldest feature of Unixes always had been to distribute log messages through network and is as easy as enabling logger daemon to receive packets. Nothing else to install. Note that the logging infrastructure was already in place when I arrived in 2013, it was done by @ryan-lane at that time. Also, ELK was really new back then.

[ryan-lane]

It was easy and quick to setup and we only had 2 people looking at the system logs at the time. I think heka + elasticsearch + kibana is a good option nowadays.