🌐 🇵🇱 Polski · 🇬🇧 EN
How can we know that something is wrong with the system even if it is not visible at first glance? Obviously, you have to check the logs! Seriously, understanding how the logging system works is very important for troubleshooting issues that may be encountered while administering a Linux environment. Logs store all information about the kernel, the startup process, and various services that have been started in the system. Besides logs, there are also tools you can use to monitor and verify the correct operation of the environment. In this part, we will see how you can use the available tools and logs to diagnose occurring problems and find their solutions. In the next part, we will also learn ways to automate this process.
WORKING WITH SYSLOG
When something goes wrong in the system, the running "syslog" service will generate a message. Messages generated in this way record how the system is operating, what actions have been taken, and with what and where a problem appeared. The "syslog" service is built into the RedHat environment and is started along with the system at boot. The "rsyslog" package is installed by default in the system and is the basic, essential tool used for logging events occurring in the system.
In RHEL5, logging was split by default between two services: syslogd and klogd. Syslog handles system messages, and klogd handles kernel messages. In the RHEL6 release, both services were replaced by rsyslogd, which handles both types of messages. If you are sticking with the RHEL5 release, you can install the syslog service from the syslogd package.
The rsyslog service has a main configuration file, /etc/rsyslog.conf, which controls where generated messages are sent. Since logs are a critical element of the operating system, this package is installed by default, becoming part of the operating system.
Step 1 - It is good practice to check if the package is actually present in the system
rsyslog-4.6.2-2.el6.x86_64
Since it is visible that it is installed, we should check two other issues
Step 2 - First - make sure that the service has been set to start with the system
# chkconfig rsyslog --list
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on6:off
Step 3 - Second - check if the service is actually running
# service rsyslog status
rsyslogd (pid 1279) is running...
For the RHEL5 release:
Step 1 - Check if the package is in the system:
# rpm -qa | grep klog
sysklogd-1.4.1-44.el5
Step 2 - Check if it is started with the system during boot
# chkconfig syslog --list
syslog 0:off 1:off 2:on 3:on 4:on 5:on6:off
Step 3 - Check if it is running
# service syslog status
syslogd (pid 2214) is running...
klogd (pid 2217) is running...
Now you can see the slight differences between RHEL releases regarding logging. The rsyslog service performs better, and it is recommended to use and install it wherever possible.
All messages sent by the rsyslog service are stored in the /var/log directory, divided into various files and subdirectories. You can also define your own log file and directory. The biggest problem with logs is the fact that if they are neglected, it is difficult to manage them later and keep them in order. Since logging absolutely everything is an art in itself, it is better to plan to log only the elements relevant to a given system. There are nine different notification levels used by the syslog service:
- emerg
- alert
- crit
- err
- warning
- notice
- info
- debug
- none
CONFIGURATION FILE
The /etc/rsyslog.conf file is divided into the following sections:
- Modules - Specifying which module should be loaded or disabled
- Global directives - Service startup configurations (all start with the $ symbol)
- Rules - Specifying the cooperation between the selector and the action
- Selector - Based on <facility>.<priority>
- Action - Definition of what to do with a message after it has been sorted by the selector
Let's look at the default configuration contained in the file:
# cat /etc/rsyslog.conf
#rsyslog v3 config file
# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance
#### MODULES ####
$ModLoad imuxsock.so # provides support for local system logging (e.g. via
logger command)
$ModLoad imklog.so # provides kernel logging support (previously done by
rklogd)
#$ModLoad immark.so # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp.so
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp.so
#$InputTCPServerRun 514
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# File syncing capability is disabled by default. This feature is usually not
required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don’t log private authentication messages!
*.info;mail.none;authpriv.none;cron.none /var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.*
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg *
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.* /var/log/boot.log
# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$WorkDirectory /var/spppl/rsyslog # where to place spool files
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList # run asynchronously
#$ActionResumeRetryCount -1 # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###
You can see how the log files have been sorted and what default rules they are subject to.
LOG ROTATION
Logs left to their own devices can grow to incredible sizes. Fortunately, we can use the "logrotate" command, which allows for splitting and rotating log files before they become too large. By default, "logrotate" relies on daily cycles, creating a new file every day. Parameters and configuration are located in the /etc/cron.daily/logrotate file, and further work is performed daily by "cron". You can also always invoke the logrotate command if you want to perform log rotation, however, it is not recommended to do this with high frequency; you simply need to maintain moderation and caution to keep order and tidiness in the organization of log files.
Logrotate command syntax: # logrotate [options] configuration_file
Logrotate command options:
- -d - Debug, do nothing but test
- -f - Force log rotation
- -v - Display information about actions being performed
For a test, you can perform rotation of current logs by defining it in the configuration file:
# logrotate /etc/logrotate.conf
CENTRALIZED LOGGING
If we want, we can establish a single place for storing logs from multiple machines in a given environment. You must decide which server will be designated for storing logs. For example, we can use an RHEL01 machine and use it as a central log storage location.
Step 1 - Let's look again at the sections of the /etc/rsyslog.conf file
#### MODULES ####
$ModLoad imuxsock.so # provides support for local system logging
(e.g. via logger command)
$ModLoad imklog.so # provides kernel logging support (previously
done by rklogd)
#$ModLoad immark.so # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp.so
#$UDPServerRun 514
# Provides TCP syslog reception
#$ModLoad imtcp.so
#$InputTCPServerRun 514
In the modules section, we see two places that concern storing logs in a different location. It is standard to use the UDP protocol on port 514. Uncommenting the UDP section will enable the operation of a central server for storing logs on the network.
Step 2 - Let's remove the comment character for the two lines.
$ModLoad imudp.so
$UDPServerRun 514
Step 3 - We save the file and restart the syslog service.
# service rsyslog restart
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
After these steps, for the service to be visible on the network, you need to set the appropriate permission in the firewall allowing UDP connection on port 514.
Step 4 - Let's apply "iptables" by creating an appropriate rule
# iptables -I INPUT 5 -p udp -m udp --dport 514 -j ACCEPT
Step 5 - Let's save the firewall rule after creation
# service iptables save
Saving firewall rules to /etc/sysconfig/iptables: [ OK ]
Step 6 - Restart the "iptables" service
# service iptables restart
iptables: Flushing firewall rules: [ OK ]
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Unloading modules: [ OK ]
iptables: Applying firewall rules: [ OK ]
At this moment, firewall settings might not be the most important thing because we are using standard rules, and in fact, to test the operation of the discussed topic, you can disable the "iptables" service.
Now that we have the central log server service running, we can take care of the client configuration. We will do this on the RHEL02 machine. The configuration will cause all logs from the RHEL02 machine to be sent to the log server, i.e., RHEL01.
According to the configuration file structure, logs are to be sent to the central log server.
<log file> @<hostname or IP of system (local or remote)>
You need to perform edits on the RHEL02 machine:
# vi /etc/rsyslog.conf
authpriv.* @172.168.1.1
We save the file and restart the syslog service on the RHEL02 machine
# service rsyslog restart
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
If any security event is generated on the RHEL02 machine and saved to the log, you will be able to see such entries in the files located on the RHEL01 machine.
When specifying which logs should be sent to a central server located locally or remotely, you can use 5 options:
# Send messages using the TCP protocol instead of UDP
Authpriv.* @@172.168.1.1
# Displays messages on the console instead of sending them remotely
Authpriv.* @system
# Discard all messages generated for this log file
Authpriv.* ~
# Send alert to all log files ** use with caution **
Authpriv.* *
These options can be used for sending logs to a remote server while maintaining their local copies.
CENTRALIZED LOGGING version for RHEL05
Step 1 - We edit the /etc/sysconfig/syslog file containing the entry with the "-r" option
# vi /etc/sysconfig/syslog
# Options to syslogd
# -m 0 disables ‘MARK’ messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages received with -r
# See syslogd(8) for more details
SYSLOGD_OPTIONS=”-m 0 -r”
Step 2 - After saving the file, we restart the syslog service
# service syslog restart
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
When the syslog service already accepts remote log storage, we need to set the appropriate firewall rule. This proceeds identically as in the case of RHEL06, which was shown just a moment ago. The last change that needs to be made in RHEL05 is setting SELinux security.
Step 3 - Query invoking boolean values
# getsebool -a | grep logd
klogd_disable_trans --> off
syslogd_disable_trans --> off
Step 4 - Changing values to allow external logs to be admitted
# setsebool -P klogd_disable_trans=1 syslogd_disable_trans=1
Step 5 - Checking settings after changes
# getsebool -a | grep logd
klogd_disable_trans --> on
syslogd_disable_trans --> on
Although the SELinux configuration has not been fully performed yet, there are default values that had to be changed; to understand exactly what was done, you can return to this point when the SELinux topic is discussed in later parts.
LOGGING USER SYSTEM LOGIN EVENTS
Besides the normal logs generated by the syslog service, there are two special commands regarding user logging. These commands create special entries that can also be read using the same commands.
- lastlog - Record of the last login
- faillog - Last failed login
We can use these two commands to review events related to users logging into the system. These commands also allow us to track attempts to break into the system using techniques called brute-force-attack.
Lastlog command syntax: # lastlog [options]
Lastlog command options:
- -b DAYS - Displays results older than the given day
- -u LOGIN - Displays results for a given user
Let's see when the user USER01 last logged into the system:
# lastlog -u user01
Username Port From Latest
user01 tty Fri Sep 10 05:16:42 -0400 2010
We can also find out more details using the faillog command.
Faillog command syntax: # faillog [options]
Faillog command options:
- -a - Display all events
- -l SEC - Lock account after SEC seconds after providing a wrong login
- -u LOGIN - Display entries regarding a given user (LOGIN)
So let's see more information related to user USER01
# faillog -u user01
Login Failures Maximum Latest On
user01 0 0
The ability to work with the syslog service is important because it facilitates troubleshooting issues related to the improper operation of the Linux system, not just RedHat. If you need to find a solution to improper system operation, the first step is always to familiarize yourself with the entries in the logs.
Comments