Linux RH442 - System monitoring tools. Vmstat, sar, iostat, mpstat.

An important element of system control and operation is reading current parameters of individual hardware components, such as memory capacity and its current utilization. Equally important parameters include CPU and disk resources, as well as the number of open files and running processes; all of these affect the efficient operation of a Linux environment. It is therefore beneficial to have the ability to quickly view the current status of these parameters, which can be done using tools available in a standard system installation. By having insight into the current levels of individual components, an administrator can take appropriate actions to protect the environment from overload, ensuring its continuous operation at an appropriate level. 

VMSTAT - Virtual Memory Statistics

Vmstat is one of the most commonly used basic tools for diagnosing what is happening with system memory. This tool is part of the "procps" package, which also contains other well-known tools such as "free" or the popular "top". 

The vmstat command, when used without any parameters, displays average values measured since the system boot. With vmstat we can use two parameters, one of which specifies the time interval for the measurement in seconds. Memory statistics are presented in KiB by default, but we can set this to a different value using the second switch. Options include kb, MB, or MiB. To change the unit in which statistics are displayed, we add the following accordingly: 

-S k or -S m or -S M 

Example results may look like this: 

The first line displayed by vmstat covers the average values since system start. If we want to perform several measurements at specific time intervals, we provide the time between consecutive measurements in seconds. We can also remove the lines containing headers so that the received data can be directly processed by another tool. 

vmstat 10 | tail -n +4 

The above command will display memory information every 10 seconds without header data. 

Returning to the information headers, they mean the following: 

procs

-r                   - number of processes waiting for run time

-b                  - number of processes in uninterruptible sleep

memory 

- swpd            - amount of virtual memory used

- free              - amount of idle memory

- buff              - amount of memory used as buffers

- cache           - amount of memory used as cache

swap 

- si                  - memory swapped in from disk per second

- so                 - memory swapped out to disk per second

io (I/O block statistics)

- bi                  - blocks received from a block device per second

- bo                 - blocks sent to a block device per second

system

- in                   - number of interrupts per second

- cs                  - number of context switches per second

cpu 

- us             - time spent running non-kernel code (user time)

- sy              - time spent running kernel code (system time)

- id              - time spent idle

- wa             - time spent waiting for IO

- st              - time stolen from a virtual machine

If we want to perform a specific number of measurements, e.g., 20 measurements at 5-second intervals, the command will look like this: 

vmstat 5 20 

SAR - The System Activity Reporter

The sar command is a multi-tool belonging to the sysstat package. Sar has two operating modes. With this tool, we can read results from cron jobs every 10 minutes, and we can also collect current system data. Cron tasks are located in /etc/cron.d/sysstat and are called from /usr/lib64/sa/sa1 and /usr/lib64/sa/sa2; data is collected by /usr/lib64/sa/sadc and sar. Collected information is stored in /var/log/sa/saxx where xx are two digits representing the month and day. 

To achieve the best results using sar, ensure that the $LANG variable is set correctly to allow report generation in a consistent and readable time format. For example, if we want to sort data by the first column, in en_US.UTF-8 locale, 01:00:00 PM comes before 02:00:00 PM. Such sorting can distort data points and skew charts. The recommended setting is to use $LANG set to C. With sar, it is easy to override the variable as follows: 

LANG=C sar -q 

The result of the above command looks like this 

$ LANG=C sar -q

Linux 2.6.9-67.ELsmp (station1)    07/15/13

00:00:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15

00:10:01            1       984      0.53      0.34      0.30

00:20:01            0       984      0.10      0.20      0.24

For simplicity, you can use an alias for the sar command: 

alias sar='LANG=c sar'

Sar can be used to read and analyze log files using the "-f" switch. 

$ sar -q -f /var/log/sa/sa14

Linux 2.6.9-67.ELsmp (zenon)    07/14/2013

12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15

12:10:01 AM         0      1005      0.64      0.82      0.77

12:20:02 AM         2      1005      0.97      0.93      0.84

12:30:01 AM         1      1008      0.78      1.07      0.97

12:40:01 AM         0      1005      0.93      0.96      0.93

Example sar usage 

Task 1 

Configuring the sar view in 24-hour format.

# echo "alias sar='LANG=C sar'" >> /etc/bashrc 

Task 2 

CPU utilization report 

Task 3 

I/O and transfer report 

Task 4 

Network interface utilization report from the current log file 

iostat and mpstat 

Like its predecessor sar, iostat and mpstat are in the sysstat package. Iostat is used to diagnose whether our disk resources are healthy, which in practice means monitoring our block devices. Hard drive saturation is highly undesirable, which can cause our system to run significantly slower; thanks to iostat, we can quickly detect this and react accordingly. 

If iostat indications show utilization at 80-100%, the cause should be found and eliminated; this situation means our processor spends most of its time waiting for data rather than processing it.

The task of mpstat is to display information about CPU load in the system. In systems with multi-core processors, each core will be seen as a separate unit. Usually, this command is used with the following switches: 

mpstat -P ALL

It is also possible to save analysis from a specific time to a file, which we do as follows: