Linux Networking: How to Identify and Diagnose Bandwidth "Hogs" at the Kernel Level

Slow internet on a Linux server is a common headache for administrators, but the root cause often lies not in the router but within the system itself. How can you detect processes "hogging" bandwidth before they overload the network? The answer lies in kernel-native tools—from eBPF to conntrack—and the skillful combination of classic CLI commands with modern monitoring methods.

Why Linux "Eats" All the Bandwidth: Common Symptoms at First Glance

Before diving into kernel-level diagnostics, it’s worth recognizing the typical symptoms of bandwidth issues. Often dismissed network slowdowns may stem from hidden processes that aren’t necessarily visible in standard system tools. Here are the symptoms to watch for:

1. Sudden network "drops" despite low router traffic. The router shows minimal load, but the server operates sluggishly.
2. High network interface utilization—checked with ip -s link or ethtool -S eth0, where values rx_bytes or tx_bytes skyrocket.
3. Processes stuck in "D" (uninterruptible sleep) state in top or ps, indicating blocked network operations.
4. Bandwidth limit exceeded on the interface—e.g., a 1 Gbps link with total traffic hitting 900 Mbps even when no users are active.
5. Kernel logs flooded with network error messages (dmesg | grep -i net).

“Remember: network issues aren’t always the ISP’s fault. Often, the culprit is a background process uploading data to the cloud or participating in a DDoS attack.”

Kernel-Level Traffic Monitoring Tools: eBPF, nftables, and conntrack in Action

The Linux kernel provides powerful mechanisms for network traffic monitoring that operate with minimal system overhead. Below, we’ll explore three key tools to help identify bandwidth "hogs" at the OS level.

1. eBPF: Dynamic Network Traffic Monitoring

Extended Berkeley Packet Filter (eBPF) is a technology that allows real-time kernel program execution without recompilation. It’s particularly useful for network traffic monitoring because it operates at the packet level.

Example of using eBPF to track network connections:

sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter {
    if (args->id == SYS_CONNECT) {
        @[comm] = count();
    }
}'

The output will show which processes (comm) are establishing the most connections. You can also monitor specific ports:

sudo bpftrace -e 'kprobe:tcp_connect {
    @[comm] = count();
}'

Advantages of eBPF:

• Low CPU usage (runs in kernel space).
• Ability to filter traffic at the packet level.
• No need to restart the system.

Limitations:

• Requires admin privileges (sudo).
• eBPF scripts must be adapted for specific kernel versions.

2. nftables and Real-Time Traffic Tracking

nftables is a modern alternative to iptables that offers packet-level network traffic tracking. You can configure rules to monitor specific connections or enable trace mode:

sudo nft monitor trace

The above command will display all nftables rules along with real-time traffic data. To set up a dedicated monitoring rule, use:

sudo nft add table ip monitor
sudo nft add chain ip monitor output { type filter hook output priority 0 \; }
sudo nft add rule ip monitor output ip protocol tcp counter

Advantages of nftables:

• Precise traffic filtering capabilities.
• Low system overhead.
• Integration with system logs.

3. conntrack: Tracking Active Connections

The conntrack tool enables real-time monitoring of active network connections. It’s especially useful for identifying processes that establish many connections or maintain them for extended periods.

Example commands:

• Display all active connections:

sudo conntrack -L

• Filter connections by port (e.g., 443):

sudo conntrack -L -p tcp --dport 443

• Show connection statistics:

sudo conntrack -S

Advantages of conntrack:

• Fast and easy to use.
• Available by default in most distributions.
• Can integrate with system logs.

CLI Tools for Per-Process Traffic Monitoring: What to Choose?

While iftop and nethogs are widely known, there are several other CLI tools that may prove more useful in specific scenarios. Below is a comparison of the most popular solutions available in standard Debian, RHEL, and Arch repositories.

When to Use Which Tool?

• Need a quick traffic overview? Use bmon or nload.
• Want to monitor specific ports or protocols? iftop or nftables.
• Must identify processes consuming bandwidth? tcptrack or ss -tulnp.
• Need historical data? vnstat.

Monitoring in Containers: Why nethogs Fails and How to Fix It

Tools like nethogs or iftop cannot see traffic inside Docker or LXC containers because they operate in separate network namespaces. To monitor container traffic, you’ll need alternative methods.

Network Traffic Monitoring Methods in Containers

1. Monitoring the docker0 interface:

   sudo nethogs docker0

   This shows traffic on the bridge but doesn’t distinguish between individual containers.
2. Using docker stats:

   docker stats --no-stream

   Displays CPU, memory, and network usage at the container level, but not detailed per-process traffic.
3. Entering the container’s network namespace:

   sudo nsenter -t $(docker inspect --format '{{.State.Pid}}' ) -n nethogs eth0

   Allows running nethogs inside the container.
4. Monitoring traffic on veth* interfaces:

   sudo nethogs vethXXX

   Docker uses virtual interfaces veth, which can be monitored directly.

Limitations of Containerization

• No visibility into container-internal traffic for tools running on physical interfaces.
• Complex diagnostics—requiring multiple tools (e.g., docker stats + nsenter).
• Hidden process issues—some processes may run in "rootless" mode, preventing monitoring.

Is libpcap Better Than Kernel Tools? Performance Comparison

Tools based on libpcap (e.g., tcpdump, nethogs) and those using kernel interfaces (e.g., ss, ip) have different strengths and weaknesses, depending on the use case.

libpcap-Based Tools

Advantages:

• Full packet visibility—you can analyze packet headers, useful for advanced diagnostics (e.g., MTU issues, fragmentation).
• Packet-level filtering—e.g., tcpdump 'port 443 and host 1.2.3.4'.
• Multi-protocol support—not just TCP/IP, but also ICMP, ARP, etc.

Disadvantages:

• High CPU usage under heavy traffic—libpcap must capture and analyze every packet.
• No process-level insights unless integrated with proc (e.g., nethogs).
• Vulnerable to traffic hiding—some malicious processes (e.g., rootkits) may evade libpcap.

Tools Based on Kernel Interfaces (ss, ip)

Advantages:

• Speed and low overhead—kernel tools operate directly on kernel data structures without packet capture.
• Process-level insights—ss -tulnp shows which processes use specific ports.
• Stability—less prone to errors than libpcap-based tools.

Disadvantages:

• Limited packet visibility—you won’t see packet header details.
• No packet-level filtering—you must rely on connection filtering, not individual packets.
• Poor handling of certain traffic types—e.g., UDP or multicast may be less effectively monitored.

Summary: If you need quick traffic overviews or process identification, use kernel tools. If you require detailed packet analysis (e.g., protocol debugging), use libpcap.

How to Find "Hidden" Bandwidth-Consuming Processes? Advanced Admin Methods

Some processes—especially those running with elevated privileges or in user space—intentionally hide from standard monitoring tools. To identify them, you’ll need more advanced techniques.

1. Using lsof to Identify Processes by Ports

lsof (List Open Files) lets you display all processes using a specific port:

sudo lsof -i :443 -P -n | awk '{print $1}' | sort | uniq -c

The output will show which processes ($1) are using port 443. You can also filter by protocol:

sudo lsof -i tcp:80 -P -n

2. Monitoring Control Groups (cgroup) with systemd-cgtop

Systemd allows monitoring resource usage by process groups. To check network traffic for a specific cgroup:

systemd-cgtop

Then press n (network) and i (interactive) for a more detailed view. You can also configure cgroup for specific processes:

sudo systemd-run --slice=network-heavy --scope -t -p CPUQuota=50% -p MemoryLimit=1G /ścieżka/do/procesu

3. Logging Network Connections with auditd

auditd is a system activity monitoring tool that can log all network connections of suspicious processes:

sudo auditctl -a exit,always -F arch=b64 -S socket -k network-connection
sudo cat /var/log/audit/audit.log | grep network-connection

You can also configure rules to log specific processes:

sudo auditctl -w /ścieżka/do/procesu -p x -k suspicious-process

4. Using strace for Process Activity Monitoring

strace lets you track all system calls made by a process, including network operations:

sudo strace -p PID_PROCESU -e trace=network

This will reveal all network operations performed by the process, helping identify suspicious activity.

Automating Network Traffic Monitoring: Scripts, Cron, and Alert Systems

To effectively monitor network traffic, manual tool checks aren’t enough. Admins should implement automation to:

• Generate regular traffic reports.
• Detect anomalies and send alerts.
• Perform long-term trend analysis.

1. Traffic Monitoring Scripts

Below is a sample Bash script that monitors interface traffic and sends an alert if it exceeds a threshold:

#!/bin/bash

INTERFACE="eth0"
THRESHOLD=1000000  # 1 Mbps
LOG_FILE="/var/log/network_monitor.log"

# Pobierz ruch w ciągu ostatnich 5 minut
RX_BYTES=$(cat /sys/class/net/$INTERFACE/statistics/rx_bytes)
TX_BYTES=$(cat /sys/class/net/$INTERFACE/statistics/tx_bytes)

# Oblicz ruch w Mbps (przy założeniu stałej prędkości interfejsu)
RX_MBPS=$(echo "scale=2; $RX_BYTES * 8 / 1000000 / 300" | bc)
TX_MBPS=$(echo "scale=2; $TX_BYTES * 8 / 1000000 / 300" | bc)

# Zapisz do loga
echo "$(date) - RX: $RX_MBPS Mbps, TX: $TX_MBPS Mbps" >> $LOG_FILE

# Sprawdź próg
if (( $(echo "$RX_MBPS > $THRESHOLD" | bc -l) )) || (( $(echo "$TX_MBPS > $THRESHOLD" | bc -l) )); then
    echo "ALERT: Wysoki ruch sieciowy na $INTERFACE!" | mail -s "Alert sieciowy" admin@example.com
fi

To run the script every 5 minutes, add it to cron:

*/5 * * * * /ścieżka/do/skryptu.sh

2. Integration with Prometheus and Grafana

If you use Prometheus for monitoring, you can configure a network exporter (node_exporter) to collect network traffic statistics:

Example prometheus.yml configuration:

scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

Then configure Grafanę to visualize the data:

1. Add a panel with the query:

rate(node_network_receive_bytes_total[5m]) * 8 / 1000000

2. Set up an alert in Prometheus Alertmanager:

- alert: HighBandwidthUsage
  expr: rate(node_network_receive_bytes_total[5m]) > 1e8  # 100 Mbps
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Wysoki ruch sieciowy na {{ $labels.instance }}"

3. Using Netdata for Real-Time Monitoring

Netdata is a lightweight monitoring agent that automatically collects network traffic statistics and visualizes them in real time. Installation is straightforward:

bash <(curl -Ss https://my-netdata.io/kickstart.sh)

After installation, Netdata will automatically configure dashboards for network traffic, bandwidth usage, and active connections.

Summary: How to Effectively Monitor Network Traffic in Linux

Diagnosing bandwidth issues in Linux requires combining modern kernel tools (eBPF, nftables) with classic monitoring methods. The key is:

• Identifying bandwidth "hogs"—use ss, tcptrack, or eBPF to track processes.
• Monitoring in containers—use docker stats, nsenter, or veth interfaces.
• Automating the process—implement scripts, cron, or systems like Prometheus + Grafana.
• Detecting hidden processes—use auditd, lsof, or strace.

Remember: every tool has its limitations—the best results come from combining multiple methods. If network issues are critical, consider a security audit to rule out rootkits or attacks.

Sources

• https://www.tecmint.com/find-process-using-bandwidth-linux/
• https://www.kernel.org/doc/html/latest/networking/index.html
• https://www.kernel.org/doc/html/latest/bpf/index.html
• https://wiki.nftables.org/
• https://www.tcpdump.org/manpages/pcap.3pcap.html
• https://manpages.debian.org/testing/iproute2/ss.8.en.html
• https://docs.docker.com/network/
• https://github.com/netdata/netdata
• https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_audit_system/audit-filtering-configuration_configuring-audit-system
• https://prometheus.io/docs/alerting/latest/overview/