Student Manual — Module 4: System Resource Monitoring and Optimization
Student Manual

# Linux Administration Essentials

## Module 4: System Resource Monitoring and Optimization

LNX-001

1-day course · 5 modules · beginner level

### Learning Objectives

After completing this module, you will be able to:

- Monitor system resources using command-line utilities
- Identify and manage resource-intensive processes
- Analyze disk usage and optimize storage allocation
- Interpret system logs for performance troubleshooting

# Lesson 1: Understanding System Resources and the Linux Process Model

## Understanding System Resources and the Linux Process Model

Linux system administrators must understand how their systems allocate and consume resources. Every running program on a Linux system becomes a process—a discrete unit of execution with its own memory space, CPU time allocation, and system resources. Understanding the process model is fundamental to effective system monitoring and optimization.

### Core System Resources

Linux systems manage four primary resource categories:

- CPU (Central Processing Unit): The computational power available for executing instructions. Modern systems have multiple cores, allowing parallel process execution.
- Memory (RAM): Volatile storage where active processes store data and instructions. When memory is exhausted, the system uses swap space on disk, significantly degrading performance.
- Disk I/O: The rate at which data is read from or written to storage devices. Slow disk operations can create bottlenecks even when CPU and memory are available.
- Network I/O: Bandwidth consumed by network communications. Network-intensive applications can saturate available bandwidth.

### The Process Lifecycle

When you execute a command or launch an application, the system creates a new process with a unique Process ID (PID). Each process has a parent process (except init/systemd, PID 1), creating a hierarchical tree structure. Processes transition through states: running (actively using CPU), sleeping (waiting for resources or events), stopped (paused by signals), or zombie (terminated but not yet cleaned up by parent).

### Why Monitoring Matters

Effective resource monitoring enables administrators to identify performance bottlenecks before they impact users, plan capacity upgrades based on trends, troubleshoot application issues, and optimize resource allocation. Without monitoring, problems often go unnoticed until system performance degrades severely or services fail entirely.

Modern Linux systems provide robust built-in tools for real-time monitoring and historical analysis, making proactive system management achievable without expensive third-party solutions.

Illustrated overview showing the four key system resources (CPU as a processor chip, RAM as memory modules, disk storage as a drive array, and network as connected cables) with processes flowing between them like a circulatory system

Key Takeaways

- Processes are the fundamental unit of execution in Linux with unique PIDs
- Four primary resources require monitoring: CPU, memory, disk I/O, and network I/O
- Processes move through lifecycle states: running, sleeping, stopped, and zombie
- Proactive monitoring prevents performance issues and enables capacity planning

# Lesson 2: Process Monitoring with top, htop, and ps Commands

## Process Monitoring with top, htop, and ps Commands

Linux provides several powerful command-line utilities for monitoring active processes. Each tool offers different perspectives on system activity, and mastering all three gives administrators comprehensive visibility into system performance.

### The top Command: Real-Time Dashboard

The `top` command provides a dynamic, real-time view of running processes. When you launch top, you see a continuously updating display showing system summary information (uptime, load averages, CPU states, memory usage) at the top, followed by a process list sorted by CPU usage by default.

Key metrics in the top display include:

- Load average: Three numbers representing average system load over 1, 5, and 15 minutes. Values below the number of CPU cores indicate healthy system load.
- CPU percentages: Time spent in user processes (us), system/kernel operations (sy), idle time (id), and waiting for I/O (wa). High wa values indicate disk bottlenecks.
- Memory statistics: Total, used, free, and cached memory. Linux aggressively caches data, so low "free" memory with high "cached" values is normal and healthy.

Interactive commands within top allow you to change the sort order (press M for memory, P for CPU), kill processes (press k and enter PID), and adjust display options.

### The htop Command: Enhanced Interactive Monitor

The `htop` command (often requiring separate installation) provides a more user-friendly, color-coded interface than top. It displays individual CPU cores graphically, uses intuitive function key commands, allows mouse interaction for selecting processes, and provides a tree view showing parent-child process relationships.

The visual presentation makes htop excellent for quickly identifying resource hogs and understanding process hierarchies.

### The ps Command: Process Snapshots

Unlike top and htop's continuous updates, `ps` provides a snapshot of current processes. The command `ps aux` shows all processes with detailed information in a format perfect for scripting and filtering.

Common ps usage patterns:

    # Show all processes with full details
    ps aux

    # Find specific processes using grep
    ps aux | grep apache

    # Show process tree structure
    ps auxf

    # Custom output format
    ps -eo pid,ppid,cmd,%cpu,%mem --sort=-%cpu | head -10

The ps command excels in automation scenarios where you need to programmatically check process states or extract specific information for scripts.

Illustrated comparison showing three monitoring scenarios: a dashboard with live gauges and meters for top, an enhanced color-coded control panel with visual graphs for htop, and a snapshot photograph with annotations for ps

Key Takeaways

- top provides real-time continuous monitoring with interactive controls
- htop offers enhanced visualization with color coding and tree views
- ps generates process snapshots ideal for scripting and filtering
- Load averages indicate overall system demand relative to available CPU cores

# Lesson 3: CPU and Memory Utilization Analysis

## CPU and Memory Utilization Analysis

Accurately interpreting CPU and memory metrics is essential for diagnosing performance problems and making informed optimization decisions. Misunderstanding these metrics can lead to unnecessary hardware purchases or missed optimization opportunities.

### CPU Utilization Deep Dive

CPU usage is divided into several categories that reveal what your system is doing:

- User time (us): CPU cycles spent executing user-space applications. High user time with good performance is normal; it means your system is doing productive work.
- System time (sy): CPU cycles spent in kernel operations like file system access, networking, and device drivers. Extremely high system time (>30%) may indicate kernel-level problems or excessive system calls.
- I/O wait (wa): Time CPU spends idle while waiting for disk or network I/O to complete. High I/O wait (>20%) indicates storage bottlenecks, not CPU problems—faster disks or storage optimization will help, not more CPU.
- Idle time (id): Unused CPU capacity. Consistently low idle time (<5%) suggests you may need more CPU resources or should optimize applications.

### Load Average Interpretation

Load averages show the number of processes waiting for CPU time. On a single-core system, a load average of 1.0 means full utilization. On a four-core system, load averages below 4.0 indicate available capacity.

Compare the three load values (1, 5, and 15 minute) to identify trends: rising values indicate increasing demand, falling values suggest load is decreasing, and consistent values indicate stable workload.

### Memory Usage Patterns

Linux memory management differs from Windows and can confuse newcomers. The kernel uses unused RAM for caching disk data, dramatically improving performance. This means "used" memory includes both active applications and disk cache.

Key memory concepts:

- Active memory: Currently in use by processes and cannot be reclaimed without impacting performance.
- Cached memory: Disk data stored in RAM for fast access. The kernel releases this instantly if applications need it.
- Swap usage: Disk space used as virtual memory. Some swap usage is normal, but high swap activity (swapping) causes severe performance degradation.

### Identifying Memory Pressure

True memory problems exhibit specific symptoms: consistently high swap usage with active swapping, low or zero cache values (indicating memory shortage), and the OOM (Out of Memory) killer terminating processes. If you have low "free" memory but high "cached" values and minimal swap usage, your system is healthy and efficiently using available RAM.

Monitor the rate of swap in/out (si/so in top) rather than total swap usage. Zero or minimal swap activity with some swap usage indicates the kernel moved old, inactive pages to disk—this is fine and normal.

Illustrated infographic showing CPU time divided into colorful segments (user, system, I/O wait, idle) and memory represented as containers with different fill levels and labels showing active, cached, and swap areas

Key Takeaways

- High I/O wait indicates storage bottlenecks, not CPU problems
- Load averages should be interpreted relative to number of CPU cores
- Linux uses free RAM for disk caching, so low free memory is often normal
- Monitor swap activity (si/so) rather than just total swap usage to identify memory pressure

# Lesson 4: Process Priority and Nice Values

## Process Priority and Nice Values

Linux provides mechanisms for controlling how the CPU scheduler allocates time to processes. Understanding process priority enables administrators to ensure critical services receive adequate resources while preventing low-priority tasks from impacting system performance.

### The Nice Value System

The "nice" value determines a process's CPU scheduling priority. The name comes from the idea that a process with a higher nice value is "nicer" to other processes by yielding CPU time more readily.

Nice values range from -20 (highest priority) to +19 (lowest priority):

- -20 to -1: High priority, receives more CPU time. Requires root privileges to set.
- 0: Default priority for normal user processes.
- 1 to 19: Lower priority, receives less CPU time. Any user can reduce their own process priority.

The kernel's CPU scheduler uses nice values to calculate process priority but also considers other factors like I/O patterns and historical CPU usage.

### Setting Process Priority

Launch a process with a specific nice value using the `nice` command:

    # Start a backup script with low priority
    nice -n 19 ./backup-script.sh

    # Start a critical service with high priority (requires root)
    sudo nice -n -10 /usr/local/bin/critical-service

Change the nice value of a running process using `renice`:

    # Lower priority of process 1234
    renice -n 10 -p 1234

    # Raise priority of all processes owned by user apache (requires root)
    sudo renice -n -5 -u apache

### Practical Applications

Common scenarios for adjusting process priority include:

- Batch jobs and backups: Set nice value to 15-19 so they run during idle periods without impacting interactive users.
- Background maintenance: Database optimization, log rotation, and system updates can run with lower priority.
- Critical services: Web servers, databases, and monitoring agents might receive slightly elevated priority (-5 to -10) to ensure responsiveness.
- Resource-intensive compilations: Development builds and CI/CD pipelines can run with reduced priority to avoid disrupting other work.

### Real-Time Priority

Beyond nice values, Linux supports real-time scheduling policies (SCHED_FIFO and SCHED_RR) for processes requiring deterministic timing. Real-time priorities range from 1 to 99 and completely override nice values. These are used for specialized applications like industrial control systems, audio processing, and scientific equipment control.

Most system administrators never need real-time priorities. Incorrectly configured real-time processes can monopolize the CPU and make systems unresponsive. Use real-time scheduling only when you understand its implications and have specific timing requirements.

Illustrated scene showing processes as runners on a multi-lane track, with different colored lanes representing priority levels from -20 (express lane) to +19 (slow lane), and a scheduler figure directing traffic

Key Takeaways

- Nice values range from -20 (highest priority) to +19 (lowest priority)
- Only root can increase process priority (set negative nice values)
- Use nice to set priority at process launch, renice to adjust running processes
- Real-time priorities exist but should be used only for specialized applications

# Lesson 5: Process Termination with Kill Signals

## Process Termination with Kill Signals

Administrators frequently need to stop misbehaving processes, restart services, or terminate unresponsive applications. Linux provides a sophisticated signaling system for process control, with the `kill` command as the primary interface.

### Understanding Signals

Signals are software interrupts delivered to processes, notifying them of events or requesting specific actions. Despite the name "kill," signals serve many purposes beyond termination. Each signal has a number and symbolic name.

Common signals administrators use:

- SIGTERM (15): Polite termination request. The default kill signal. Allows processes to clean up, save data, and exit gracefully.
- SIGKILL (9): Immediate termination. Cannot be caught or ignored. The kernel stops the process instantly without allowing cleanup. Use only when SIGTERM fails.
- SIGHUP (1): Historically "hang up," now commonly used to tell daemons to reload configuration files without restarting.
- SIGSTOP (19): Pause a process. Cannot be caught or ignored.
- SIGCONT (18): Resume a paused process.
- SIGINT (2): Interrupt signal, equivalent to pressing Ctrl+C. Requests termination.

### Using the kill Command

The basic kill syntax is: `kill [signal] PID`

    # Gracefully terminate process 1234
    kill 1234
    kill -15 1234
    kill -TERM 1234  # All three are equivalent

    # Force immediate termination if process won't respond
    kill -9 1234
    kill -KILL 1234

    # Reload service configuration
    kill -HUP 1234
    kill -1 1234

    # Pause and resume a process
    kill -STOP 1234
    kill -CONT 1234

### Best Practices for Process Termination

Always try SIGTERM first. This allows applications to:

- Flush buffered data to disk
- Close open files and network connections properly
- Release locks and shared resources
- Save state information
- Notify dependent processes

Wait 5-10 seconds after sending SIGTERM before escalating to SIGKILL. Most well-behaved processes terminate quickly when asked politely.

Use SIGKILL (9) only when necessary. Forceful termination can cause:

- Data loss or corruption
- Orphaned child processes
- Unreleased locks that block other processes
- Incomplete transactions
- Temp files left behind

### Alternative Tools

The `pkill` command kills processes by name rather than PID:

    # Terminate all processes named firefox
    pkill firefox

    # Terminate all apache processes owned by www-data
    pkill -u www-data apache

The `killall` command also terminates by name but requires an exact match, while pkill accepts partial matches and patterns.

Illustrated sequence showing three scenes: first, a process receiving a polite SIGTERM message and cleaning up gracefully; second, an unresponsive process ignoring the message; third, SIGKILL depicted as an emergency stop that immediately halts the process

Key Takeaways

- SIGTERM (15) requests graceful termination and should always be tried first
- SIGKILL (9) forces immediate termination without cleanup—use only when necessary
- SIGHUP (1) commonly tells services to reload configuration without restarting
- pkill and killall terminate processes by name rather than PID

# Lesson 6: Disk Space Monitoring with df and du Commands

## Disk Space Monitoring with df and du Commands

Disk space exhaustion causes system failures, application errors, and data loss. Proactive disk space monitoring is a fundamental administrative responsibility. Linux provides the `df` and `du` commands for analyzing disk usage at different levels.

### The df Command: File System Overview

The `df` (disk free) command reports file system disk space usage. It shows how much space is used and available on each mounted file system.

Common df usage:

    # Human-readable output with sizes in GB/MB
    df -h

    # Show inode usage instead of block usage
    df -i

    # Display specific file system type
    df -h -t ext4

Key df output columns:

- Filesystem: Device name or network path
- Size: Total capacity
- Used: Currently consumed space
- Available: Space available for use
- Use%: Percentage of capacity used
- Mounted on: Directory where the file system is accessible

### Critical Thresholds

Monitor these warning signs:

- 80% used: Plan for cleanup or expansion
- 90% used: Take immediate action—some applications may begin failing
- 100% used: System instability, log rotation fails, applications crash

Some space remains "reserved" for root on ext4 file systems (typically 5%), preventing complete exhaustion from impacting system administration tasks.

### The du Command: Directory Usage Analysis

While df shows file system totals, `du` (disk usage) analyzes specific directories to identify what's consuming space.

Common du usage:

    # Show size of current directory and all subdirectories
    du -h

    # Show only total size of current directory
    du -sh

    # Show size of immediate subdirectories only
    du -h --max-depth=1

    # Sort results by size
    du -h | sort -h

    # Find the 10 largest directories
    du -h --max-depth=2 | sort -hr | head -10

### Common Space Consumption Areas

When investigating disk usage, check these typical problem areas:

- /var/log: Log files grow indefinitely without rotation. Old logs may consume gigabytes.
- /tmp: Temporary files from applications that crashed or didn't clean up.
- /home: User files, downloads, and caches.
- /var/cache: Package manager caches and application temporary data.
- /var/spool: Mail queues, print jobs, and cron job output.

### Inode Exhaustion

File systems have a limited number of inodes (index nodes), which store file metadata. You can exhaust inodes while having free disk space if you create millions of small files.

Symptoms of inode exhaustion:

- "No space left on device" errors despite df showing available space
- `df -i` shows 100% inode usage
- Cannot create new files even when deleting existing files doesn't free inodes

Fix inode exhaustion by deleting files, not just clearing their contents. Common causes include mail queues with millions of messages and cache directories with excessive small files.

Illustrated split-screen showing df command represented as a fuel gauge or dashboard meter displaying overall file system capacity levels, and du command represented as a magnifying glass examining individual folders and directories with varying sizes

Key Takeaways

- df shows file system level usage, du analyzes directory-level consumption
- Monitor file systems and take action before reaching 90% capacity
- Common space consumption areas include /var/log, /tmp, /home, and /var/cache
- Inode exhaustion can prevent file creation even with available disk space

# Lesson 7: System Log Analysis Using journalctl

## System Log Analysis Using journalctl

System logs provide the diagnostic information administrators need to troubleshoot problems, track security events, and understand system behavior. Modern Linux distributions use systemd's journal for centralized logging, accessed through the `journalctl` command.

### The systemd Journal

The systemd journal collects logs from multiple sources:

- Kernel messages
- System services managed by systemd
- Application output
- Authentication events
- Hardware events

Unlike traditional syslog files scattered across /var/log, the journal stores everything in a structured, indexed binary format that enables efficient querying and filtering.

### Basic journalctl Usage

    # View all journal entries (oldest first)
    journalctl

    # View newest entries first, with paging
    journalctl -r

    # Follow new log entries in real-time (like tail -f)
    journalctl -f

    # Show only entries from current boot
    journalctl -b

    # Show entries from previous boot
    journalctl -b -1

### Filtering by Priority

Logs have priority levels (also called severity):

- emerg (0): System unusable
- alert (1): Immediate action required
- crit (2): Critical conditions
- err (3): Error conditions
- warning (4): Warning conditions
- notice (5): Normal but significant
- info (6): Informational messages
- debug (7): Debug messages

Filter by priority:

    # Show only errors and higher priority (crit, alert, emerg)
    journalctl -p err

    # Show warnings and higher
    journalctl -p warning

### Filtering by Service and Time

    # Show logs for specific service
    journalctl -u sshd.service
    journalctl -u apache2.service

    # Show logs since specific time
    journalctl --since "2024-01-15 14:00:00"
    journalctl --since "1 hour ago"
    journalctl --since yesterday

    # Show logs in time range
    journalctl --since "2024-01-15" --until "2024-01-16"

    # Combine filters
    journalctl -u nginx.service --since "10 minutes ago" -p err

### Output Formatting

    # Verbose output showing all available fields
    journalctl -o verbose

    # JSON format for parsing
    journalctl -o json-pretty

    # Short format (traditional syslog style)
    journalctl -o short

### Troubleshooting Workflow

When investigating system issues:

1. Identify the timeframe: When did the problem occur?
2. Check error messages: `journalctl -p err --since "[timeframe]"`
3. Focus on relevant services: `journalctl -u [service-name] --since "[timeframe]"`
4. Look for patterns: Use grep to find repeated messages
5. Check previous boots: If the system crashed, examine logs from the failed boot

### Journal Maintenance

The journal can consume significant disk space. Manage journal size:

    # Show current journal disk usage
    journalctl --disk-usage

    # Remove old entries, keeping only last 2 days
    sudo journalctl --vacuum-time=2d

    # Limit journal to 500MB
    sudo journalctl --vacuum-size=500M

Configure persistent limits in `/etc/systemd/journald.conf` by setting SystemMaxUse to cap total journal size.

Illustrated library or archive room with organized filing systems, color-coded sections for different log priorities (red for errors, yellow for warnings, blue for info), a timeline on the wall, and a researcher examining specific entries with filters and search tools

Key Takeaways

- journalctl provides unified access to all systemd journal logs
- Filter logs by priority level (-p), service (-u), and time (--since/--until)
- Priority levels range from emerg (0) to debug (7), with err (3) being a common troubleshooting starting point
- Use --vacuum-time or --vacuum-size to manage journal disk consumption

# Lesson 8: Performance Baseline Establishment and Trending

## Performance Baseline Establishment and Trending

Effective performance management requires understanding what "normal" looks like for your systems. Without baselines, you cannot distinguish between typical behavior and actual problems. Establishing performance baselines and tracking trends enables proactive capacity planning and rapid anomaly detection.

### What Is a Performance Baseline?

A performance baseline is a documented snapshot of system metrics under typical operating conditions. It includes:

- Average and peak CPU utilization during business hours and off-hours
- Memory usage patterns including cache behavior
- Disk I/O rates and latency measurements
- Network throughput and connection counts
- Typical process counts and types
- Application-specific metrics (web requests per second, database query times)

Baselines answer the question: "What does good look like?" When investigating performance issues, comparing current metrics to baseline values quickly reveals deviations.

### Establishing Your Baseline

Create baselines during normal operations, not during problems:

1. Identify representative periods: Capture metrics during typical workload periods. For business systems, this means normal business hours with average user activity.
2. Monitor for sufficient duration: Collect data for at least one complete business cycle (a full week including weekends) to capture patterns.
3. Document conditions: Note the software versions, user count, and workload characteristics during baseline establishment.
4. Capture multiple metrics: Record CPU, memory, disk, network, and application metrics simultaneously.

Simple baseline collection:

    # Log system metrics every 60 seconds for one week
    while true; do
      date >> /var/log/baseline.log
      top -bn1 | head -20 >> /var/log/baseline.log
      df -h >> /var/log/baseline.log
      sleep 60
    done

For production systems, use dedicated monitoring tools like Prometheus, Grafana, Nagios, or cloud provider monitoring services.

### Recognizing Normal Patterns

Healthy systems exhibit predictable patterns:

- Daily cycles: CPU and memory usage peak during business hours, drop overnight
- Weekly cycles: Weekday patterns differ from weekends
- Gradual growth: Resource usage increases slowly as business grows
- Event-driven spikes: Expected increases during backups, reports, batch jobs

### Anomaly Detection

Compare current metrics to baselines to identify problems:

- Sudden spikes: Unexpected load increase may indicate application issues, traffic surges, or attacks
- Sustained elevation: Gradual increases that don't return to baseline suggest resource leaks or capacity limits
- Missing patterns: Expected cycles that don't occur may indicate service failures
- Changed ratios: Shifts in resource proportions (like I/O wait suddenly dominating CPU time) signal problems

### Capacity Planning

Baselines and trends enable proactive capacity planning:

1. Plot historical growth: Graph resource utilization over months
2. Calculate growth rate: Determine percentage increase per month
3. Project future needs: Extrapolate when you'll reach capacity limits
4. Plan upgrades: Schedule hardware or infrastructure expansion before problems occur

For example, if memory usage grows 5% monthly and you're at 60% utilization, you have approximately eight months before reaching 100%. This allows plenty of time to plan and execute upgrades.

### Continuous Monitoring

Baseline establishment isn't a one-time task. Update baselines:

- After major application changes or deployments
- When infrastructure is upgraded
- After significant workload changes
- Periodically (quarterly or semi-annually) as part of regular maintenance

Maintaining current baselines ensures your performance management remains effective as your environment evolves.

Illustrated before-and-after comparison showing a system administrator consulting a historical performance map or chart with normal patterns in one panel, then using that reference to identify an anomaly on a current monitoring display in the second panel

Key Takeaways

- Performance baselines document normal system behavior for comparison during troubleshooting
- Establish baselines during typical operations over a complete business cycle
- Recognize normal patterns including daily cycles, weekly variations, and gradual growth
- Use baseline trends for capacity planning to anticipate and prevent resource exhaustion

# Lab: Monitoring and Managing System Resources

Objective: Monitor system resources, identify resource-intensive processes, manage process priorities, analyze disk usage, and interpret system logs to optimize Linux system performance

Estimated Time: 45 minutes

Platform: Ubuntu Linux VM on VirtualBox (Windows host)

Prerequisites:

- Ubuntu Linux VM running in VirtualBox
- Terminal access with sudo privileges
- Basic command-line navigation skills

## Procedures

### Step 1: Open the terminal and create a baseline snapshot of current system resource usage

Launch the terminal application in Ubuntu by pressing `Ctrl+Alt+T` or clicking the terminal icon in the application menu.

Run the following command to display current system resource usage:

    top -bn1 | head -20

The `-b` flag runs top in batch mode, `-n1` specifies one iteration, and we pipe to `head -20` to see the first 20 lines of output.

Expected Output:

You should see output similar to:

    top - 14:23:15 up 2:34, 1 user, load average: 0.15, 0.22, 0.18
    Tasks: 187 total,   1 running, 186 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  2.3 us,  1.1 sy,  0.0 ni, 96.4 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 st
    MiB Mem :   3924.5 total,   1342.8 free,   1234.2 used,   1347.5 buff/cache
    MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.   2456.7 avail Mem

      PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
     1234 student   20   0 2847256 234512  98432 S   1.2   5.8   0:45.23 gnome-shell
     5678 student   20   0  789432  87654  45678 S   0.3   2.2   0:12.45 firefox

✔ Verification: Verify that the load average values are displayed and the process list shows running processes with their CPU and memory usage percentages.

### Step 2: Install htop for enhanced process monitoring capabilities

Install the `htop` utility, which provides a more user-friendly and interactive process monitoring interface:

    sudo apt update
    sudo apt install htop -y

The `-y` flag automatically answers yes to installation prompts.

Expected Output:

    Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
    Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB]
    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    The following NEW packages will be installed:
      htop
    0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
    Need to get 92.7 kB of archives.
    After this operation, 251 kB of additional disk space will be used.
    Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 htop amd64 3.0.5-7build2 [92.7 kB]
    Fetched 92.7 kB in 1s (92.7 kB/s)
    Selecting previously unselected package htop.
    Unpacking htop (3.0.5-7build2) ...
    Setting up htop (3.0.5-7build2) ...

✔ Verification: Confirm htop is installed by running `which htop`. This should return `/usr/bin/htop`.

### Step 3: Launch htop and explore the interactive process monitoring interface

Start htop with the following command:

    htop

Once htop is running:

- Observe the CPU usage bars at the top (one per CPU core)
- Note the memory and swap usage bars
- Use the arrow keys to navigate through the process list
- Press `F6` to access sort options, then select `PERCENT_CPU` to sort by CPU usage
- Press `F10` or `q` to quit htop when finished exploring

Expected Output:

An interactive, colorful display showing:

- CPU cores numbered 1-N with usage bars
- Memory bar showing used/total memory in green/blue
- Swap bar showing swap usage
- Process list with columns: PID, USER, PRI, NI, VIRT, RES, SHR, S, CPU%, MEM%, TIME+, Command
- Function key menu at the bottom (F1-F10)

✔ Verification: You should be able to navigate the process list with arrow keys and see real-time updates of CPU and memory usage. Press `q` to exit.

### Step 4: Create a test process that consumes CPU resources for monitoring practice

Open a second terminal window (press `Ctrl+Alt+T` again) and create a CPU-intensive process using this command:

    yes > /dev/null &

This command runs the `yes` command (which outputs 'y' continuously) in the background, redirecting output to `/dev/null`. The `&` symbol runs it as a background process.

Note the process ID (PID) that is displayed after running the command.

Expected Output:

    [1] 12345

The number in brackets is the job number, and the second number (12345 in this example) is the process ID (PID).

✔ Verification: Run `jobs` to confirm the process is running in the background. You should see output like `[1]+ Running yes > /dev/null &`

### Step 5: Use the ps command to identify the CPU-intensive process

Return to your first terminal and use the `ps` command with sorting options to find processes consuming the most CPU:

    ps aux --sort=-%cpu | head -10

Breakdown of the command:

- `aux` shows all processes with detailed information
- `--sort=-%cpu` sorts by CPU usage (descending, highest first)
- `head -10` displays only the top 10 results

Expected Output:

    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    student  12345 99.8  0.0   6480   712 pts/1    R    14:30   0:45 yes
    root      1234  1.2  2.3 234567 98765 ?        Ssl  12:00   1:23 /usr/lib/systemd/systemd
    student   5678  0.8  4.5 876543 123456 ?       Sl   12:05   0:34 /usr/bin/gnome-shell

Your `yes` process should appear at or near the top with close to 100% CPU usage.

✔ Verification: Verify that the `yes` process appears in the output with high CPU percentage (90-100%) and matches the PID from step 4.

### Step 6: Modify the process priority using the renice command

Lower the priority of the CPU-intensive process to reduce its impact on system performance. Replace `12345` with your actual PID from step 4:

    sudo renice -n 10 -p 12345

The `-n 10` sets the nice value to 10 (lower priority). Nice values range from -20 (highest priority) to 19 (lowest priority). Normal processes start with nice value 0.

Verify the change with:

    ps -o pid,ni,cmd -p 12345

Expected Output:

Renice output:

    12345 (process ID) old priority 0, new priority 10

Verification output:

      PID  NI CMD
    12345  10 yes

✔ Verification: The NI (nice) column should show 10 instead of 0, confirming the priority has been lowered.

### Step 7: Terminate the test process using the kill command

Stop the CPU-intensive process using the `kill` command. Replace `12345` with your actual PID:

    kill 12345

By default, `kill` sends the SIGTERM signal (signal 15), which requests graceful termination.

Verify the process has stopped:

    ps -p 12345

Expected Output:

The first command produces no output if successful.

The verification command should show:

      PID TTY          TIME CMD

Or an error message:

    error: process ID list syntax error

Both indicate the process no longer exists.

✔ Verification: Run `jobs` in the terminal where you started the process. You should see `[1]+ Terminated yes > /dev/null` or no output at all.

### Step 8: Analyze disk space usage across all mounted filesystems

Use the `df` command to display disk space usage for all mounted filesystems:

    df -h

The `-h` flag displays sizes in human-readable format (KB, MB, GB).

For more detailed information including filesystem type:

    df -hT

The `-T` flag adds the filesystem type column.

Expected Output:

    Filesystem     Type      Size  Used Avail Use% Mounted on
    tmpfs          tmpfs     393M  1.5M  392M   1% /run
    /dev/sda1      ext4       25G  8.2G   16G  35% /
    tmpfs          tmpfs     2.0G     0  2.0G   0% /dev/shm
    tmpfs          tmpfs     5.0M  4.0K  5.0M   1% /run/lock
    /dev/sda2      ext4      100G   45G   50G  48% /home
    tmpfs          tmpfs     393M  112K  393M   1% /run/user/1000

✔ Verification: Verify that your root filesystem (/) and other partitions are displayed with usage percentages below 90%.

### Step 9: Identify large directories consuming disk space in your home directory

Use the `du` command to analyze disk usage and find the largest directories:

    du -h --max-depth=1 ~ | sort -hr | head -10

Command breakdown:

- `du -h` shows disk usage in human-readable format
- `--max-depth=1` limits the scan to one level deep from home directory
- `~` represents your home directory
- `sort -hr` sorts results in human-readable format, reverse order (largest first)
- `head -10` shows top 10 results

Expected Output:

    4.5G    /home/student
    2.1G    /home/student/Downloads
    1.2G    /home/student/.cache
    892M    /home/student/Documents
    324M    /home/student/Pictures
    156M    /home/student/.local
    89M     /home/student/Videos
    45M     /home/student/Desktop
    23M     /home/student/.config
    12M     /home/student/Music

✔ Verification: The output should show directories sorted by size with the total home directory size at the top.

### Step 10: Create a test directory structure to practice disk usage analysis

Create a test directory with some files to practice disk analysis:

    mkdir -p ~/lab_test/large_files
    cd ~/lab_test/large_files

Create three test files of different sizes:

    dd if=/dev/zero of=file1.dat bs=1M count=50
    dd if=/dev/zero of=file2.dat bs=1M count=100
    dd if=/dev/zero of=file3.dat bs=1M count=25

Each `dd` command creates a file filled with zeros. The `bs=1M` sets block size to 1 megabyte, and `count` determines how many blocks to write.

Expected Output:

For each dd command:

    50+0 records in
    50+0 records out
    52428800 bytes (52 MB, 50 MiB) copied, 0.123456 s, 425 MB/s

The exact speed will vary based on your VM's disk performance.

✔ Verification: Run `ls -lh` in the ~/lab_test/large_files directory. You should see three files: file1.dat (50M), file2.dat (100M), and file3.dat (25M).

### Step 11: Use du to find the total size of the test directory and individual files

Analyze the test directory with detailed file-level information:

    du -ah ~/lab_test | sort -hr

The `-a` flag includes files (not just directories) in the output.

To see only the total size of the directory:

    du -sh ~/lab_test

The `-s` flag provides a summary (total only).

Expected Output:

First command output:

    175M    /home/student/lab_test
    175M    /home/student/lab_test/large_files
    100M    /home/student/lab_test/large_files/file2.dat
    50M     /home/student/lab_test/large_files/file1.dat
    25M     /home/student/lab_test/large_files/file3.dat

Second command output:

    175M    /home/student/lab_test

✔ Verification: The total size should be approximately 175M (50M + 100M + 25M).

### Step 12: Monitor system memory usage with the free command

Check current memory and swap usage:

    free -h

For continuous monitoring that updates every 2 seconds:

    free -h -s 2

Press `Ctrl+C` to stop the continuous monitoring.

The `-s 2` flag sets the update interval to 2 seconds.

Expected Output:

                  total        used        free      shared  buff/cache   available
    Mem:          3.8Gi       1.2Gi       1.3Gi        89Mi       1.3Gi       2.4Gi
    Swap:         2.0Gi          0B       2.0Gi

Your values will differ based on your VM's configuration and current usage.

✔ Verification: Verify that the 'total' memory matches your VM's allocated RAM and that the 'available' memory shows memory that can be used for new processes.

### Step 13: View system logs related to resource usage and system events

Use `journalctl` to view recent system logs:

    sudo journalctl -n 50 --no-pager

The `-n 50` shows the last 50 log entries, and `--no-pager` displays output directly without pagination.

To filter logs for a specific time period (last hour):

    sudo journalctl --since "1 hour ago" --no-pager

To view kernel messages only:

    sudo journalctl -k -n 30 --no-pager

Expected Output:

    Dec 15 14:35:42 ubuntu systemd[1]: Started Session 3 of user student.
    Dec 15 14:35:45 ubuntu kernel: [12345.678901] CPU0: Core temperature above threshold
    Dec 15 14:36:12 ubuntu systemd[1]: Reached target Network is Online.
    Dec 15 14:36:15 ubuntu snapd[1234]: 2023/12/15 14:36:15.123456 autorefresh.go:123: auto-refresh

Log entries include timestamp, hostname, service, and message content.

✔ Verification: You should see timestamped log entries from various system services. Look for entries from systemd, kernel, and other system components.

### Step 14: Search system logs for specific error patterns or priority levels

View only error-level and critical messages from the logs:

    sudo journalctl -p err -n 20 --no-pager

The `-p err` filters for priority 'err' and higher (err, crit, alert, emerg).

To search for specific keywords (e.g., "memory" or "disk"):

    sudo journalctl --grep="memory|disk" --since "1 day ago" --no-pager

The `--grep` option uses regular expressions to search log content.

Expected Output:

For error logs:

    Dec 15 13:22:15 ubuntu kernel: [  456.789012] sd 2:0:0:0: [sda] Sense Key : Medium Error [current]
    Dec 15 14:10:33 ubuntu systemd[1]: Failed to start Some Service.

For keyword search:

    Dec 15 14:15:22 ubuntu kernel: [12345.678] Memory: 4012345K/4023456K available
    Dec 15 14:20:45 ubuntu systemd[1]: disk-usage-check.service: Succeeded.

✔ Verification: If you see 'No entries' or very few entries, this is normal on a freshly installed system with few errors. The commands executed correctly.

### Step 15: Create a summary report of system resource status

Compile a comprehensive resource snapshot by running these commands and saving output to a file:

    echo "=== System Resource Report ===" > ~/system_report.txt
    echo "Generated: $(date)" >> ~/system_report.txt
    echo "" >> ~/system_report.txt
    echo "--- CPU and Load ---" >> ~/system_report.txt
    uptime >> ~/system_report.txt
    echo "" >> ~/system_report.txt
    echo "--- Memory Usage ---" >> ~/system_report.txt
    free -h >> ~/system_report.txt
    echo "" >> ~/system_report.txt
    echo "--- Disk Usage ---" >> ~/system_report.txt
    df -h >> ~/system_report.txt
    echo "" >> ~/system_report.txt
    echo "--- Top 5 CPU Processes ---" >> ~/system_report.txt
    ps aux --sort=-%cpu | head -6 >> ~/system_report.txt

View the report:

    cat ~/system_report.txt

Expected Output:

    === System Resource Report ===
    Generated: Fri Dec 15 14:45:23 UTC 2023

    --- CPU and Load ---
     14:45:23 up  3:16,  2 users,  load average: 0.15, 0.22, 0.18

    --- Memory Usage ---
                  total        used        free      shared  buff/cache   available
    Mem:          3.8Gi       1.2Gi       1.3Gi        89Mi       1.3Gi       2.4Gi
    Swap:         2.0Gi          0B       2.0Gi

    --- Disk Usage ---
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/sda1        25G  8.2G   16G  35% /

    --- Top 5 CPU Processes ---
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    student   2345  1.2  4.5 876543 123456 ?      Sl   12:05   0:34 /usr/bin/gnome-shell

✔ Verification: Run `ls -lh ~/system_report.txt` to confirm the file was created. The file size should be approximately 1-2 KB.

## Troubleshooting

Problem: The 'yes' process doesn't appear at the top of the ps output or shows low CPU usage
Solution: Run the ps command multiple times, as CPU usage percentages are calculated over sampling intervals. If the process is niced to a low priority, it may not consume 100% CPU if other processes need resources. You can also create multiple 'yes' processes with 'yes > /dev/null &' repeated several times.
Problem: Permission denied error when running renice or journalctl commands
Solution: These commands require elevated privileges. Ensure you prefix them with 'sudo'. If sudo still fails, verify your user account is in the sudo group by running 'groups' and checking for 'sudo' in the output. If not present, contact your system administrator.
Problem: htop displays 'command not found' error even after installation
Solution: The package manager cache may need updating. Run 'sudo apt update' first, then retry 'sudo apt install htop -y'. If the problem persists, verify your Ubuntu VM has internet connectivity by running 'ping -c 3 google.com'.
Problem: The kill command doesn't stop the process and it keeps running
Solution: Some processes may ignore the default SIGTERM signal. Use a stronger signal: 'kill -9 PID' where PID is your process ID. The -9 sends SIGKILL, which cannot be ignored. Always try regular kill first, as kill -9 prevents processes from cleaning up properly.
Problem: du command takes a very long time to complete or appears frozen
Solution: If scanning large directories, du can take several minutes. Press Ctrl+C to cancel and try scanning a smaller directory first, such as 'du -h --max-depth=1 ~/Documents'. Avoid scanning system directories like / without sudo, as permission errors will slow the scan.

## Challenge Exercise

### Performance Investigation and Optimization

Your system has been experiencing slow performance during business hours. Using the monitoring tools and techniques from this lab, investigate and document the following:

1. Create a simulated high-load scenario by starting 3-4 CPU-intensive processes simultaneously (like the 'yes' command used earlier). Document their PIDs.

2. While these processes are running, capture system metrics including:

   - Overall CPU usage and load average
   - Memory utilization percentage
   - The top 10 processes by CPU and memory usage
   - Any swap usage occurring

3. Implement optimization strategies:

   - Adjust nice values on the test processes to different priority levels (try values: 5, 10, 15, 19)
   - Monitor how the priority changes affect CPU distribution
   - Systematically terminate processes starting with the lowest priority

4. Create a before-and-after comparison report that includes:

   - System metrics before optimization
   - Actions taken with specific commands and PIDs
   - System metrics after optimization
   - Analysis of which processes had the biggest impact

5. Use journalctl to search for any system warnings or errors that occurred during your high-load simulation period

Save all output and your analysis to a file called ~/performance_investigation.txt. Your report should demonstrate understanding of process priorities, resource allocation, and systematic troubleshooting methodology.

# Practice Questions

1. What does a high I/O wait percentage (wa) in the top command indicate?

A) The system needs more CPU cores

B) The system is spending significant time waiting for disk or network operations

C) Memory is nearly exhausted

D) Too many processes are running

2. On a system with 4 CPU cores, what load average indicates the system is fully utilized but not overloaded?

A) 1.0

B) 4.0

C) 10.0

D) 0.25

3. What is the main difference between the ps and top commands?

A) ps requires root privileges while top does not

B) ps provides a snapshot at a point in time while top continuously updates

C) ps shows only system processes while top shows all processes

D) ps can kill processes while top cannot

4. You notice a Linux system showing 90% memory usage but minimal swap activity. What does this indicate?

A) The system is about to run out of memory and will crash

B) The system is likely healthy and using available RAM efficiently for caching

C) Applications are experiencing significant slowdowns due to memory pressure

D) The swap configuration is incorrect and needs to be fixed

5. What nice value should you use to run a backup script with the lowest possible priority?

A) -20

B) 0

C) 10

D) 19

6. Which signal should you try FIRST when attempting to terminate a misbehaving process?

A) SIGKILL (9)

B) SIGTERM (15)

C) SIGHUP (1)

D) SIGSTOP (19)

7. What is the primary difference between the df and du commands?

A) df shows disk usage while du shows network usage

B) df reports file system level usage while du analyzes directory-specific consumption

C) df requires root privileges while du does not

D) df is for physical disks while du is for virtual file systems

8. You receive 'No space left on device' errors but df shows 30% free space. What is the most likely cause?

A) The df command is showing incorrect information

B) The file system is corrupted

C) The inode table is exhausted

D) The disk hardware is failing

9. Which journalctl command shows only error messages from the SSH service in the last hour?

A) journalctl -u sshd.service -p err --since '1 hour ago'

B) journalctl -f sshd.service --errors

C) journalctl --grep error --service sshd

D) journalctl -b -p warning -u sshd

10. Why is establishing a performance baseline important for system administration?

A) It allows you to compare past performance to industry standards

B) It provides a reference point to distinguish normal behavior from actual problems

C) It is required by compliance regulations

D) It automatically fixes performance problems

# Module Summary

This module covered essential system resource monitoring and optimization techniques for Linux administrators. You learned to monitor processes using top, htop, and ps commands, interpret CPU and memory metrics to identify bottlenecks, manage process priorities with nice values, and control processes using signals. You also learned to analyze disk usage with df and du, interpret system logs using journalctl, and establish performance baselines for proactive capacity planning. These skills enable you to maintain optimal system performance, troubleshoot issues effectively, and prevent resource exhaustion before it impacts users.