Every meaningful event on a Linux system leaves a trace somewhere — a failed SSH login, an unexpected sudo command, a firewall drop, a spike in 404s from a web server. The problem isn’t a lack of data; it’s knowing where to look and what pattern separates routine noise from an actual incident. This guide walks through the log sources that matter most, the commands blue teamers and sysadmins actually use day to day, and how to spot the anomalies that logs are trying to tell you about.

Table of Contents

1. Know Your Log Sources
2. Two Logging Systems: Flat Files vs. the systemd Journal
3. Key Linux Log Files and What They Contain
4. Essential Commands for Authentication Analysis
5. Log Rotation: Keeping Logs Useful and Available
6. Anomaly Patterns Worth Watching For
7. Threat Hunting Techniques on Linux Logs
8. Parsing, Enrichment, and Scaling Beyond grep
9. Conclusion

---

Know Your Log Sources

Before touching a single command, it helps to map out where different categories of evidence actually live. Logs generally fall into a handful of buckets, each useful for a different kind of investigation:

Source	Example	Blue Team Use Case
Operating System (Linux)	/var/log/auth.log	SSH login attempts, sudo usage
Operating System (Windows)	Security.evtx	Logons, process creation, privilege use
Network Devices	Firewall logs, DHCP logs	Blocked traffic, address leasing
Web Servers	Apache/Nginx access & error logs	Suspicious URIs, directory traversal, fuzzing
Security Tools	EDR, antivirus, SIEM	Threat detections, lateral movement
Cloud Infrastructure	AWS CloudTrail, Azure Activity Log	API misuse, cloud privilege escalation

Knowing which bucket an incident likely falls into before you start searching saves significant time — there’s no point grepping web server logs for a suspected SSH brute-force attempt.

Two Logging Systems: Flat Files vs. the systemd Journal

Modern Linux distributions run two logging mechanisms side by side, and knowing which one you’re dealing with changes your approach entirely.

Plain-text files under /var/log/ are written by traditional syslog daemons (rsyslog, syslog-ng) and application processes directly. These are the classic auth.log, syslog, and application-specific logs you can grep and tail directly.

The systemd journal, queried through journalctl, stores structured, binary log data — including kernel messages, every service’s stdout/stderr, and audit events — and supports filtering by time, service unit, severity, or even the exact executable path that generated an entry. On most current systems, both exist simultaneously: use journalctl for systemd-managed services, and flat files for anything an application manages on its own.

Key Linux Log Files

Log File	Description
/var/log/auth.log	Authentication events, sudo usage, SSH login attempts (Debian/Ubuntu)
/var/log/secure	Equivalent to auth.log on RHEL-based distributions
/var/log/syslog	General system events and kernel messages
/var/log/messages	Daemon and system messages (RHEL/CentOS)
/var/log/wtmp	Binary log of user logins and reboots
/var/log/btmp	Binary log of failed login attempts
/var/log/lastlog	Last login timestamp for every user
/var/log/sudo.log	Optional dedicated sudo log, if configured

Alt text: Data center servers generating and storing system logs

Essential Commands

These commands form the backbone of everyday authentication analysis and general triage:

View the last 20 SSH login failures:

grep "Failed password" /var/log/auth.log | tail -n 20

Follow system events in real time:

tail -f /var/log/syslog

Query successful SSH logins:

grep "Accepted password" /var/log/auth.log

Summarize failed logins by source IP:

grep "Failed password" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -nr

Review sudo usage:

grep "sudo" /var/log/auth.log

Show recent and failed logins from binary logs:

last     # recent logins (from wtmp)
lastb    # failed logins (from btmp)

Query a specific time window with journalctl:

journalctl --since "2026-06-30 14:00:00" --until "2026-06-30 14:10:00"

Pretty-print JSON-formatted logs (e.g., Suricata alerts):

cat eve.json | jq '.'

Log Rotation

Logs are only useful if they’re actually available when you need them — and if they haven’t already filled the disk. Most Linux distributions handle this with logrotate, configured through /etc/logrotate.conf and the individual files under /etc/logrotate.d/. It manages compression, rotation frequency, and eventual deletion so a busy service doesn’t quietly consume all available disk space.

Test a rotation configuration without applying it:

sudo logrotate --debug /etc/logrotate.conf

Quick PowerShell equivalent check on the Windows side, for mixed environments:

Get-EventLog -LogName Security | Measure-Object

Typical Retention Guidelines

Log Type	Recommended Retention
Authentication logs (SSH, RDP)	180–365 days
Security logs (EDR, SIEM)	1–3 years
Web server access logs	90–180 days
Firewall, IDS/IPS logs	90–365 days
Cloud API activity logs	180–365 days
Compliance-regulated systems	1–7 years

These are starting points, not hard rules — actual retention should be driven by your organization’s compliance obligations and incident-response needs, whichever is longer.

Anomaly Patterns

Raw log volume is meaningless without knowing which patterns actually indicate a problem. A few recurring ones are worth memorizing:

Pattern	Indicator
Multiple failed logins in a short window	Brute-force attempt
Login at an unusual time	Insider threat or compromised account
sudo use by an unauthorized-looking user	Privilege escalation
SSH access from an unexpected country/IP	Possible lateral movement
Frequent switching between user accounts	Session hijack or account enumeration

None of these are proof on their own — a developer working late isn’t automatically an insider threat — but each one is a reason to look closer.

Threat Hunting Techniques

Beyond passive monitoring, these targeted queries help surface activity that blends into normal log noise.

Hunting interactive root shell access. Direct root shells are unusual outside specific administrative tasks — worth flagging when they appear unexpectedly:

grep 'COMMAND=/bin/bash' /var/log/auth.log | grep 'USER=root'

Looking for lateral movement via SSH. Filter out known jump hosts or bastions to surface internal connections that shouldn’t be happening directly:

grep "Accepted password" /var/log/auth.log | grep -vE '192\.168\.0\.1|10\.0\.0\.1'

Monitoring a sudden sudo usage spike. Build a frequency table by user to catch accounts that suddenly start escalating privileges when they normally don’t:

grep "sudo" /var/log/auth.log | awk '{print $1, $2, $3, $9}' | sort | uniq -c | sort -nr

Detecting user enumeration scans. Repeated attempts against nonexistent usernames are a strong signal of automated account discovery:

grep "Invalid user" /var/log/auth.log | awk '{print $(NF)}' | sort | uniq -c | sort -nr

Querying audit events directly, if auditd is enabled:

ausearch -m USER_LOGIN -ts recent
ausearch -ua 1001

Parsing and Enrichment

Manual grep-and-awk workflows work well for triage and small environments, but they don’t scale to fleets of servers or high log volumes. For that, dedicated parsing and enrichment tools take over:

• Logstash, Fluent Bit, or syslog-ng convert raw log lines into structured, enriched events — adding context like geolocation, asset ownership, or threat-intel matches before the data ever reaches an analyst.
• Centralized platforms like an ELK stack or a SIEM aggregate logs from every host into one searchable index, which is essential once you’re correlating events across more than a handful of machines.
• journalctl‘s structured filtering — by unit, priority, time window, or even the exact executable path (_EXE=/usr/bin/python3) — is worth learning well, since it can isolate every log line generated by a suspicious process across a system’s entire history without needing an external pipeline.

The right tool depends on scale: a single server is well served by grep and journalctl; a fleet of dozens or hundreds needs centralized aggregation to make correlation practical at all.

Conclusion

Log analysis isn’t about reading everything — it’s about knowing which file holds the answer to a specific question, and which command gets you there fastest. Authentication logs tell you who’s trying to get in and how; the systemd journal tells you what services actually did; retention policy determines whether that evidence still exists when you need it weeks later. Building fluency with grep, awk, journalctl, and a short list of known anomaly patterns covers the vast majority of real-world investigations, and gives you a solid foundation before reaching for a full SIEM.