IntermediateLesson 6 of 16

Regular Expressions and Text Processing

Master regex patterns for parsing logs, extracting data from unstructured text, and validating input—DevOps' secret weapon for automated log analysis and data extraction.

🧒 Simple Explanation (ELI5)

Regular expressions (regex) are patterns you use to search for text. Without regex, searching needs exact matches. With regex, you can say "find lines starting with ERROR", "find all IP addresses", "find timestamps". Regex is like a search filter with superpowers—you describe what you are looking for and regex finds all matches.

🔧 Why Do We Need Regex?

⚙️ Technical Explanation

Regular expressions are patterns using special characters to match text. . matches any character, * means zero or more, + means one or more, ? means optional, [] matches character sets, () groups, ^ and $ anchor to start/end.

💡
Use Raw Strings for Regex Patterns

Always use raw strings (r"pattern") for regex to avoid escaping backslashes. r"\d+" works; "\d+" fails because Python interprets \ first. Raw strings tell Python to pass the pattern to the regex engine as-is.

⌨️ Regex Patterns and Matching

python
import re

# ===== BASIC MATCHING =====
pattern = r"ERROR"
text = "2024-01-15 10:30:45 ERROR Database connection failed"

if re.search(pattern, text):
    print("Pattern found")

# ===== COMMON PATTERNS =====
r"\d+"          # one or more digits: 123, 42
r"\d{3}"        # exactly 3 digits: 192
r"\w+"          # word characters (letters, digits, _): server01, config_v2
r"\s+"          # whitespace (spaces, tabs, newlines)
r"[a-z]"        # lowercase letters
r"[A-Z]"        # uppercase letters
r"[0-9a-f]"     # digits or a-f (hexadecimal)
r"."            # any character (except newline)
r"\."           # literal dot (escaped)
r"\d+\.\d+"     # decimal number: 192.168 or 3.14
r".*"           # any characters, zero or more
r".+"           # any characters, at least one
r"[A-Z][a-z]?" # uppercase letter optionally followed by lowercase

# ===== IP ADDRESS REGEX =====
ip_pattern = r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}"
log_line = "Connection from 192.168.1.100 at 10:30:45"
match = re.search(ip_pattern, log_line)
if match:
    print(f"Found IP: {match.group()}")  # 192.168.1.100

# ===== EXTRACTING WITH GROUPS =====
# Extract timestamp and level from log line
log_pattern = r"(\d{4}-\d{2}-\d{2})\s(\d{2}:\d{2}:\d{2})\s(\w+)"
log_line = "2024-01-15 10:30:45 ERROR System failure"

match = re.search(log_pattern, log_line)
if match:
    date = match.group(1)       # "2024-01-15"
    time = match.group(2)       # "10:30:45"
    level = match.group(3)      # "ERROR"
    print(f"Date: {date}, Time: {time}, Level: {level}")

# ===== FINDALL: GET ALL MATCHES =====
# Extract all IPs from log file
text = """
Connection from 192.168.1.100
Error from 10.0.0.50
Backup to 172.16.0.1
"""

ips = re.findall(ip_pattern, text)
for ip in ips:
    print(f"IP: {ip}")

# ===== SUBSTITUTION (FIND AND REPLACE) =====
# Redact sensitive data
config = "password=abc123 user=admin key=secret"
redacted = re.sub(r"(password|key)=\S+", r"\1=***REDACTED***", config)
print(redacted)  # password=***REDACTED*** user=admin key=***REDACTED***

# ===== SPLIT BY PATTERN =====
# Split by whitespace or comma
data = "hostname: server01,  ip:  192.168.1.1,  port:  8080"
parts = re.split(r"\s+|,", data)
print(parts)  # ['hostname:', 'server01', 'ip:', '192.168.1.1', 'port:', '8080']

# ===== FLAGS =====
text = "Error encountered"
# Case-insensitive search
match = re.search(r"error", text, re.IGNORECASE)
if match:
    print("Found (case-insensitive)")

# Multi-line mode: ^ and $ match line boundaries
multiline_text = """
ERROR: Failed
WARNING: Check
ERROR: Again
"""
errors = re.findall(r"^ERROR:", multiline_text, re.MULTILINE)
print(errors)  # ['ERROR:', 'ERROR:']  (not just the first one)

# ===== REAL-WORLD EXAMPLE: PARSE APACHE LOGS =====
# Apache log format: 192.168.1.100 - admin [15/Jan/2024:10:30:45] "GET /api HTTP/1.1" 200 1024
apache_pattern = (
    r'(\d+\.\d+\.\d+\.\d+)\s'           # IP
    r'-\s+(\S+)\s+'                      # Username
    r'\[([^\]]+)\]\s+'                   # Timestamp
    r'"(\w+)\s+(\S+)\s+HTTP"'            # Method and Path
    r'\s+(\d{3})\s+'                     # Status code
    r'(\d+)'                             # Bytes sent
)

log_line = '192.168.1.100 - admin [15/Jan/2024:10:30:45] "GET /api HTTP/1.1" 200 1024'
match = re.search(apache_pattern, log_line)
if match:
    groups = match.groups()
    print(f"IP: {groups[0]}, User: {groups[1]}, Status: {groups[4]}")

# ===== EMAIL VALIDATION =====
email_pattern = r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"
email = "admin@example.com"
if re.match(email_pattern, email):
    print("Valid email")

# ===== COMPILE PATTERN FOR REUSE =====
# If using same pattern multiple times, compile for performance
pattern = re.compile(r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}")
for line in ["IP: 192.168.1.1", "Not found", "Server: 10.0.0.1"]:
    if pattern.search(line):
        print(f"Found IP in: {line}")

💼 Example (Real-world Use Case)

A log analysis script reads Kubernetes audit logs, extracts all API calls from a specific user using regex, groups them by resource (Pod, Service, Deployment), counts access patterns, and flags suspicious activity (like 100+ deletions in 1 minute). Regex extracts user, action, resource, and timestamp from each log line, then Python logic aggregates and alerts.

🧪 Hands-on

  1. Write a regex to match ISO 8601 timestamps (2024-01-15T10:30:45Z).
  2. Extract all IP addresses from a block of text.
  3. Write a script that validates email addresses using regex.
  4. Create a regex to extract hostnames from URLs (e.g., from "https://api.example .com/v1" extract "api.example.com").
  5. Parse a log line and extract timestamp, level, and message.
🎮
Try It Yourself

Write a regex-based log processor that reads a log file, extracts all ERROR lines, groups errors by the word immediately after ERROR (e.g., "ERROR_Database", "ERROR_Auth"), and counts occurrences. Output a summary.

🐛 Debugging Scenario

Problem: regex works in an online tester but not in Python code.

🎯 Interview Questions

Beginner

What is the difference between re.search() and re.match()?▾

re.match() looks for the pattern at the start of the string. re.search() looks anywhere in the string. Use re.match() to validate format (email starts with alphanumeric), use re.search() to find text anywhere in a log line.

Why use raw strings (r"...") for regex patterns?▾

Raw strings prevent Python from interpreting backslashes. Without r"", "\d" becomes a literal character d after Python processes the backslash. With r"\d", the pattern engine receives the literal string "\d" and interprets it as "any digit".

What does re.findall() return?▾

re.findall() returns a list of all matches. If the pattern has groups (captured parts in parentheses), it returns a list of tuples. If no groups, it returns a list of strings. If no matches, it returns an empty list (not None).

Scenario-based

Write a regex to extract username and realm from an auth log line like "Failed auth for user@domain.com from 192.168.1.1". How would you extract just the username?▾

Pattern: r"user\s+(\w+)@(\S+)\s+from\s+(\d+\.\d+\.\d+\.\d+)". Use groups: group(1) is username, group(2) is domain, group(3) is IP. Or simpler: split the line and extract the "@" part, then split by "@" again.

🌐 Real-world Usage

Ansible template j2 uses regex for conditionals and substitutions. Prometheus and Grafana use regex for label matching. CloudWatch Logs Insights uses regex for filtering. Every log aggregation tool uses regex—this is a mandatory DevOps skill.

📝 Summary

Regex patterns match text using special characters: \d for digits, \w for word chars, * for zero+, + for one+, [] for sets. re.search() finds one match, re.findall() finds all. Use raw strings (r"...") to avoid backslash issues. Groups () capture parts of the match for extraction. Regex is essential for log parsing, validation, and find-replace operations across DevOps automation.

← Back to Course