Log Analysis with Machine Learning

Structured: JSON format with fixed fields. {"timestamp": "...", "service": "api", "level": "ERROR", "message": "timeout"}

Unstructured: Free-form text. "2024-01-15 10:30 ERROR: Payment API timed out after 5s waiting for db response"

Why it matters: ML can't directly process text. Unstructured logs need NLP (parsing, tokenization, embedding) before ML can use them. Structured logs are already parsed, so ML gets faster training.

Production practice: Always log in structured format (JSON). If you're stuck with unstructured, use regex parsing to extract key fields first.

Deduplication: Grouping similar-looking log messages into one pattern. E.g., 500K logs of "Connection timeout" all count as one pattern occurrence every 1 millisecond.

Why needed: Without dedup, you have 500K separate data points for the same problem. With dedup, you have 1 data point ("connection timeout") with count=500K. Much easier to analyze and alert on.

Example: Without dedup: alert if any of 500K timeout logs appear. Result: alerts fire constantly; alert fatigue. With dedup: alert if "connection timeout" pattern count exceeds baseline. Result: smarter alerting.

TF-IDF = Term Frequency × Inverse Document Frequency

Purpose: Convert text to a numerical vector, where words that appear in many documents get lower weight (less important), and words unique to a document get higher weight (more important).

In logs: "error" appears in every log → low weight. "OOMKilled" appears only in 2 logs → high weight. When comparing two log messages, TF-IDF emphasizes the rare, distinctive terms.

Use case: Cluster logs by TF-IDF vectors. Similar logs → similar TF-IDF vectors → same cluster.

Trace ID: A unique identifier attached to a request as it flows through multiple services. All logs for one request carry the same trace ID.

Example:

• User initiates checkout (trace_id=xyz123)
• API logs: "[xyz123] Received checkout request"
• Payments service logs: "[xyz123] Processing payment"
• Database logs: "[xyz123] Inserting order record"

Benefit: Query all logs with trace_id=xyz123 to get end-to-end view of that one request. Spot where it failed: if log is missing from database service, database was bottleneck.

Key: Sample by pattern, not randomly.

Wrong approach: Random sample 10K logs. Result: you might miss rare error patterns (e.g., "database deadlock" happens 100 times/day → 0.001% of volume → random sample probably misses it).

Right approach:

• Sample all unique patterns at least once (so rare errors are included)
• Over-sample from high-volume patterns (otherwise dominated by common logs like "request received")
• Use stratified sampling: 10K samples from ERROR logs, 10K from WARN, 10K from INFO

Result: 30K logs instead of 1B, balanced representation of all patterns.

Sudden spike detection: Easy. Error rate: 5/min today, 150/min now → alert.

Gradual degradation: Hard. Error rate grows: 5 → 7 → 10 → 14 → 20 → 30 → 50 over 2 hours. No single spike, but systemic problem growing. Humans might not notice until it's crisis.

Why harder: Threshold-based alerts miss gradual changes (you'd tune threshold at 30, but by hour 1 it was 50 and breaking things). Moving average struggles similarly.

Solution: Trend detection. Track error rate slope (rate of change). If slope > X for > 30 minutes, alert on trend not absolute value. E.g., "error rate increasing by 2/min every minute for 30 min → alert."

Root cause: Data distribution shift (domain adaptation problem)

US-East logs are in English. EU-West logs might include German/French/Spanish service names, error messages, or different code paths for GDPR compliance. The TF-IDF vectorizer was trained on English vocabulary → weak on European languages.

Fixes:

• Retrain on multi-region data: Retrain clustering model on European logs (or combined US + EU logs). Add language-agnostic features (error codes, service names, which are often in English anyway)
• Use multilingual embeddings: Instead of TF-IDF, use multilingual transformer embeddings (like mBERT) that understand multiple languages natively
• Transfer learning: Take US-East model, fine-tune on small sample of EU-West logs (100 logs) to adapt without full retraining

Lesson: Always validate model on target domain (EU-West needs independent validation before production).

Approach:

• Trace ID propagation: Every request carries trace_id from entry point (API gateway) to exit (database). All services log with this ID
• Central log aggregation: All 50 services ship logs to central store (ELK, Splunk, CloudWatch Logs) with trace_id indexed
• Incident query: When outage detected (error rate spike), query ALL logs for that time window
• Cross-service analysis: For each service, check: did it log errors in that window? In what order?
  • ServiceA: no errors
  • ServiceB: errors starting T=0s
  • ServiceC: errors starting T=2s (delayed, likely downstream of B)
  • ServiceD: no errors
• Root cause: ServiceB failed first → likely root cause. ServiceC failed after → consequence.

Automation: Build dependency graph. When incident fires, trace backward: which service failed first (earliest error log timestamp)? That's your root cause.

Problem: Logs might include user IDs, email addresses, payment card tokens, etc. If you train ML models on this data, you embed PII in model weights. GDPR/CCPA violation if exposed.

Solutions:

• Redaction before training: Strip or hash PII before logs reach ML pipeline. E.g., "User john@example.com failed login" → "User [HASHED_USER_ID] failed login"
• Pattern-based anonymization: Replace IP addresses, email addresses, credit card numbers with tokens before analysis
• Separate sensitive data: Log transaction IDs, not user ID. Log transaction status (success/failure), not payment amount
• Encryption at rest: Even with redaction, store logs encrypted in case of breach

Best practice: Redaction happens at log ingestion time, before data reaches ML pipeline. Output models never see raw PII.

Approach:

• Weekly retraining schedule: Every Sunday night, retrain model on last 7 days of logs (captures current patterns)
• Data preparation: Hourly aggregation: count of each log pattern per hour. Remove low-frequency patterns (noise)
• Validation: Before deploying new model, test on held-out test set (logs from 7 days ago). Ensure cluster quality didn't degrade
• A/B test: Route 10% of new logs to new model, 90% to old model. Compare alert quality (false positive rate, missed incidents). If new model better, roll out to 100%
• Rollback trigger: If new model produces > 2x false positive rate, automatically revert to previous model
• Monitoring: Track model drift: percentage of logs that would cluster differently between old and new model. If > 20%, investigate

Frequency tuning: Weekly is good for stable systems. High-velocity startups might need daily retraining. Legacy systems might only retrain quarterly

Problem: 50GB data, 6 hour training. If you need daily updates, this is unacceptable.

Solutions (in order of effort):

• Reduce data volume: Deduplicate raw logs first. If 90% of logs are duplicates, you're down to 5GB
• Stratified sampling: Instead of training on all 50GB, sample 1-5GB stratified by log type. Maintain pattern coverage, reduce volume
• Incremental learning: Don't retrain from scratch daily. Use online/streaming ML (mini-batch updates). New logs incrementally update model → hours → minutes
• Feature caching: Pre-compute TF-IDF vectors at ingestion time. Retraining then only does clustering (fast) not vectorization (slow)
• Distributed training: Use Spark MLlib or Dask to parallelize clustering across multiple machines. 6 hours → 30 minutes on 10-machine cluster

Practical recommendation: Combine sampling (reduce to 2GB) + incremental learning + distributed training → get to < 1 hour daily.

Root cause: Attack generates MB of unique log messages (random payloads to bypass WAF). Normally, clustering would deduplicate these into 1 pattern. But the clustering model ran out of memory trying to fit millions of unique vectors.

Why it happened: Log volume doesn't scale linearly with attack volume. An attacker sending 10K requests/sec with random variations = 10K unique-looking log entries, overwhelming the tokenizer/vectorizer.

Recovery (immediate):

• Stop consuming new logs (block at ingestion)
• Increase memory allocation to log analyzer service
• Restart analyzer service (restart clears memory)
• Enable WAF rule to drop "SQL injection attempt" logs (too noisy)

Long-term fixes:

• Add memory limits / early termination in clustering: if data > 1GB, exit and alert instead of crashing
• Add rate limiting: if one pattern appears > 1000 times in 1 minute, deduplicate it immediately (don't wait for full batch analysis)
• Add volume thresholds: if log volume 10x normal, enable sampling mode (analyze 1 in 10 logs)

Architecture (real-time log correlation):

• Event streaming: Ship logs from all services to Kafka topic (per-service partition). All messages have trace_id
• Stream processor (Kafka Streams / Flink): Join streams by trace_id. For each trace, collect all events from all services
• Windowing: Process logs in 10-second tumbling windows. For each window, identify cross-service patterns
• Correlation logic:
  • If ServiceA logs ERROR for trace X
  • AND ServiceB logs ERROR for same trace X within 2 seconds
  • AND ServiceC logs ERROR for same trace X within 2 seconds
  • THEN fire "cross-service failure detected" incident
• Output: Incident stream → Alert system

Why this architecture: Kafka handles volume (160K logs/min across all services). Stream processor handles correlation with low latency (sub-second). Partitioning by trace_id ensures logs for same trace go to same processor

Signal interpretation: High correlation between memory log and restart. Possibilities:

• Memory leak (most likely): Application's heap grows to X MB → garbage collector can't keep up → application auto-restarts due to OOM. Correlation = 95% because this is happening repeatedly
• Memory threshold trigger: Some orchestration config might say "if heap > X MB, restart pod." Then correlation is expected and OK
• Resource exhaustion cascade: Memory spike → slow performance → health check times out → orchestrator decides pod is unhealthy → restarts

What to do:

• Trace back: which version of code introduced "Memory heap at X"? Compare code between version N (no pattern) and N+1 (pattern present) → find memory leak
• Check correlation with features: does pattern only appear in certain circumstances (high load, specific endpoints)? This guides where to instrument deeper
• Set alert on pattern: "Memory heap at 80% + application restart within 30s" = red flag, page on-call engineer

Scenario: Attacker exploits app to dump MB of debug logs, flooding log system. Goal: hide their malicious requests in the noise.

Detection (ML approach):

• Monitor log volume per service: if ServiceA usually logs 10K events/min and suddenly 1M events/min → anomaly
• Monitor entropy of log messages: normal logs have ~200 unique patterns. "Log bomb" often produces high-variance (random-looking) messages → detectable
• Alert: "Log volume spike + new high-entropy pattern detected"

Response:

• Immediate: Enable sampling on ServiceA logs (log 1 in 100 events). This reduces volume while maintaining representative sample
• Investigation: Before sampling, capture raw log sample for forensics. Extract trace_ids that correlate with attack (often have unusual patterns like SQL injection attempts)
• Breach investigation: For traces that contain actual attack payload (not noise), trace backward through ServiceA's code to find the injection point
• Patch: Fix the injection vulnerability while handling the volume surge

Prevention: Add circuit breaker: if log rate > 10x baseline for > 1 minute, trigger SEV-1. Auto-drop low-priority logs. Never let volume spike consume all disk space