An intelligent, autonomous Kubernetes incident investigation and root cause analysis engine written in Go. It ingests Alertmanager webhooks or Kubernetes informer events, inspects real pod states and logs via direct Kubernetes API, queries Prometheus & Loki for correlated signals, classifies root causes across 10 deterministic failure modes with numeric confidence scoring, and enriches findings with multi-round LLM analysis.
Architecture Overview
🚨
Triggers
Alertmanager Webhook & K8s Informer events
→
🛡️
Dedup Engine
5m cooldown & 2m namespace fan-in
→
🔍
Evidence Collector
Parallel K8s API, Prometheus & Loki queries
→
🧠
Root Cause Engine
10 deterministic modes with 0.2–0.99 score
→
🤖
Multi-LLM & CRD
Claude/GPT enrichment & live SRE dashboard
Key Technical Capabilities
- Direct Kubernetes API Inspection: Uses
client-go to introspect pod phase, container restart counts, exit codes (OOMKilled exit 137, CrashLoopBackOff), deployment rollout stalled status, and previous container crash logs.
- Deterministic-First Failure Modes: Classifies 10 failure modes (
CrashLoopBackOff, OOMKilled, ImagePullBackOff, PendingPod, DeploymentRolloutFailure, ServiceSelectorMismatch, ProbeFailed, ServiceEndpointsUnhealthy, SecretOrCredentialRegression, NamespaceWideInstability).
- Multi-Round Investigation Loop: Executes follow-up actions iteratively until confidence reaches target threshold (≥ 0.90) or planner budget exhausts.
- Multi-LLM Enrichment (Fail-Open): Pluggable support for Anthropic Claude (claude-3-5-sonnet), OpenAI (GPT-4o), and OpenRouter models. If LLM is unreachable, deterministic root causes stand unaffected.
- Kubernetes CRD Durability: Stores all incidents and investigation runs as native Kubernetes Custom Resources (
incidents.platform.example.com) backed by etcd.
- Live Operational Dashboard: Real-time browser dashboard with status workflows (New → Acknowledged → Reviewing → Resolved → Closed).
Synthetic Trigger & Execution Sample
# Trigger synthetic incident:
curl -X POST http://localhost:8080/webhook/synthetic \
-H "content-type: application/json" \
-d '{
"source":"synthetic",
"namespace":"payments",
"workload":"payment-api",
"reason":"OOMKilled",
"severity":"high"
}'
# Live Investigation Result:
✓ Evidence collected: 14 items (K8s API + Prometheus memory peak 489Mi)
✓ Root cause: OOMKilled (Confidence: 94%)
✓ Immediate Fix: Increase container memory limit from 512Mi to 1Gi in Helm values
✓ Long-term Fix: Profile JVM/Go heap on /v1/checkout endpoint to catch memory leaks
Migrated personal production infrastructure from AWS EC2 to a self-hosted on-premise environment running locally. The site is served by Nginx and securely exposed to the public internet via Cloudflare Tunnel (cloudflared daemon). This architecture eliminates all open inbound router ports, requires zero port forwarding or public static IPs, leverages Cloudflare's Anycast edge for global caching and DDoS mitigation, and brings infrastructure cloud costs to $0.
Architecture & Tunnel Flow
💻
Local Server
Nginx serving static assets on local port (127.0.0.1:80)
→
🚇
cloudflared Daemon
Outbound-only encrypted tunnel connection
→
🛡️
Cloudflare Edge
Anycast routing, DDoS shield & Edge TLS termination
→
🌐
Global Visitors
Ultra-low latency HTTPS at mevinod.com
Key Engineering Highlights
- Zero Inbound Ports: No NAT port forwarding, no router firewall pinholes, and no public IP exposure on the local network.
- Outbound Tunnel Daemon:
cloudflared initiates redundant outbound-only connections to the nearest Cloudflare edge Points of Presence.
- Edge TLS & HTTP/3: Cloudflare terminates modern TLS 1.3 and HTTP/3 at the edge automatically, providing instant SSL without managing local certbot renewals.
- Local Web Server: Optimized Nginx instance handling gzip compression, static file caching headers, and security headers locally.
- Automated CI/CD Delivery: Lightweight authenticated webhook trigger from GitHub Actions directly to a local listener executing zero-overhead
git pull.
- Cost & Efficiency: 100% cloud compute savings ($0/month vs EC2) while maintaining 99.9% edge availability backed by Cloudflare's global CDN cache.
Cloudflare Tunnel Ingress Configuration
# ~/.cloudflared/config.yml
tunnel: <tunnel-uuid>
credentials-file: ~/.cloudflared/<tunnel-uuid>.json
ingress:
- hostname: mevinod.com
service: http://localhost:80
- hostname: www.mevinod.com
service: http://localhost:80
- service: http_status:404
A lightweight DevOps utility that retrieves AWS ECR login tokens through two interfaces: a lightweight Flask web API and a self-contained PyInstaller Linux CLI binary. Solves token expiration issues in automated CI/CD runners, local development scripting, and air-gapped container workflows.
Architecture Overview
💻
Client Request
HTTP POST or CLI invocation
→
🐍
Flask Web App / CLI
Parses credentials & region
→
☁️
boto3 SDK
Calls GetAuthorizationToken API
→
🔐
AWS ECR
Returns 12h Docker login auth
Key Components
- Dual Mode Operation: Run as a lightweight REST microservice on port 5000 or as a standalone CLI executable.
- PyInstaller Packaging: Compiles Python code and dependencies into a single Linux binary with zero runtime prerequisites.
- Dockerized Distribution: Multi-stage Docker builds providing minimal Alpine-based container footprints.
- Automated Testing:
pytest unit test suites integrated into GitHub Actions CI pipeline.
Usage Syntax
# CLI Mode:
./ecr_token_cli --access-key AKIA... --secret-key ... --region us-east-1
# Web API Mode:
curl -X POST http://localhost:5000/get-token \
-H "Content-Type: application/json" \
-d '{"aws_access_key":"...","aws_secret_key":"...","region":"us-east-1"}'
# Response:
{ "token": "eyJwYXNzd29yZCI6...", "expires_at": "2026-10-02T21:00:00Z" }
A self-hosted, private photo management platform replacing Google Photos. Immich runs as an orchestrated multi-container Docker stack on private network infrastructure, automatically backing up mobile photos in the background and serving as an on-premise personal media cloud.
Architecture Overview
📱
Mobile Devices
Immich iOS/Android auto-sync
→
🏠
Local Gateway
Private home network routing
→
🐳
Immich Stack
Microservices, Redis, Postgres
→
💾
Local Storage
Encrypted persistent storage
Key Components
- Multi-Container Orchestration: Immich server, microservices, machine learning model, PostgreSQL database, and Redis cache configured via Docker Compose.
- Historical Google Takeout Ingestion: Decompressed and organized multi-gigabyte Google Takeout archives, preserving metadata and EXIF timestamps.
- Background Synchronization: Automatic background upload from mobile devices whenever connected to designated local Wi-Fi networks.
- Volume Mounts & Backups: Isolated persistent volumes for RAW originals, web thumbnails, and transactional database dumps.
A fast, responsive typing practice and assessment web app inspired by 10fastfingers. Runs entirely in the client browser with low-latency keypress capture, dynamic word scrambling, real-time WPM/accuracy tracking, and a rolling 24-hour leaderboard.
Architecture Overview
🌐
Browser UI
Zero-dependency static frontend
→
⌨️
Typing Engine
60-second real-time countdown
→
📊
Telemetry
Live WPM, CPM & error metrics
→
🏆
Leaderboard
Rolling 24-hour top scores
Key Components
- Zero Build Step: Plain vanilla JavaScript, semantic HTML5, and CSS3 without framework overhead.
- High-Frequency Key Listening: Sub-millisecond keystroke evaluation calculating gross WPM, net WPM, and accuracy percentages.
- Rolling 24h Leaderboard: Automatically prunes expired scores while maintaining competition integrity.