모니터링 스택

Prometheus + Grafana + Alertmanager로 구성하는 완전한 모니터링 시스템

---

Prometheus + Grafana + Alertmanager로 구성하는 완전한 모니터링 시스템

목표: 프로덕션 환경의 메트릭을 수집, 시각화, 알림할 수 있는 수준

---

📊 모니터링 스택 개요

아키텍처

graph TB
    subgraph "모니터링 스택"
        APP["📦 Application"] -->|"/metrics"| PROM["🎯 Prometheus"]
        NODE["💻 Node"] -->|"node_exporter"| PROM
        K8S["☸️ Kubernetes"] -->|"kube-state-metrics"| PROM
        
        PROM -->|"질의"| GRAFANA["📊 Grafana"]
        PROM -->|"알림"| ALERT["🚨 Alertmanager"]
        ALERT -->|"알림"| SLACK["💬 Slack"]
        ALERT -->|"알림"| PAGERDUTY["📟 PagerDuty"]
    end

---

🎯 Prometheus

설치

# Docker
 docker run -d \
  --name prometheus \
  -p 9090:9090 \
  -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

# Kubernetes
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus

설정

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

rule_files:
  - /etc/prometheus/rules/*.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']
  
  - job_name: 'application'
    static_configs:
      - targets: ['app:8080']
    metrics_path: '/metrics'
    scrape_interval: 5s

---

📈 메트릭 수집

애플리케이션 메트릭

# Python (prometheus_client)
from prometheus_client import Counter, Histogram, Gauge, start_http_server
import time

# 카운터
requests_total = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

# 히스토그램
request_duration = Histogram(
    'http_request_duration_seconds',
    'HTTP request duration',
    ['method', 'endpoint']
)

# 게이지
active_connections = Gauge(
    'active_connections',
    'Number of active connections'
)

# 사용
@request_duration.time()
def handle_request(method, endpoint):
    active_connections.inc()
    try:
        # 처리
        requests_total.labels(method=method, endpoint=endpoint, status='200').inc()
    except Exception:
        requests_total.labels(method=method, endpoint=endpoint, status='500').inc()
    finally:
        active_connections.dec()

# 메트릭 서버
start_http_server(8000)

Node Exporter

# 설치
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.6.1.linux-amd64.tar.gz
sudo cp node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/

# 서비스 등록
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
[Unit]
Description=Node Exporter

[Service]
ExecStart=/usr/local/bin/node_exporter
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl enable node_exporter
sudo systemctl start node_exporter

---

🚨 Alertmanager

설정

# alertmanager.yml
global:
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

route:
  group_by: ['alertname', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
      continue: true
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'default'
    slack_configs:
      - channel: '#alerts'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ .CommonAnnotations.description }}'
  
  - name: 'slack'
    slack_configs:
      - channel: '#warnings'
        send_resolved: true
  
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_KEY'
        severity: critical

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

알림 규칙

# rules.yml
groups:
  - name: node_alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage detected"
          description: "CPU usage is above 80% for 5 minutes"
      
      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 90
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High memory usage detected"
          description: "Memory usage is above 90% for 5 minutes"
      
      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space"
          description: "Disk space is below 10%"
      
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance is down"
          description: "{{ $labels.instance }} has been down for more than 1 minute"

---

📊 Grafana

설치

# Docker
 docker run -d \
  --name grafana \
  -p 3000:3000 \
  -e GF_SECURITY_ADMIN_PASSWORD=admin \
  -v grafana-storage:/var/lib/grafana \
  grafana/grafana

# Kubernetes
helm install grafana grafana/grafana

데이터소스 설정

# datasources.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true
  
  - name: Loki
    type: loki
    url: http://loki:3100
    access: proxy

대시보드

{
  "dashboard": {
    "title": "Node Exporter Full",
    "panels": [
      {
        "title": "CPU Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "100 - (avg by(instance) (irate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
            "legendFormat": "{{instance}}"
          }
        ]
      },
      {
        "title": "Memory Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100",
            "legendFormat": "{{instance}}"
          }
        ]
      }
    ]
  }
}

---

📋 체크리스트

• Prometheus 설치
• 메트릭 수집 설정
• 알림 규칙 작성
• Alertmanager 설정
• Grafana 설치
• 데이터소스 연결
• 대시보드 생성
• 알림 채널 설정
• SLA 모니터링
• 장기 저장 설정

---

🔗 관련 노트

• 03-05-모니터링-Prometheus-Grafana — Kubernetes 모니터링
• 02-05-로그-집계 — 로그 수집
• 03-03-SRE-원칙 — SLO/SLI

---

마지막 업데이트: 2026-06-01