로그 집계

중앙 집중식 로그 수집, 저장, 분석 시스템 구축

중앙 집중식 로그 수집, 저장, 분석 시스템 구축

목표: 분산 시스템의 로그를 효과적으로 수집하고 분석할 수 있는 수준

📚 로그 집계 개요

아키텍처

graph TB
    subgraph "로그 수집"
        APP["📦 Application"] -->|"stdout"| AGENT["🚚 Log Agent"]
        SYS["💻 System"] -->|"syslog"| AGENT
    end
    
    subgraph "로그 처리"
        AGENT -->|"전송"| BUFFER["📦 Buffer"]
        BUFFER -->|"처리"| PARSER["🔧 Parser"]
    end
    
    subgraph "로그 저장"
        PARSER -->|"저장"| STORE["🗄️ Storage"]
    end
    
    subgraph "로그 분석"
        STORE -->|"검색"| SEARCH["🔍 Search"]
        STORE -->|"시각화"| VISUAL["📊 Visualization"]
    end

🚚 수집 에이전트

Fluentd

# fluentd.conf
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-docker.pos
  tag kubernetes.*
  <parse>
    @type multi_format
    <pattern>
      format json
      time_key time
      time_format %Y-%m-%dT%H:%M:%S.%NZ
    </pattern>
    <pattern>
      format regexp
      expression /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
    </pattern>
  </parse>
</source>

<filter kubernetes.**>
  @type kubernetes_metadata
</filter>

<match kubernetes.**>
  @type elasticsearch
  host elasticsearch
  port 9200
  logstash_format true
  logstash_prefix k8s-logs
  <buffer>
    @type file
    path /var/log/fluentd-buffers/kubernetes.system.buffer
    flush_mode interval
    retry_type exponential_backoff
    flush_thread_count 2
    flush_interval 5s
    retry_forever false
    retry_max_interval 30
    chunk_limit_size 2M
    queue_limit_length 8
    total_limit_size 500M
  </buffer>
</match>

Filebeat

# filebeat.yml
filebeat.inputs:
- type: container
  paths:
    - /var/log/containers/*.log
  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"

output.elasticsearch:
  hosts: ['elasticsearch:9200']
  index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

setup.kibana:
  host: "kibana:5601"

logging.level: info

Fluent Bit

# fluent-bit.conf
[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   On
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token

[OUTPUT]
    Name            es
    Match           *
    Host            elasticsearch
    Port            9200
    Logstash_Format On
    Logstash_Prefix k8s-logs

🗄️ 저장소

Elasticsearch

# elasticsearch.yml
cluster.name: logging-cluster
node.name: ${HOSTNAME}
path.data: /usr/share/elasticsearch/data
path.logs: /usr/share/elasticsearch/logs
network.host: 0.0.0.0
discovery.type: single-node
xpack.security.enabled: false

# 인덱스 템플릿
PUT _index_template/logs
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-policy",
      "index.lifecycle.rollover_alias": "logs"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "level": { "type": "keyword" },
        "message": { "type": "text" },
        "service": { "type": "keyword" },
        "host": { "type": "keyword" }
      }
    }
  }
}

Loki (경량 대안)

# loki-config.yml
auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h

storage_config:
  boltdb:
    directory: /tmp/loki/index
  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

📊 시각화

Kibana

# kibana.yml
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://elasticsearch:9200"]

Grafana + Loki

# datasources.yml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    url: http://loki:3100
    jsonData:
      maxLines: 1000

🔍 로그 분석

검색 쿼리

# 기본 검색
kubernetes.pod.name: "my-app-*"

# 로그 레벨
level: "ERROR"

# 시간 범위
@timestamp:[now-1h TO now]

# 복합 쿼리
kubernetes.namespace.name: "production" AND level: "ERROR"

# 정규식
message: /exception|error|failed/

LogQL (Loki)

# 기본 선택
{job="kubernetes-pods"}

# 레이블 필터
{job="kubernetes-pods", namespace="production"}

# 라인 필터
{job="kubernetes-pods"} |= "error"
{job="kubernetes-pods"} != "debug"
{job="kubernetes-pods"} |~ "err.*or"

# 파서
{job="kubernetes-pods"} 
  | json 
  | level="error" 
  | line_format "{{.message}}"

# 집계
sum(rate({job="kubernetes-pods"} |= "error" [1m])) by (namespace)

📋 체크리스트

🔗 관련 노트

02-04-모니터링-스택 — 메트릭 모니터링
03-07-분산추적 — 분산 추적
03-06-로깅-EFK-스택 — Kubernetes 로깅

마지막 업데이트: 2026-06-01