Kubernetes fluentd 배포 이해와 작성

이 글은 Fluentd를 Kubernetes 클러스터 내부에 배포해서 Spring Boot 어플리케이션 로그를 수집하고 OpenSearch로 전송하는 구조 구성을 다룹니다. 최종 Next.js Nest Rust 등을 추가적으로 다룰 예정.

 

1. 아키텍처 구성도

 

2. 쿠버네티스 내부에 구성하기

2.1. create namespace

kubectl create namespace logging-dev

 

추가된 것을 확인

 

2.2. fluentd-dev.yaml

아래의 내용을 kubectl apply -f fluentd-dev.yaml 하면 추가는 잘 될 것이다.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
  namespace: logging-dev
  labels:
    k8s-app: fluentd-logging
data:
  fluent.conf: |
    <match fluent.**>
      type null
    </match>
    <source>
      @type tcp
      port 24220
      format json
      tag applog
    </source>
    <match applog>
      @type rewrite_tag_filter
      <rule>
        key project
        pattern ^(.+)$
        tag $1.${tag}
      </rule>
    </match>

    <match **applog**>
      @type copy
      <store>
        @type opensearch
        hosts http://192.168.2.126:9200
        scheme "#{ENV['FLUENT_ELASTICSEARCH_SCHEME'] || 'http'}"
        ssl_verify "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERIFY'] || 'true'}"
        ssl_version "#{ENV['FLUENT_ELASTICSEARCH_SSL_VERSION'] || 'TLSv1_2'}"
        user "#{ENV['FLUENT_ELASTICSEARCH_USER'] || use_default}"
        password "#{ENV['FLUENT_ELASTICSEARCH_PASSWORD'] || use_default}"        
        reload_connections false
        reconnect_on_error true
        reload_on_failure true
        log_es_400_reason true
        logstash_format true
        logstash_prefix ${tag}
        logstash_dateformat %Y%m%d
        include_tag_key true
        tag_key @log_name
        request_timeout 30000s
        slow_flush_log_threshold 300.0
        flush_mode interval
        <buffer>
          flush_thread_count "8"
          flush_interval "10s"
          chunk_limit_size "5M"
          queue_limit_length "512"
          retry_max_interval "30"
          retry_forever true
        </buffer>
      </store>
    </match>
  config-copy.sh: |
    #!/bin/sh
    cp -a /config-volume/..data/fluent.conf /fluentd/etc/fluent.conf
    tini -- /fluentd/entrypoint.sh
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluentd
  namespace: logging-dev
  labels:
    k8s-app: fluentd-logging
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: fluentd-logging
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
    spec:
      containers:
        - name: fluentd
          command: ["sh", "-c", "/config-volume/..data/config-copy.sh"]
          image: fluent/fluentd-kubernetes-daemonset:v1.16-debian-opensearch-1
          env:
            - name: FLUENT_ELASTICSEARCH_HOST
              value: "192.168.2.126"
            - name: FLUENT_ELASTICSEARCH_PORT
              value: "9200"
            - name: FLUENT_ELASTICSEARCH_SCHEME
              value: "http"
            - name: FLUENTD_SYSTEMD_CONF
              value: "disable"
            - name: FLUENT_UID
              value: "0"
            - name: FLUENT_ELASTICSEARCH_SSL_VERIFY
              value: "false"
            - name: FLUENT_ELASTICSEARCH_USER
              valueFrom:
                secretKeyRef:
                  name: opensearch-credentials
                  key: username
            - name: FLUENT_ELASTICSEARCH_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: opensearch-credentials
                  key: password
          resources:
            limits:
              memory: 400Mi
            requests:
              cpu: 100m
              memory: 200Mi
          ports:
            - name: fluentd-source
              containerPort: 24220
              protocol: TCP
          volumeMounts:
            - name: config-volume
              mountPath: /config-volume
      terminationGracePeriodSeconds: 30
      volumes:
        - name: config-volume
          configMap:
            name: fluentd-config
            defaultMode: 0777
---
apiVersion: v1
kind: Service
metadata:
  name: fluentd-svc
  namespace: logging-dev
  labels:
    k8s-app: fluentd-logging
spec:
  type: ClusterIP
  selector:
    k8s-app: fluentd-logging
  ports:
    - name: fluentd-source
      port: 24220
      targetPort: fluentd-source
      protocol: TCP
---
apiVersion: v1
data:
  username: base64-value
  password: base64-value
kind: Secret
metadata:
  name: opensearch-credentials
  namespace: logging-dev

3. 실 어플리케이션 확인

본 작업에서는 Spring Boot 프로젝트의 logback-spring.xml 에서 logstash 구성 확인이다.
참고로 logstashIp 는 위에서 추가한 logging-dev fluentd-svc ClusterIP 10.106.207.140 24220 처럼
추가된 내용을 확인하면된다.

 

실제 구성된 내용으로 Spring Boot 로그를 확인해보자.

14:24:43,322 |-WARN in net.logstash.logback.appender.LogstashTcpSocketAppender[sqlLogAppender] - Log destination 10.106.207.140/<unresolved>:24220: connection failed. java.net.ConnectException: Connection refused
	at java.net.ConnectException: Connection refused
	at 	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at 	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:669)
	at 	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:549)
	at 	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
	at 	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:331)
	at 	at java.base/java.net.Socket.connect(Socket.java:630)
	at 	at net.logstash.logback.appender.AbstractLogstashTcpSocketAppender$TcpSendingEventHandler.openSocket(AbstractLogstashTcpSocketAppender.java:765)

 

위와 같이 오류 로그가 올라온다면 하나씩 트러블 슈팅을 해주어야한다.

 

1. username, password 는 (필히) base64 인코딩 재확인

2. 기본 오픈서치 클러스터 포트는 9200 (5601은 대시보드 기본포트)

3. fluentd 적용/갱신이 잘 되지 않는 다면 ? (kubectl rollout restart deployment fluentd -n logging-dev)

 

4. 요소 분석하기

kind: ConfigMap

@type tcp: 외부 애플리케이션(Spring Boot)이 로그를 보낼 TCP 포트를 오픈 (24220)
@type opensearch: 로그를 OpenSearch로 전송 (9200 포트)

Deployment: Fluentd Pod 정의

컨테이너 시작 시 ConfigMap에서 `fluent.conf`를 `/fluentd/etc`로 복사하고 Fluentd 실행하는데
기초 이미지의 내용을 사용하기에 별도 작성없이 그대로 사용

OpenSearch 접속 정보는 환경변수 + Secret 으로 분리 구성

env name FLUENT_ELASTICSEARCH_HOST
env name FLUENT_ELASTICSEARCH_PORT
env name FLUENT_ELASTICSEARCH_USER 등

kind: Secret

username: base64 인코딩 값
password: base64 인코딩 값

문제 상황 및 해결 사례

OpenSearch 주소로 `5601` 를 잘못 입력함. `9200` (클러스터) 으로 변경 배포 `kubectl rollout restart`

ConfigMap 반영 안됨 `kubectl rollout restart deployment fluentd -n logging-dev`

 

참고 명령어 정리

# ConfigMap 확인
kubectl get configmap fluentd-config -n logging-dev -o yaml

# Fluentd 로그 확인
kubectl logs -n logging-dev deploy/fluentd

# 현재 설정 적용 여부 확인
kubectl exec -it -n logging-dev deploy/fluentd -- cat /fluentd/etc/fluent.conf

# Fluentd 포트 리슨 확인
kubectl exec -it -n logging-dev deploy/fluentd -- netstat -tnlp | grep 24220