[kotlin] Spring-Batch Alert 처리 ( Logback을 이용한 Slack 연동 )

Develop/spring-batch

[kotlin] Spring-Batch Alert 처리 ( Logback을 이용한 Slack 연동 )

에디개발자 2021. 8. 2. 07:00

이번 글에서는 Spring-Batch에서 오류가 났을 경우 에러를 Alert처리하는 글을 정리하겠습니다.

모든 소스는 Github에 올려두었습니다.

Alert처리하는 방법은 여러가지 종류가 있습니다.

Prometheus, Alert-Manager를 이용한 방법
Slack을 이용한 Webhook 방법
그 외 등등..

이번 글에서는 Slack을 이용한 Webhook 방법에 대해서 정리하겠습니다.

Prometheus, Alert-Manager를 이용한 방법은 추 후에 작성예정입니다. Prometheus, Alert-Manager 구현 방법에 대해서는 여기 를 참조해주시기 바랍니다.

적용 방법

Logback을 적용하여 Error 로그가 작성된 경우 Slack으로 Webhook을 날리는 방식입니다.

Slack Channel 설정
build.gradle.kts 의존성 추가
Logback (Slack Appender) 설정
적용 후 Test

Slack Channel 설정

Slack에서 Channel을 생성

Incoming webhook 설정

앱 추가버튼을 클릭합니다.

Slack에서 추가한 Channel을 선택합니다.

웹후크 URL이 Webhook URL 입니다.

build.gradle.kts 의존성 추가

dependencies {

    // 1)
    // spring batch
    implementation("org.springframework.boot:spring-boot-starter-batch")  

    // 2)
    // slack
    val slackWebhookVersion = "1.4.0"
    implementation("net.gpedro.integrations.slack:slack-webhook:${slackWebhookVersion}")

    // 3)
    // Apache Commons
    val commonsTextVersion = "1.9"
    implementation("org.apache.commons:commons-text:$commonsTextVersion")
    
}

1) logback 의존성은 spring-boot-starter-batch에 포함되어있습니다.

2) Slack에 Webhook 설정을 위한 의존성을 추가합니다.

3) Json 데이터를 Pretty하게 보여주기 위한 의존성입니다.

Logback (Slack Appender) 설정

application.yml

Logback 설정 파일을 지정하는 설정파일을 추가합니다.

logging:
  config: classpath:logs/logback.xml

application.yml에 설정한 경로에 파일을 아래와 같이 생성합니다.

default-log-constant.xml

<?xml version="1.0" encoding="UTF-8"?>
<!-- 로그 상수 설정 -->
<included>
    <!-- CONSOLE LOG PRINT PATTERN -->
    <property name="CONSOLE_LOG_PATTERN" value="[%d][%highlight(%-5level)] %magenta(%-4relative) [%boldYellow(%t)] %cyan(%logger{30}) [%boldGreen(%method:%line)] - %msg%n%throwable"/>
    <property name="CONSOLE_LOG_PATTERN" value="${CONSOLE_LOG_PATTERN:-%clr(%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}"/>

    <!-- LOG FILE PRINT PATTERN -->
    <property name="FILE_LOG_PATTERN" value="${FILE_LOG_PATTERN:-%d{${LOG_DATEFORMAT_PATTERN:-yyyy-MM-dd HH:mm:ss.SSS}} ${LOG_LEVEL_PATTERN:-%5p} ${PID:- } --- [%t] %-40.40logger{39} : %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}}"/>

    <!-- 기본 로그 경로 -->
    <property name="DEFAULT_LOG_HOME" value="logs"/>
    <!-- ERROR 로그 경로 -->
    <property name="DEFAULT_ERROR_LOG_HOME" value="logs"/>

    <!-- 운영서버가 아닌 서버의 최대 로그 저장 기간 -->
    <property name="DEFAULT_MAX_HISTORY" value="10" />
    <!-- 운영서버 최대 로그 저장 기간 -->
    <property name="PROD_MAX_HISTORY" value="90" />
    <!-- 최대 로그 파일 사이즈 MB -->
    <property name="DEFAULT_MAX_FILESIZE" value="500MB" />

    <!-- SLACK TITLE -->
    <property name="SLACK_TITLE" value="서비스 장애 감지"/>
    <!-- 기본 BOT NAME -->
    <property name="SLACK_NAME" value="Kotlin-Batch"/>
    <!-- 기본 BOT EMOJI -->
    <property name="SLACK_EMOJI" value=":exclamation:"/>

    <property name="SLACK_CHANNEL" value="kotlin-batch-alert"/>
    <!-- HOOK LOCAL URL -->
    <property name="HOOK_URL" value="https://hooks.slack.com/services/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"/>
</included>

SLACK TITLE: Slack Message에 보여질 Title을 설정합니다.
SLACK_NAME: Slack Message에 보여질 Name을 설정합니다.
SLACK_EMOJI: Slack Message에 보여질 이모티콘을 설정합니다.
SLACK_CHANNEL: 이전에 생성한 Channel 명을 입력합니다.
HOOK_URL: 이전에 생성한 Webhook Url을 입력합니다.

default-log-setting.xml

<?xml version="1.0" encoding="UTF-8"?>
<!-- 로그 APPENDER 및 기본 로그 셋팅 -->
<included>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>

    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <layout class="ch.qos.logback.classic.PatternLayout">
            <pattern>${CONSOLE_LOG_PATTERN}</pattern>
        </layout>
    </appender>

    <appender name="DEFAULT_ERROR_LOG_APPENDER" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${DEFAULT_ERROR_LOG_HOME}/error.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${DEFAULT_ERROR_LOG_HOME}/archived/error.%d{yyyy-MM-dd}.%i.log.zip</fileNamePattern>
            <timeBasedFileNamingAndTriggeringPolicy
                    class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>${DEFAULT_MAX_FILESIZE:-500MB}</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
            <maxHistory>${DEFAULT_MAX_HISTORY:-10}</maxHistory>
        </rollingPolicy>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>ERROR</level>
        </filter>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <charset>UTF-8</charset>
            <pattern>${FILE_LOG_PATTERN}</pattern>
        </encoder>
    </appender>

    <appender name="DEFAULT_LOG_APPENDER" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${DEFAULT_LOG_HOME}/api.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${DEFAULT_LOG_HOME}/archived/api.%d{yyyy-MM-dd}.%i.log.zip</fileNamePattern>
            <timeBasedFileNamingAndTriggeringPolicy
                    class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
                <maxFileSize>${DEFAULT_MAX_FILESIZE:-500MB}</maxFileSize>
            </timeBasedFileNamingAndTriggeringPolicy>
            <maxHistory>${DEFAULT_MAX_HISTORY:-10}</maxHistory>
        </rollingPolicy>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>DEBUG</level>
        </filter>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <charset>UTF-8</charset>
            <pattern>${FILE_LOG_PATTERN}</pattern>
        </encoder>
    </appender>
    <!-- 스프링 프레임워크에서 찍는건 level을 info로 설정 -->
    <logger name="org.springframework" level="INFO"/>
</included>

ch.qos.logback.core.ConsoleAppender : 콘솔에 로그를 찍는 설정
ch.qos.logback.core.rolling.RollingFileAppender : 여러개의 파일을 순회하면서 로그를 찍는 설정

logback.xml

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="true">
    <include resource="logs/default-log-constant.xml"/>
    <include resource="logs/default-log-setting.xml"/>

    <!-- Slack 메신저에 ERROR 메세지 전송 -->
    <appender name="SLACK" class="me.practice.kotlinbatch.common.slack.appender.SlackAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>ERROR</level>
        </filter>
        <level>ERROR</level>
        <enabled>true</enabled>
        <token>${HOOK_URL}</token>
        <channel>${SLACK_CHANNEL}</channel>
        <!-- 에러 타이틀 (기본값: 서비스 장애 감지) -->
        <title>${SLACK_TITLE}</title>
        <!-- 설정 하지 않으면 기본값: App -->
        <botName>${SLACK_NAME}</botName>
        <!-- Emoji Shortcodes 참조  https://emojipedia.org (기본값: :exclamation:)-->
        <botEmoji>${SLACK_EMOJI}</botEmoji>
    </appender>
    <!-- Slack 메신저 에러 비동기 전송 처리 -->
    <appender name="ASYNC_SLACK" class="ch.qos.logback.classic.AsyncAppender">
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>ERROR</level>
        </filter>
        <appender-ref ref="SLACK" />
    </appender>

    <logger name="org.hibernate.validator" level="INFO"/>

    <!-- hibernate SQL 보기 -->
    <logger name="org.hibernate.SQL" level="debug" additivity="false"/>
    <logger name="org.hibernate.type" level="trace" additivity="false"/>

    <!--dev의 기본 패키지 로그 레벨은 DEBUG로 잡고 appender 등록, 기존의 default-log-setting.xml 파일 내의 spring 로그 레벨은 info로 등록-->
    <root level="INFO">
        <appender-ref ref="STDOUT"/>
        <appender-ref ref="DEFAULT_LOG_APPENDER"/>
        <appender-ref ref="DEFAULT_ERROR_LOG_APPENDER"/>
        <appender-ref ref="ASYNC_SLACK"/>
    </root>
</configuration>

SlackAppender를 추가하는 설정입니다.

Slack 메신저 에러를 비동기 전동 처리하는 설정입니다.

me.practice.kotlinbatch.common.slack.appender.SlackAppender

Slack에 전송할 메시지를 생성합니다.

package me.practice.kotlinbatch.common.slack.appender

import ch.qos.logback.classic.Level
import ch.qos.logback.classic.spi.ILoggingEvent
import ch.qos.logback.classic.spi.StackTraceElementProxy
import ch.qos.logback.core.UnsynchronizedAppenderBase
import ch.qos.logback.core.util.ContextUtil
import me.practice.kotlinbatch.common.slack.utils.JsonUtils
import me.practice.kotlinbatch.common.slack.utils.MDCUtils
import net.gpedro.integrations.slack.SlackApi
import net.gpedro.integrations.slack.SlackAttachment
import net.gpedro.integrations.slack.SlackField
import net.gpedro.integrations.slack.SlackMessage
import org.slf4j.Logger
import org.slf4j.LoggerFactory
import org.springframework.util.ObjectUtils
import java.net.SocketException
import java.net.UnknownHostException
import java.time.LocalDateTime
import java.time.format.DateTimeFormatter

class SlackAppender(
    private var enabled: Boolean = false,
    private var token: String? = null,
    private var channel: String? = null,
    private var level: Level? = null,
    private var title: String? = null,
    private var botName: String? = null,
    // https://slackmojis.com/
    private var botEmoji: String? = null
): UnsynchronizedAppenderBase<ILoggingEvent>() {

    override fun doAppend(eventObject: ILoggingEvent?) {
        super.doAppend(eventObject)
    }

    override fun append(loggingEvent: ILoggingEvent) {
        if (loggingEvent.level.isGreaterOrEqual(level)) {
            if (enabled) {
                toSlack(loggingEvent)
            }
        }
    }

    private fun toSlack(loggingEvent: ILoggingEvent) {
        if (loggingEvent.level.isGreaterOrEqual(level)) {
            val fields: MutableList<SlackField> = ArrayList()
            val slackAttachment = SlackAttachment()
            slackAttachment.setFallback("장애 발생")
            slackAttachment.setColor("danger")
            slackAttachment.setTitle(loggingEvent.formattedMessage)

            if (!ObjectUtils.isEmpty(MDCUtils[MDCUtils.AGENT_DETAIL_MDC])) {
                val agentDetail = SlackField()
                agentDetail.setTitle("사용자 환경정보")
                agentDetail.setValue(JsonUtils.toPrettyJson(MDCUtils[MDCUtils.AGENT_DETAIL_MDC]))
                agentDetail.isShorten = false
                fields.add(agentDetail)
            }

            val date = SlackField()
            date.setTitle("발생 시간")
            date.setValue(LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")))
            date.isShorten = true
            fields.add(date)

            val hostName = SlackField()
            hostName.setTitle("호스트명")
            hostName.setValue(getHostName())
            hostName.isShorten = true
            fields.add(hostName)

            if (!ObjectUtils.isEmpty(MDCUtils[MDCUtils.HEADER_MAP_MDC])) {
                val headerInformation = SlackField()
                headerInformation.setTitle("Http Header 정보")
                headerInformation.setValue(JsonUtils.toPrettyJson(MDCUtils[MDCUtils.HEADER_MAP_MDC]))
                headerInformation.isShorten = false
                fields.add(headerInformation)
            }
            if (!ObjectUtils.isEmpty(MDCUtils[MDCUtils.PARAMETER_MAP_MDC])) {
                val bodyInformation = SlackField()
                bodyInformation.setTitle("Http Body 정보")
                bodyInformation.setValue(MDCUtils[MDCUtils.PARAMETER_MAP_MDC])
                bodyInformation.isShorten = false
                fields.add(bodyInformation)
            }
            slackAttachment.setFields(fields)

            val slackMessage = SlackMessage("")
            slackMessage.setChannel("#$channel")
            slackMessage.setUsername("${getBotName()} - ${getTitle()}")
            // slackMessage.setIcon(botEmoji)
            slackMessage.setIcon(getBotEmoji())
            slackMessage.setAttachments(listOf(slackAttachment))
            val slackApi = SlackApi(token)
            slackApi.call(slackMessage)
        }
    }

    /**
     * Gets host name.
     *
     * @return the host name
     */
    fun getHostName(): String? {
        try {
            return ContextUtil.getLocalHostName()
        } catch (e: UnknownHostException) {
            e.printStackTrace()
        } catch (e: SocketException) {
            e.printStackTrace()
        }
        return ""
    }

    /**
     * Gets stack trace.
     *
     * @param stackTraceElements the stack trace elements
     * @return the stack trace
     */
    fun getStackTrace(stackTraceElements: Array<StackTraceElementProxy>?): String? {
        if (stackTraceElements == null || stackTraceElements.isEmpty()) {
            return null
        }
        val sb = StringBuilder()
        for (element in stackTraceElements) {
            sb.append(element.toString())
            sb.append("\n")
        }
        return sb.toString()
    }

    fun getTitle(): String? {
        return if(ObjectUtils.isEmpty(title)) "서비스 장애 감지" else title
    }

    fun setTitle(title: String?) {
        this.title = title
    }

    fun getBotName(): String? {
        return if(ObjectUtils.isEmpty(botName)) "App" else botName
    }

    fun getBotEmoji(): String? {
        return if(ObjectUtils.isEmpty(botEmoji)) ":exclamation:" else botEmoji
    }

    fun setBotEmoji(botEmoji: String?) {
        this.botEmoji = botEmoji
    }

}

Slack 메세지를 Pretty하게 도와주는 클래스 파일입니다.

JsonUtils

package me.practice.kotlinbatch.common.slack.utils

import com.fasterxml.jackson.core.JsonProcessingException
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.databind.SerializationFeature
import com.google.gson.Gson
import com.google.gson.GsonBuilder
import org.apache.commons.text.StringEscapeUtils

class JsonUtils {

    private var gson: Gson = GsonBuilder()
        .setPrettyPrinting()
        .setLenient()
        .create()
    private var mapper: ObjectMapper = ObjectMapper()

    companion object {

        /**
         * Gets instance.
         *
         * @return the instance
         */
        private fun getInstance() = JsonUtils()

        fun getGson() = getInstance().gson

        fun getObjectMapper() = getInstance().mapper.configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false)

        /**
         * From json t.
         *
         * @param <T>     the type parameter
         * @param jsonStr the json str
         * @param cls     the cls
         * @return the t
        </T> */
        fun <T> fromJson(jsonStr: String, cls: Class<T>?): T {
            return getGson().fromJson(jsonStr, cls)
        }

        /**
         * To pretty json string.
         *
         * @param json the json
         * @return the string
         */
        fun toPrettyJson(json: String?): String? {
            if (json == null) return null
            return StringEscapeUtils.unescapeJava(getGson().toJson(fromJson(json, Any::class.java)))
        }

        /**
         * To ObjectMapper pretty json string
         * @param any
         * @return
         */
        @Throws(JsonProcessingException::class)
        fun toMapperPrettyJson(any: Any?): String = getObjectMapper().writerWithDefaultPrettyPrinter().writeValueAsString(any)
    }
}

MDCUtils

package me.practice.kotlinbatch.common.slack.utils

import org.slf4j.MDC

/**
 * httpRequest가 존재하는 동안 데이터를 유지시키기 위한 유틸
 * http://logback.qos.ch/manual/mdc.html
 */
class MDCUtils {

    companion object {
        private val mdc = MDC.getMDCAdapter()

        /**
         * The constant HEADER_MAP_MDC.
         */
        val HEADER_MAP_MDC = "HEADER_MAP_MDC"

        /**
         * The constant PARAMETER_MAP_MDC.
         */
        val PARAMETER_MAP_MDC = "PARAMETER_MAP_MDC"

        /**
         * The constant AGENT_DETAIL_MDC.
         */
        val AGENT_DETAIL_MDC = "AGENT_DETAIL_MDC"

        /**
         * Set.
         *
         * @param key   the key
         * @param value the value
         */
        operator fun set(key: String, value: String) {
            mdc.put(key, value)
        }

        /**
         * Get string.
         *
         * @param key the key
         * @return the string
         */
        operator fun get(key: String): String? {
            return mdc[key]
        }
    }

}

적용 후 테스트

import org.apache.logging.log4j.LogManager

@Configuration
class SimpleJobConfiguration(
    val log = LogManager.getLogger()

    @Bean("${BatchItem.SIMPLE}_READER")
    @StepScope
    fun reader(@Value("#{jobParameters[pageSize]}") pageSize: Int?): QuerydslPagingItemReader<Person> {
        val reader = QuerydslPagingItemReader(entityManagerFactory) { personRepository.findAllInBatch() }
        reader.pageSize = pageSize!!
        log.error("[Test] Batch Error....!")  // Slack Test!!
        return reader
    }
}

여기서 중요한 점은 logback을 사용하는 log를 사용해야만 합니다. import문을 반드시 확인해주세요!

Test를 위해 log.error 메세지를 임의로 발생시켜 테스트하면 Slack Channel에 아래와 같이 Alert이 나타는 것을 확인할 수 있습니다.

'Develop > spring-batch' 카테고리의 다른 글

[Kotlin] Spring-Batch QuerydslPagingItemReader 개선편 (1)	2021.08.04
[Kotlin] Spring-Batch (JPA 적용) Junit5를 이용한 Test Code 작성 (0)	2021.07.29
[Kotlin] Spring-Batch 적용 (0)	2021.03.25
Spring batch에 Spring Data JPA 기반 Querydsl을 적용해보자! (QuerydlsPagingItemReader) (0)	2020.11.09
Spring batch 스케줄 생성! [Jenkins] (0)	2020.11.04

현재글[kotlin] Spring-Batch Alert 처리 ( Logback을 이용한 Slack 연동 )

250x250

------ 한발자국씩 성장하자 ------ Github: https://github.com/yongtaelim LinkedIn: https://www.linkedin.com/in/%EC%9A%A9%ED%83%9C-%EC%9E%84-622b69218/

프로그래밍, java, 자바, spring-kafka, kotlin, JPQL, exception, queryDSL, 스터디, Prometheus, 라이브, SpringBatch, Spring, 백기선, 코틀린, 에디, 객체지향, 엘레강트, 백기선님, JPA,

Today :
Yesterday :

일	월	화	수	목	금	토
			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30	31

에디블로그