flink版本 1.14
安裝使用CDH 6.3

解決問題1：flink 日志不打印 flink啟動日志 flink web ui日志 均不打印
解決問題2：flink打印中文日志出現(xiàn)亂碼 中文日志變成了 ？？？？
解決問題3：flink打印日志輸出到kafka

flink啟動日志 flink web ui日志 均不打印

fix1: ${FLINK_HOME}/lib 項目jar包移除

查看日志：
SLF4J: Found binding in [jar:file:****/filecache/21/flink-spark.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file::****/application_1686032130046_6235/filecache/21/flink-rongshu.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file::****/application_1686032130046_6235/filecache/21/flink-stream.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file::****/application_1686032130046_6235/filecache/11/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:****/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

發(fā)現(xiàn)lib下有大量項目相關jar包 每個jar包都包含log4j的依賴

移除${FLINK_HOME}/lib 下的項目jar包到其他路徑

fix2: 排除項目代碼中l(wèi)og4j1依賴

發(fā)現(xiàn)項目依賴的log4j1，${FLINK_HOME}/lib下代碼為log4j2

    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.25</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>1.7.25</version>
        <scope>provided</scope>
    </dependency>

    <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-to-slf4j</artifactId>
        <version>2.14.0</version>
        <scope>provided</scope>
    </dependency>

項目包內(nèi)的日志相關jar包全部排除 使用${FLINK_HOME}/lib目錄下自帶的log4j2 jar包做日志打印
問題解決 出現(xiàn)日志

中文日志出現(xiàn)亂碼

fix1: 新增 flink.yaml 指定UTF-8

env.java.opts: "-Dfile.encoding=UTF-8"
web.character.encoding: UTF-8
問題未解決

fix2: 新增 提交腳本 指定jvm編碼

-Dfile.encoding=UTF-8
-Dsun.jnu.encoding=UTF-8
問題未解決

fix3: 新增 ${FLINK_HOME}/conf/log4j.properties

log4j.appender.YourAppender.Encoding=UTF-8
問題未解決

fix4: 新增 ${FLINK_HOME}/conf/log4j.properties

appender.main.layout.charset = UTF-8
問題解決

雖然flink日志配置叫l(wèi)og4j.properties 但實際上使用的log4j2 所以指定編碼需要使用log4j2的語法

最終日志

# Allows this configuration to be modified at runtime. The file will be checked every 30 seconds.
monitorInterval=30

# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender


# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO

# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
logger.shaded_zookeeper.name = org.apache.flink.shaded.zookeeper3
logger.shaded_zookeeper.level = INFO

# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = RollingFile
appender.main.append = true
appender.main.fileName = ${sys:log.file}
appender.main.filePattern = ${sys:log.file}.%i
appender.main.layout.type = PatternLayout
# appender.main.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} [] %-5p %-60c %x - %m%n
appender.main.layout.pattern = %-4r [%t] %-5p[%d{yyyy-MM-dd HH:mm:ss}][%c] %x-%m%n
appender.main.policies.type = Policies
appender.main.policies.size.type = SizeBasedTriggeringPolicy
appender.main.policies.size.size = 100MB
appender.main.policies.startup.type = OnStartupTriggeringPolicy
appender.main.strategy.type = DefaultRolloverStrategy
appender.main.strategy.max = ${env:MAX_LOG_FILE_NUMBER:-10}
# 2023.07.13 add encoding
appender.main.layout.charset = UTF-8


# 新增 Kafka appender
rootLogger.appenderRef.kafka.ref=kafka

appender.kafka.type = Kafka
appender.kafka.name = kafka
appender.kafka.topic = FLINK_APPLICATION_LOG

appender.kafka.property.type=Property
appender.kafka.property.name=bootstrap.servers
appender.kafka.property.value=kafka:9092
appender.kafka.layout.charset = UTF-8

appender.kafka.layout.type = PatternLayout
appender.kafka.layout.pattern = %d{yyyy-MM-dd HH:mm:ss}@|$%p@|$[%c{3}]:%L@|$${flink.job.name}@|$%m%n


# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name = org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF